IBM DB2 Analytics Accelerator

Transcription

1 IBM DB2 Analytics Accelerator Andreas Peschke Client Technical Architect zsw

2 Disclaimer Copyright IBM Corporation All rights reserved. U.S. Government Users Restricted Rights - Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp. THE INFORMATION CONTAINED IN THIS PRESENTATION IS PROVIDED FOR INFORMATIONAL PURPOSES ONLY. WHILE EFFORTS WERE MADE TO VERIFY THE COMPLETENESS AND ACCURACY OF THE INFORMATION CONTAINED IN THIS PRESENTATION, IT IS PROVIDED AS IS WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED. IN ADDITION, THIS INFORMATION IS BASED ON IBM S CURRENT PRODUCT PLANS AND STRATEGY, WHICH ARE SUBJECT TO CHANGE BY IBM WITHOUT NOTICE. IBM SHALL NOT BE RESPONSIBLE FOR ANY DAMAGES ARISING OUT OF THE USE OF, OR OTHERWISE RELATED TO, THIS PRESENTATION OR ANY OTHER DOCUMENTATION. NOTHING CONTAINED IN THIS PRESENTATION IS INTENDED TO, NOR SHALL HAVE THE EFFECT OF, CREATING ANY WARRANTIES OR REPRESENTATIONS FROM IBM (OR ITS SUPPLIERS OR LICENSORS), OR ALTERING THE TERMS AND CONDITIONS OF ANY AGREEMENT OR LICENSE GOVERNING THE USE OF IBM PRODUCTS AND/OR SOFTWARE. IBM, the IBM logo, ibm.com, DB2, InfoSphere, Cognos, and InfoSphere Warehouse on System z are trademarks or registered trademarks of International Business Machines Corporation in the United States, other countries, or both. If these and other IBM trademarked terms are marked on their first occurrence in this information with a trademark symbol ( or ), these symbols indicate U.S. registered or common law trademarks owned by IBM at the time this information was published. Such trademarks may also be registered or common law trademarks in other countries. A current list of IBM trademarks is available on the Web at Copyright and trademark information at Other company, product, or service names may be trademarks or service marks of others

3 Would You Still Use Google If It Took 3 Days And 7 People To Get A Search Result?

4 Patrick Hempeler days for a single query constant tuning Nearly 70% of data warehouses experience performance-constrained issues of various types. - Gartner 2010 Magic Quadrant specialized resources required months to deploy

5 Traditional Systems Landscape Applications OLTP Staging Area Operational Data Store Enterprise DWH Data Marts ETL ETL ETL ETL Historical reasons: Different access patterns impact on performance EDW as the data integration hub again, impact on performance Different life-cycle characteristics and again, impact on performance Different Service Level Agreements (SLA) Lack of broadly available workload management capabilities Choice of lower cost-of-acquisition offerings Negative ramifications: Complexity both in systems management and in applications Difficulties in supporting real time analytics Inability to match ever more demanding SLA requirements High total cost of ownership

6 Traditional data warehouses are just too complex They are based on databases optimized for transaction processing NOT to meet the demands of advanced analytics on big data. Too complex infrastructure Too inefficient at analytics Too complicated to deploy Too many people needed to maintain Too much tuning required Too costly to operate Too long to get answers

7 The right data warehouse is now mission critical. Data continues to expand exponentially. Analytics are becoming more complex as business demands faster answers

8 Visionary Systems Landscape Applications OLTP Staging Area Operational Data Store Enterprise DWH Data Marts ETL ETL ETL ETL Benefits Consolidating all the components into a single system Uniform access to any data Efficient data movement within the system (ideally, no network) Opportunity to remove some of the components Challenges Mixed workload management capabilities Universal processing capabilities to deliver best performance for both transactional and analytical workloads Providing industry leading availability, security and reliability to all types of workload System z Data Sharing and Parallel Sysplex technology provides all the needed characteristics except one: Special purpose processing for analytical workloads to minimize the need for manual tuning

9 GA: November 19,

10 ISAOpt V1 Needed Following Enhancements 1. Increase applicability by relaxing current off-load restrictions 2. Increase applicability by supporting larger amount of data 3. Support concurrent query execution 4. Improve data currency 5. Support disaster recovery 6. DB2 10 support

11 GA: November 25,

12 IBM DB2 Analytics Accelerator Capitalizing on the best of both worlds System z and Netezza What is it? The IBM DB2 Analytics Accelerator is a workload optimized, appliance add-on, that enables the integration of business insights into operational processes to drive winning strategies. It accelerates select queries, with unprecedented response times. How is it different Performance: Unprecedented response times to enable 'train of thought' analyses frequently blocked by poor query performance. Integration: Connects to DB2 through deep integration providing transparency to all applications. Self-managed workloads: queries are executed in the most efficient way Transparency: applications connected to DB2 are entirely unaware of the Optimizer Simplified administration: appliance hands-free operations, eliminating many database tuning tasks Breakthrough Technology Enabling New Opportunities

13 IDAA Preserves ISAOpt V1 Key Value Propositions DB2 continues to own data (both OLTP and DW) Access to data (authorization, privileges, ) Data consistency and integrity (backup, recovery, ) Enables extending System z QoS characteristics to BI/DW data as well Applications access data (both OLTP and DW) only through DB2 DB2 controls whether to execute query in DB2 mainline or route to ISAO DB2 returns results directly to the calling application Enables mixed workloads and selection of optimal access path (within DB2 mainline or ISAOpt/IDAA) depending on access pattern IDAA continues to be implemented as DB2 internal component DB2 provides key IDAA status and performance indicators as well as typical administration tasks by standard DB2 interfaces and means No direct access (log-on) to IDAA Enables operational cost reduction through skills, tools and processes consolidation

14 Deep DB2 Integration within zenterprise Applications DBA Tools, z/os Console,... Application Interfaces (standard SQL dialects) Operational Interfaces (e.g. DB2 Commands) DB2 for z/os Data Manager Buffer Manager... IRLM Log Manager IBM DB2 Analytics Accelerator Superior availability reliability, security, Workload mgmt z/os on System z Industry leading DW performance, ease of use Netezza 1000 HW

15 Query Execution Process Flow Application Interface Optimizer Heartbeat Application Query execution run-time for queries that cannot be or should not be off-loaded to ISAO ISAO interface DB2 for z/os Heartbeat (IDAA availability and performance indicators) Queries executed without IDAA Queries executed with IDAA

16 IDAA - Bringing Netezza AMPP TM Architecture to DB2 AMPP = Asymmetric Massively Parallel Processing Netezza 1000 CPU FPGA Advanced Analytics Memory BI Legacy Reporting DB2 for z/os V9 V10 IBM DB2 Analytics Accelerator IBM DB2 Analytics Accelerator SMP Host CPU FPGA Memory CPU FPGA DBA Memory Network Fabric S-Blades Disk Enclosures IBM DB2 Analytics Accelerator

17 The Key to the Speed select DISTRICT, PRODUCTGRP, sum(nrx) from SALES where MONTH = ' ' and MARKET = and SPECIALTY = 'GASTRO group by FPGA Core CPU Core Slice of table SALES (compressed) Uncompress Project Restrict, Visibility Complex Joins, Aggs, etc. sum(nrx) select DISTRICT, PRODUCTGRP, sum(nrx) where MONTH = ' ' and MARKET = and SPECIALTY = 'GASTRO'

18 Netezza 1000 basic design aspects Disk Enclosures SMP Hosts Snippet Blades TM (S-Blades, SPUs) Slice of User Data Swap and Mirror partitions High speed data streaming High compression rate EXP3000 JBOD Enclosures 12 x 3.5 1TB, 7200RPM, SAS (3Gb/s) max 116MB/s ( MB/s compressed data) e.g. TF12: 8 enclosures 96 HDDs 32TB uncompressed user data ( 128TB) IDAA Server SQL Compiler Query Plan Optimize Administration 2 front/end hosts, IBM 3650M3 clustered active-passive 2 Nehalem-EP Quad-core 2.4GHz per host Processor & streaming DB logic High-performance database engine streaming joins, aggregations, sorts, etc. e.g. TF12: 12 back/end SPUs

19 The Netezza S-Blade

20 Disk Mirroring and Failover Primary Mirror Temp All user data and temp space mirrored Disk failures transparent to queries and transactions Failed drives automatically regenerated Bad sectors automatically rewritten or relocated

21 Connectivity Options Multiple DB2 systems can connect to a single IDAA A single DB2 system can connect to multiple IDAAs Multiple DB2 systems can connect to multiple IDAAs Better utilization of IDAA resources Scalability High availability Full flexibility for DB2 systems: residing in the same LPAR residing in different LPARs residing in different CECs being independent (non-data sharing) belonging to the same data sharing group belonging to different data sharing groups

22 Network Configuration Options Option 1 Simple Direct attachment Virtual IP definition both on System z and Netezza Only one network link active at a time if Netezza fails over to standby host, connection might get lost OSA ports are configured such that only one link is active at a time. If the connection on this link breaks, the other is activated System z OSA port 1 port 2 SMP Host 1 port 1 port 2 10GbE SMP Host 2 IDAA port 1 port 2 10GbE Option 2 Additional redundancy or additional CEC requires Switch(es) Can address cable failures and Netezza fail over to standby host For higher availability requirements, a second switch is required OSA ports are configured such that packages are sent alternating, using both ports: multipathperconnection configuration System z OSA OSA port 1 port 2 port 1 port 2 Switch 1 Switch 2 SMP Host 1 port 1 port 2 10GbE SMP Host 2 port 1 port 2 10GbE IDAA

23 Network Configuration Options (continued) Option 3 zbx TOR Switch For clients with an installed zbx connected to zenterprise, the top-of-rack switch may be leveraged to connect to the IDAA Connection between zbx and IDAA can be direct or switched Ethernet Switch Private Service Network TOR Switch 10 GbE OSA- Express3 10 GbE OSX Private Data Network zenterprise zbx IDAA

24 Feedback from Beta Customer: Fast Time to Value IBM DB2 Analytics Accelerator (Netezza ) Production ready - 1 person, 2 days Table Acceleration Setup 2 Hours DB2 Add Accelerator Choose a Table for Acceleration Load the Table (DB2 copy to Netezza) Knowledge Transfer Query Comparisons Initial Load Performance 5.1 GB in 1 Min 25 Seconds (24M rows) 400 GB in 29 Min (570M rows) Actual Query Acceleration 1908x faster 2 Hours 39 Minutes to 5 Seconds CPU Utilization Reduction 99% less CPU 24M rows: 56.5 CPU seconds to 0.4 CPU seconds Actual customer results, October 2011

25 Large Insurance Company Adding value by Accelerating the Delivery of Business Reporting DB2 Only DB2 with IDAA Times Faster With Accelerated Time to Value IBM DB2 Analytics Accelerator (Netezza ) Production ready - 1 person, 2 days Table Acceleration Setup in 2 Hours - DB2 Add Accelerator - Choose a Table for Acceleration - Load the Table (DB2 Loads Data to the Accelerator) - Knowledge Transfer - Query Comparisons Total Rows Query Reviewed Query 1 591,941,065 2,813, ,320 2:39 9, ,908 Query 2 591,941,065 2,813, ,780 2:16 8, ,644 Query 3 813,343,052 8,260, :16 4, Query 4 283,105,125 2,813, ,197 1:08 4, Query 5 591,941,089 3,422, :57 4, Query 6 813,343,052 4,290, :53 3, Query 7 591,941, ,521 58,236 0:51 3, Customer Quote: Total Qualifying Rows Total Rows Returned Hours Sec(s) Hours Sec(s) Query 8 813,343,052 3,425, :44 2, ,320 Query 9 813,343,052 4,130, :42 2, Initial Load Performance 400 GB Loaded in 29 Minutes 570 Million Rows (Actual: Loaded 800 GB to 1.3 TB per hour) Extreme Query Acceleration x faster 2 Hours 39 minutes to 5 Seconds CPU Utilization Reduction Up to 35% we had this up and running in days with queries that ran over 1000 times faster

26 ISAO V1 Needed Following Enhancements IDAA addresses all of them! 1. Increase applicability by relaxing current off-load restrictions See further documentation for details 2. Increase applicability by supporting larger amount of data Up to 32TB of uncomressed data, e.g. with 1:4 compression ratio, up to 128TB of user data 3. Support concurrent query execution Exploiting Netezza workload management capabilities 4. Improve data currency Partition-scope update 5. Support disaster recovery Building blocks provided 6. DB2 10 support IDAA supports both DB2 9 and DB2 10* Requires zenterprise (z196 or z114) *DB2 10 support is planned for early 2012

27 You can also benefit! Are there long running queries that could provide business value if they could be run in seconds vs 30 minutes or more? What about performance challenges with complex and ad hoc queries? Is modernization of the data warehouse or Operational Business Analytics a topic of interest? If you answer one or more of the Are there following thoughts questions about extending with yes the use please of operational give platform data to perform business analysis and daily reporting? me another 3 minutes to show you how we can help you analyzing your Is there a data warehouse acceleration running potenzial on System z or an intention to do so? Is an OLAP application running out of steam due to growth in data. These are usually a single subject area such as Accounting, Sales or Inventory. The forgotten query: Have queries been elected to set aside due to performance challenges? Is there a System z/196 or z114 or a plan for one in the near future?

28

29 Quick Workload Test Customer Collecting information from dynamic statement cache, supported by stepby-step instruction and REXX script (small effort for customer) Uploading compressed file (up to some MB) to IBM FTP server Report for a first assessment: Acceleration potential for Queries Estimated time CP cost IBM / Center of Excellence Importing data into local database Quick analysis based on known DB2 Analytics Accelerator capabilities Customer Database Documentation and REXX procedure Data package (mainly unload data sets) Pre-process and load IBM lab Database Quick Workload Test Tool Report Assessment

30 Workload Assessment Output: Sample PDF Report Summary based on queries, query blocks, elapsed time and CPU time Reasons why certain query blocks might not run on Smart Analytics Optimizer How much of the current elapsed time may run on Smart Analytics Optimizer Detailed query-level assessment of the workload Elapsed time per query SQL statement per query

31 In eigener Sache... 2 Events 1 Preis Zusammen mit den Experten der IBM Technical Support Organisation haben unsere deutschsprachigen IBM System z Spezialisten ein umfassendes Themenspektrum zu aktuellen Neuerungen und integrierte Software- und Hardware-Lösungen rund um den legendären Mainframe ausgewählt. Wir freuen uns darauf, Ihnen Informationen in XXL sowie brandheißes Wissen aus erster Hand zu präsentieren. In insgesamt 10 Tracks erfahren Sie, wie Sie das Potenzial der leistungsstarken Plattform für Ihr Unternehmen maximal nutzen können. Namhafte Sprecher der IBM zeigen Ihnen anschaulich, wie die neuesten Entwicklungen in den Bereichen der IBM Mainframetechnologie dazu beitragen können, Innovationen voranzutreiben, Kosten zu sparen und Ihre bestehenden Investitionen, geschäftskritischen Daten sowie Prozesse unternehmensweit zu schützen. Und das alles zu einem sensationellen Preis: Profitieren Sie bis zum 15. März 2012 vom exklusiven Frühbucherrabatt von 199, für beide Tage und melden Sie sich am besten gleich hier an! Nicht vergessen! Sie haben die einmalige Möglichkeit, an zwei Events zu einem Preis teilzunehmen: Das IBM Information Management Forum 2012 zum Thema Manage Big Data findet zeitgleich statt

32 Disaster Recovery Considerations (1 of 2) App 1 SYSPLEX DSG Member 1 DSG Member 2 App 2 of App 4 of App 5 App 4 App 3 of App 1 of App 2 of App 3 App 5 Short Range Switch Short Range IDAA Instance 1 Long Range Short Range Switch Short Range IDAA Instance 2 of App 1 of App 2 of App 3 of App 4 of App

33 IBM DB2 Analytics Accelerator Disaster Recovery Considerations (2 of 2) App 1 SYSPLEX App 1 DSG Member 1 DSG Member 2 App 2 of App 4 App 2 of App 5 App 3 of App 1 App 3 of App 2 of App 3 App 4 Short Range Short Range Long Range Switch Short Range Short Range IDAA Instance 2 IDAA Instance 1 CREATE/LOAD of App of App 2 App 5 Switch of App 3 of App 1 of App 2 of App 4 of App 3 of App 5