Overview of How SAP IQ Augments the SAP Technology Landscape with Temperature Sensitive Data Management

Transcription

1 Orange County Convention Center Orlando, Florida June 3-5, 2014 Overview of How SAP IQ Augments the SAP Technology Landscape with Temperature Sensitive Data Management Andrew Neugebauer, Director, SAP IQ Product Management

2 LEARNING POINTS The SAP HANA platform is a cohesive and integrated set of technologies, which supports a comprehensive suite of business applications SAP IQ is a key component of the HANA platform for warm/cool data management, and extreme data volume scenarios SAP IQ has become the preferred technology for temperature sensitive data management at SAP as demonstrated by these solutions: SAP BW NLS (Near Line Store) SAP ERP ILM (Information Lifecycle Management) SAP HANA SDA (Smart Data Access) to SAP IQ SAP HANA extended storage

3 RETURN ON INVESTMENT SAP IQ is a highly scalable analytics database that runs on commodity hardware. SAP IQ can scale out an SAP HANA big data deployment at an economical price point. SAP HANA with SAP IQ together can handle the bulk of your big data analytic problems without bringing in alternative technologies such as Hadoop.

4 Agenda Business today needs a new data platform SAP IQ value in the SAP HANA platform Smart Data Access: SAP HANA and SAP IQ federation SAP IQ as Near-Line Storage (NLS) for BW SAP IQ as Information Lifecycle Management (ILM) for Business Suite HANA extended tables Key learnings

5 BUSINESS TODAY NEEDS A NEW DATA PLATFORM

6 End to End Data Management for Real Time Business Workforce of the Future Cloud Big Data Industries Internet of Things Business & Consumer Applications Custom Development ISVs & OEMs ERP TRANSACT STORE ANALYZE PREDICT SAP DATA MANAGEMENT 2014 SAP AG or an SAP affiliate company. All rights reserved. 6

7 SAP In-Memory Data Fabric Results at the Speed of Memory Data Fabric Layer SAP HANA SQL or SAP River Business Applications SAP HANA Streams SDA SDA MapReduce / Hive SAP Data Services Other Sources Real-time Events/ Machine-generated Data Petabytes of Structured Data Op RDBMS Hadoop Load Source Databases

8 SAP IQ VALUE IN THE SAP HANA PLATFORM

9 SAP PowerDesigner modeling SAP HANA Platform Unified open software platform for real-time business SAP Real-Time Data Platform SAP ASE transactions SAP ESP streams Common programming APIs In-memory/ real-time SAP HANA Hadoop Big data SAP IQ EDW SAP SQL Anywhere mobile/embedded SAP Data Services / SAP Information Steward information management SAP Control Center monitoring SAP HANA Platform foundations Cross-paradigm data access for new models of value discovery Hyper-performance on all classes of application and usage scenarios Price-performance value across all use cases Benefits Execute, record, analyze, and optimize without system limitations Embrace & extend across variations of data forms and processing models Common modeling, integrated development environment, shared systems management infrastructure, and deployment-independent solutions Trusted and unified data environment See Appendix for abbreviations

10 What is SAP IQ? SAP IQ (formerly SAP Sybase IQ) is a market leading, high-speed columnar analytics server and data warehouse which is designed specifically for Data Analytics. Designed for Analytics Columnar Database designed for high speed analytics Eliminates the problems caused by traditional row-based systems Unsurpassed Performance Up to 1,000X faster than row-based systems Enables high speed performance of complex queries against large datasets Lowest TCO Reduce the need for aggregates and indexes Greatly decreased ongoing maintenance costs when compared with row-based systems All Data Sources Large Data Volumes Single Data Warehouse Rapid Analytics

11 SAP IQ + SAP HANA Evolution in integration... Integrate Optimize Synthesize Next generation near-line smart store solution for BW/SAP HANA Cost effective store for SAP BW near-line, cool data Extended storage for BW/SAP HANA Extend large BW/SAP HANA system tables into SAP IQ for high data volume scenarios Extended storage for SAP HANA Extend HANA tables into IQ for flexible and value-based information processing SAP innovation without customer disruption See Appendix for abbreviations This is the current state of planning and may be changed by SAP at any time.

12 SAP IQ Relevance Number of Customers SAP IQ Teradata ExaData Netezza Source: Magic Quadrant for Data Warehouse Data Management Systems, Gartner, Inc. February, Mark A. Beyer, Donald Feinberg, Merv Adrian, Roxane Edjlail Exadata Numbers are from data warehouse and OLTP installs SAP NetWeaver NLS for SAP BW 100x Faster Queries then rowbased RDBMS 10X Faster Data Loads Reduced TCO by keeping older data in IQ SAP IQ for BusinessObjects BI Platform Bundled into platform Superior analytic performance Reduced TCO compared to competitors SAP HANA, BIG DATA EDITION Bundled into edition Data Tiering, Hadoop Access, Cold store Others: HANA REAL TIME DATA EDITION, Netweaver ILM, Applied Analytics, Lumira, Infinite Insights IQ is bundled into various SAP Offerings and will likely be bundled into more going forward.

13 Guinness World s Fastest Data Loader Fastest Data Loader Audited Record: Peak loading: 34.3TB / hour with 80 cores Sustained loading: 30TB / hour (even as tables and indexes grew, and storage capacity is approached) SAP Sybase IQ 16 BMMsoft EDMT 9 Red Hat Enterprise Linux 6.2 Intel 64 Documents, audio, video 6.5 million 13 parallel streams Structured data 500 million 2 parallel streams , SMS 5 million 1 stream HP ProLiant DL980 G7 8 x Intel Xeon 2.40GHz 1TB RAM 16 x Fibre Channel 3PAR StoreServ (8 nodes) 320 x 600GB 15Krpm

14 Guinness World s Largest Data Warehouse SAP HANA SAP IQ Running on 5 HP ProLiant DL580 G7 Servers 4 Active nodes with 1 standby 6.2TB of data SAP IQ multiplex running on 20 HP ProLiant DL580 G7 Servers 12.1PB of data (compressed into approx. 3.1PB of storage)

15 SMART DATA ACCESS: SAP HANA AND SAP IQ FEDERATION

16 SAP HANA smart data access capability Data virtualization for on-premise and hybrid cloud environments Benefits Transactional + Analytical SAP HANA HANA Tables Virtual Tables Enables access to remote data access just like local table Provides SAP HANA to SAP HANA queries Smart query processing including query decomposition with predicate push-down, functional compensation Supports data location agnostic development No special syntax to access heterogeneous data sources Non-disruptive evolution Teradata IQ Heterogeneous data sources SAP HANA to Hadoop (Hive) Hadoop SAP HANA ASE SAP HANA to Teradata SAP HANA to SAP HANA SAP HANA to SAP ASE SAP HANA to SAP IQ

17 Value Proposition - SAP HANA smart data access Easily utilize enterprise wide data leveraging HANA capabilities High Performance Secure access to remote data Leverage Big Data processing Seamless archived data access Easy to setup via virtual tables, and start writing apps using HANA features No need to load data from source to start the project saves cost, and is non-disruptive Significant query optimizations to leverage processing capability of target sources Move minimal data between SAP HANA and sources Access remote data securely utilizing secondary credentials Integrate output of Map-Reduce jobs in Hadoop/HIVE and access seamlessly from SAP HANA Store hot data in HANA, warm data in systems like SAP IQ and Hadoop, - get seamless access from HANA Queries in SAP HANA can integrate data from these systems

18 SIMPLE TO USE Decide Data Needs Decide what remote data sources are needed for the application running on HANA Define Sources Define the remote data source with the appropriate security credentials Create Virtual Tables Write The App Create virtual tables which reference the remote data source (table) Write your application, using HANA tables and virtual tables Query processor in HANA does the rest of the optimizations and data access

19 USE CASES Developing Applications Leveraging Data from across the Enterprise - where it sits with minimal data movement Big Data Integration with Hadoop / Hive Using SAP IQ to store warm archived data while providing seamless access to it from SAP HANA SAP NetWeaver Business Warehouse on HANA leveraging data in SAP Business Suite on HANA seamlessly

20 Use SAP IQ as a warm archive for SAP HANA Hana Studio Analytical and Transactional Applications SAP IQ can be used as a cost effective archive for warm data with SAP HANA Hana Server Hana tables SAP IQ Virtual tables READ SAP HANA contains the critical hot data and SAP IQ retains and process the archived warm data that is seamless accessible from SAP HANA.

21 SAP IQ AS Near-Line Storage (NLS) FOR SAP BW

22 SAP BW/HANA: Multi Temperature Data Approach hot Data is read and/or written frequently In memory No restrictions, all features available warm cold Non-Active Data Concept Infrequent access On disk, no need to keep in memory all the time No restrictions, all features available NLS Management for read-only data Sporadic access Not stored in HANA DB; stored in Near-line Storage Restricted to NLS capabilities Providing lower TCO by optimized data volume management

23 BW NLS Implementation for SAP IQ BW BW Porting InfoProvider Near-Line Adapter (ABAP based ) DBSL for SAP ASE (/w IQ ext.) General Porting (Kernel based; Open SQL + Native IQ SQL via ADBC) NLS Interface ODBC Driver from IQ DB Client HANA DB SAP IQ

24 Optimized BW Query-access to NLS via SDA Query BW InfoProvider with NLS Master data HANA SAP IQ Online Query CalcEngine Facts Master data Nearline Query SQL-Engine Virtual Table Archive Facts SELECT "S1"."SID","X1"."S PGRO", SUM ( "O1". AMOUNT" ) FROM Virtual Table" "O1" JOIN /BIC/SCUST" "S1" ON "O1". "CUSTOMER" = "S1". "/BIC/CUST" JOIN "/BIC/XPROD" "X1" ON "O1". "PRODUCT" = "X1". "/BIC/PROD" WHERE GROUP BY "S1"."SID","X1"."S PGRO"

25 SAP IQ AS Information Lifecycle Management (ILM) FOR BUSINESS SUITE

26 ILM Retention Management (RM) on SAP IQ Benefits at a Glance SAP ILM RM SAP + = IQ One point of contact (vendor) One storage infrastructure One archiving strategy Innovation Store and manage your archive index and data on column-based DB Eliminate costly 3 rd party compliant store SW and HW Foundation for next generation ILM Benefits Reduce system landscape complexity Increase performance for archiving and data access Increase search capabilities Optimal data compression WORM-like security (certificates, signatures) Availability Storage of archive information system (AS) indexes (available since Jan 2013 with NetWeaver 7.31 SP07) Storage of archived data in write-once space (available since Dec 2013 with NW 7.40 SP05 and NW 7.31 SP10)

27 Classic Storage in ERP and BW Store ERP DB Archiving / ILM Framework SAP ERP ILM (Information Lifecycle Management) utilizes a third party store for archiving cold data. BW 3 rd Party Store 3 rd Party NLS SAP BW NLS interface utilizes a third party solution to implement a near line store on IQ or another database platform. DB IQ Classic storage in ERP and BW

28 Store ILM Database Storage Option (on SAP IQ) ERP DB ILM Framework 3 rd Party Store ERP DB ILM Framework AS* SAP IQ Arch. Data SAP ERP ILM now owns the complete archiving solution, and stores both archive indexes and archive data in SAP IQ. write-once space BW 3 rd Party NLS DB IQ Classic storage in ERP and BW *archive information structures

29 Store ILM DB Storage combined with NLS storage ERP ERP BW ILM Framework ILM Framework DB DB SAP IQ DB SAP BW NLS** 3 rd Party Store AS * Arch. Data, & BW data BW write-once space 3rd Party NLS With SAP ILM RM on SAP IQ + BW NLS, customers can store both types of data in a single storage environment! DB IQ Classic storage in ERP and BW

30 HANA EXTENDED TABLES

31 Warm and cold data archiving for SAP products As you have seen, SAP IQ plays a key role... NLS (Near Line Store) for SAP BW Optimized NLS data transfer throughput using IQ Loader functionality SAP HANA and IQ share the same columnar paradigm, and similar data compression rates Ready for large data volumes Suitable for ad-hoc queries with long history Minimum administration effort ILM for SAP Business Suite Fast archive index read w/o additional secondary DB indexes Increased search capabilities Faster archive I/O fewer layers (software, network, storage hardware) ERP archive files as well as archive indexes stored in SAP IQ Warm archive for HANA using smart data access (SDA) Using SDA, HANA customers can access IQ as a federated store Store warm data in IQ, and real time data in HANA Push query processing down to IQ

32 SAP HANA Extended Storage with SAP IQ The next wave: transparent integration with SAP HANA... Cool data archival NLS Store for SAP BW ILM for SAP Business Suite Warm archive for HANA, federated via SDA Warm data management Extended storage for SAP HANA Handle all data seamlessly via HANA tables Steady performance across data volumes, variety, velocity with scalable capacity Practical for both on-premise and hosted models Native Big Data solution allows customers to gain real-time insights by cost-effectively managing and analyzing ALL enterprise data

33 HANA EXTENDED STORAGE Why is warm data management important for HANA? What about Hadoop for warm data storage and processing? Why is SAP HANA extended storage the best solution for warm data management? While SAP HANA provides excellent performance for big data applications, size and cost constraints may preclude a HANA-only solution A significant portion of enterprise data is considered warm and does not require the continuous in-memory residence and real-time processing of hot data Keeping warm data segregated allows SAP HANA to perform streamlined processing of only the most critical, hot data Hadoop plays an important role in SAP s big data story, for storing and processing massive volumes of data. SAP has integrated Hadoop into the HANA platform through data movement and federation methods Although Hadoop improves all the time, it does not match the performance characteristics of a columnar store when processing data at a scale that can be accommodated within a DBMS in the case of HANA extended storage, petabyte scale! Hadoop does no data compression, and data expands to at least 3x its raw size! HANA extended storage is not simply a storage tier but a smart column store with superlative data loading, data compression, and query performance characteristics HANA extended storage has been proven to scale capably to petabyte levels, at an economical price point HANA extended storage is exhibiting promising performance already, with additional improvements to come HANA extended storage allows both hot and warm data to be accessed natively through HANA tables and views

34 HANA extended storage requirements Manage data cost effectively, yet with desired performance based on SLAs Application defines which data is hot, and which data is warm Handle very large data sets terabytes to petabytes Update and query all data seamlessly via HANA tables Provide a native Big Data solution that covers most enterprise use cases without Hadoop Queries and updates initiated from HANA Table Table SAP IQ SAP HANA Extended table (access via HANA) (warm/cool data)

35 Extended Storage Configuration and Usage One-time definition of extended storage source server Tables mapped to extended storage using CREATE or ALTER TABLE CREATE EXTENDED STORAGE CONFIGURATION 'Driver=libdbodbc16_r.so;ServerName=hana_iq;CommLinks=tcpip(host=localhost:30015)' WITH CREDENTIAL TYPE 'PASSWORD' USING 'user=dba;password=sql;hanauser=system;hanapassword=sql'; CREATE TABLE table2(c1 integer, c2 integer) USING EXTENDED STORAGE; ALTER TABLE table2 NOT USING EXTENDED STORAGE; ALTER TABLE table2 USING EXTENDED STORAGE ENABLE DELTA; --- Rest is SQL as usual --- INSERT INTO table2(c1 c2) VALUES(4, 7); SELECT C1 FROM table2 WHERE C2 < 10; ALTER TABLE table2 add C3 int; DROP TABLE table2;

36 Extended Storage Optimizer Query Decomposition with Forwarded Search Arguments (SARG) HANA Connection Original Query HANA Instance Extended Storage Optimizer HANA Table <dim> HANA Table Query Fragment SELECT fact.c1 FROM fact, dim WHERE fact.c2=dim.c2 AND dim.c1<10 SELECT c2 FROM dim where dim.c1<10 IQ Instance Temp Table IQ Query Engine IQ Table <fact>

37 HANA Studio Integrated Query Plans Double- Click

38 Extended storage in SAP BW on HANA Optimized persistence for staging and corporate memory Staging Area Corporate Memory BW DataSources and writeoptimized DSOs can have the DataSource wo-dso DataSource wo-dso property Extended Table HANA Write and read operations are re-directed to IQ IQ PSA Table Corporate Memory Table RTDP All BW standard operations are supported no changes Only minor temporary RAM required in HANA to support extended tables IQ Table IQ Table

39 KEY LEARNINGS

40 KEY LEARNINGS The SAP HANA platform is a cohesive and integrated set of technologies for enterprise data management and applications. The SAP HANA platform delivers federated access to information with its Hana-powered In-Memory Data Fabric. SAP IQ has become the preferred technology for warm/cool data management and high volume scenarios within the SAP HANA In-Memory Data Fabric.

41 THANK YOU THANK YOU FOR PARTICIPATING Please provide feedback on this session by completing a short survey via the event mobile application. SESSION CODE: 0310 For ongoing education on this area of focus, visit