SAP BW Near-line Storage (NLS)

Transcription

1 SAP BW Near-line Storage (NLS) Dr. Michael Hahne Hahne Consulting GmbH February 27, 2013

2 Disclaimer This presentation outlines SAP s general product direction and should not be relied on in making a purchase decision. This presentation is not subject to any license agreement or any other agreement with SAP. SAP has no obligation to pursue any course of business outlined in this presentation or to develop or release any functionality mentioned in this presentation. This presentation and SAP's strategy and possible future developments are subject to change and may be changed by SAP at any time for any reason without notice. This document is provided without a warranty of any kind, either express or implied, including but not limited to, the implied warranties of merchantability, fitness for a particular purpose, or non-infringement. SAP assumes no responsibility for errors or omissions in this document, except if such damages were caused by SAP intentionally or grossly negligent Copyright Hahne Consulting GmbH SAP AG. All rights reserved. 2

3 Agenda SAP BW Near-line Storage (NLS) n What is Near-line Storage (NLS)? n History, Strategy, Definition, Differentations n What is the NLS Interface in SAP Business Warehouse (BW)? n Architecture, Functionality, Modeling Objects, Designtime vs. Runtime... n What is an Implementation for the SAP BW NLS Interface? n Architecture, Partner Solutions, Differentiators, License Model... n What s new with the SAP owned BW NLS Implementation for Sybase IQ? n Benefits, Realization, Availability, Piloting, Licensing, Roadmap Copyright Hahne Consulting GmbH SAP AG. All rights reserved. 3

4 Data-Aging Strategies resp. Information Life Cycle Definition of Near-line Storage (NLS) for SAP BW Storage Type / Data Category Online Database Near-Line Storage (read only) Classic Archive (read only) Frequently read / changed data (actual) Infrequently read data (mature) Very rarely read data Near-line Storage (NLS) n alternative storage type with SQL based direct access capabilities for reporting, ETL, and look-ups n extracted NLS partitions are deleted in RDBMS (NLS storage and Online Storage together consistently reflect the BW data persistency of an InfoProvider) n NLS storage is much more cheaper than DB storage n NLS partitions of an InfoProvider are write-protected (!) n NLS is primarily partitioned by time/age (aged) Copyright Hahne Consulting GmbH SAP AG. All rights reserved. 4

5 Design Aspects for a Classic BW Environment Near-line Storage (NLS) vs. BW Accelerator (BWA) BW InfoMarts (InfoCube) ADK Archive BWA Acceleration Archiving Near-line Storage Acquisition RDBMS Access - very frequently frequently not frequently rarely Copyright Hahne Consulting GmbH SAP AG. All rights reserved. 5

6 Multi Temperature Data Data volume Performance hot Data is read and/or written frequently In memory No restrictions, all features available warm Non-Active Data Concept Infrequent access On disk, no need to keep in memory all the time No restrictions, all features available cold NLS Management for readonly Sporadic access data Not stored in HANA DB; stored in Near-line Storage Restricted to NLS capabilities Providing lower TCO by optimized data volume management Copyright Hahne Consulting GmbH SAP AG. All rights reserved. 6

7 Near-line Storage Implementation Challenges n Analysis and Reporting are operating on a combination of Online-, and Near-line data whereby data consistency is an indispensable requirement. n Transition processes touching several storage types have to guarantee consistency: n Archiving and Deleting of online data has to be one Logical Unit of Work (LUW). n Rollback mechanisms for single archiving steps have to be available. n The Archive is getting the attidude of a database. For this reason the processing of an Data Archiving Process (DAP) is structured in different phases: Copyright Hahne Consulting GmbH SAP AG. All rights reserved. 7

8 Agenda SAP BW Near-line Storage (NLS) n What is Near-line Storage (NLS)? n History, Strategy, Definition, Differentations n What is the NLS Interface in SAP Business Warehouse (BW)? n Architecture, Functionality, Modeling Objects, Designtime vs. Runtime... n What is an Implementation for the SAP BW NLS Interface? n Architecture, Partner Solutions, Differentiators, License Model... n What s new with the SAP owned BW NLS Implementation for Sybase IQ? n Benefits, Realization, Availability, Piloting, Licensing, Roadmap Copyright Hahne Consulting GmbH SAP AG. All rights reserved. 8

9 NLS Interface Design and Runtime Objects BW InfoProvider NLS - Interface DAP Design Time n Define partitioning criteria n Generate DAP Data Archiving Process (DAP) DB DBMS Younger Q1/2011 Q4/2010 Q3/2010 DAP Scheduling / Monitoring / Run- Time: n Describe partition to be extracted n Generate request n InfoProvider Manage Menu n Process Chain n Archive / Extract (DAP menu, Process Chain n Restore (DAP menu) NLS Implementation Q2/2010 Q1/2010 Add. Features n Query access to NLS n DTP access to NLS n Look up during Transforms Copyright Hahne Consulting GmbH SAP AG. All rights reserved. 9

10 Data Management with Near-line Storage Implementation Aspects Create a Data Archiving Process Create and schedule archiving requests Restore archiving requests Load data to subsequent Data Targets 5 n Look-up during Transformation 6 n Query Settings n 7 Multiprovider Settings Reporting Layer SAP Sales InfoCube (Architected Data Marts) 6 Data Propagation Layer Nearline Storage Data Acquisition Layer DTP DTP DTP InfoPackage MultiProvider 7 Corporate Memory PSA 4 5 DTP InfoSource DataSource 2 3 DAP Nearline Storage Nearline Storage 1 LSA Copyright Hahne Consulting GmbH SAP AG. All rights reserved. 10

11 SAP BW - Key facts about NLS Interface n NLS should be a part of an Information Lifecycle Management (ILM) strategy n Based on well-established SAP / SAP BW archiving concepts n Data consistency guaranteed before deleting the data from source n Data archived in NLS can be incorporated into reporting n Saves storage costs and other system resources n High compression rate (up to 95% depending on implementation) n Increases retention period for analysis data n Included in the query statistic data collection (RSRT) n NLS Interface Features n Supports archiving of InfoCubes and DataStore Objects n Mainly time-based archiving, yet can also be based on other characteristics n Lock of the archived data slice in the original InfoProviders n NLS Process Steps in Process Chain Handling n Copes with changes in the meta data to the BW objects of the archived data n Deletion of NLS partitions supported n Direct load from ADK archive files into NLS supported Copyright Hahne Consulting GmbH SAP AG. All rights reserved. 11

12 Advantages of a Data Management Strategy Performance Faster Data Loads Faster Queries Costs Less storage costs Less administrative effort controlled system growth Availability of data: faster rollups and change runs etc. of system: less downtime for backups, upgrades etc. Enterprise Data Warehousing Data Marts on Demand : more flexibility, faster rollout SAP AG. All rights reserved. Improve Performance and lower Costs for SAP BW 12

13 Agenda SAP BW Near-line Storage (NLS) n What is Near-line Storage (NLS)? n History, Strategy, Definition, Differentations n What is the NLS Interface in SAP Business Warehouse (BW)? n Architecture, Functionality, Modeling Objects, Designtime vs. Runtime... n What is an Implementation for the SAP BW NLS Interface? n Architecture, Partner Solutions, Differentiators, License Model... n What s new with the SAP owned BW NLS Implementation for Sybase IQ? n Benefits, Realization, Availability, Piloting, Licensing, Roadmap Copyright Hahne Consulting GmbH SAP AG. All rights reserved. 13

14 The Near-line Storage Solution for SAP BW Based on the NearLine Storage Interface Development Partners can implement their Solutions for Archiving and NLS into SAP BW 3rd Party NLS Solutions are implemented within the SAP BW ABAP Stack in partner specific namespaces have to pass a certification process can offer specific Application Area in the SAP Support Portal have to be licensed in addition to SAP licenses can have a different release cycle compared to SAP NetWeaver BW NLS Partner Solution Copyright Hahne Consulting GmbH SAP AG. All rights reserved. 14

15 SAP Defined Certification Category: Nearline Storage Certified Partners (January 2013) (see also ) Copyright Hahne Consulting GmbH SAP AG. All rights reserved. 15

16 Informatica ILM Nearline for SAP BW (Former SAND Nearline) Reduce production storage costs by moving infrequently accessed data to a nearline data store. Does not require any database administration. Data is stored in encrypted files on standard file system, simplifying backup and restore processes as well as replication to other environments. Reduce non-compliance risk with continued access to data in immutable, compressed files - compression rates as high as 98%, with average of 90%+ compression rate. Contact: rsobral@informatica.com Copyright Hahne Consulting GmbH SAP AG. All rights reserved. 16

17 SAP NetWeaver BW Nearline Storage on DataVard OutBoard DataVard s OutBoard is the all-in-sap nearline storage solution Built in ABAP, runs in your existing SAP landscape, no additional database or hardware required OutBoard is priced per productive system landscape. No matter how much data you load onto it. Compresses all InfoProviders, PSA and Changelogs up to 95% Contact: outboard@datavard.com SAP cluster tables Business Warehouse OutBoard All SAP-certified RDBMS Big data ready (e.g. Apache Hadoop) File Apache Hadoop Corporate Cloud External / Remote DB File / Cloud Storage Mgmt. OutBoard Near-Line Storage SAP NLS Interface HANA or DB End of lifecycle Deletion OutBoard Storage Management Copyright Hahne Consulting GmbH SAP AG. All rights reserved. 17

18 SAP NetWeaver BW Nearline Storage on PBS CBW With CBW PBS provides an add-on solution for the data retrieval of archived/nearlined BW data. On the basis of the SAP ADK files, the CBW solution creates aggregates on the archive data and (optionally, if needed) index structures to enable a fast retrieval option for the archived data directly from the end-user's original query. PBS CBW NLS IQ for Sybase Analytics Server IQ utilizes the Sybase Analytics Server IQ as a nearline database and offers extremely fast query response times. Contact: klaus.zimmer@pbs-software.com Copyright Hahne Consulting GmbH SAP AG. All rights reserved. 18

19 SAP NetWeaver BW Nearline Storage on IBM DB2 Takes full advantage of DB2 compression Fully exploits DB2 s self managing capabilities Included in SAP DB2 OEM licnese at no extra cost SAP NLS (Near Online Storage) solution for DB2 has been downported to NW 7.0 EhP1 and will be released for SAP customers with SPS06 end The solution is also contained in NW 7.0 EhP2 and NW 7.20 Contact: KARL.FLECKENSTEIN@DE.IBM.COM Install Near-line Storage data base Copyright Hahne Consulting GmbH SAP AG. All rights reserved. 19

20 SAP owned MaxDB based Sample Solution of NLS Demo and verification implementation of the interface SAP BW Not for productive use MaxDB schema for user SAPR3 with NLS objects Because the implemention of the Nearline Interface is completely delegated to partners a test and reference implementation is necessary. See SAP Notes and (with attachment specifying the NLS API) Table structure Copyright Hahne Consulting GmbH SAP AG. All rights reserved. 20

21 Overview SAP BW NLS Partner Solutions n Different storage types supported n BW Database inbound n Underlying BW DB (from Datavard) n Databases n n n n n n n File System n n IBM DB6 (from IBM) Sybase IQ (from PBS) Sybase IQ (from DataVard) MaxDB ( for SAP s T&E implementation) Informatica Nearline (former SAND DNA) (from Informatica) Apache Hadoop (from Datavard) ADK storage systems (from PBS) File or Apache Hadoop (from Datavard) n Different license models n # of systems based n Volume based n User based n Mixed approach Copyright Hahne Consulting GmbH SAP AG. All rights reserved. 21

22 Data analysis and assistance for ROI analysis n Sizing of NearLine-Storage solutions: n Hardware sizing of the NearLine-Storage solution has to be done by the vendor n n Different NearLine-Storage technologies on the market From database solutions, to file-based solutions, to column-based storage solutions n System Analysis e.g. with DataVard Fitnesstest n Deliver a thorough analysis of BW objects distribution n Can help on estimating the data volume that may be archived / transferred to NLS for the largest InfoProviders within the system n Considers only technical facts (and not the customer s business requirements ) Copyright Hahne Consulting GmbH SAP AG. All rights reserved. 22

23 BWFT Result: Best case 1,7 TB savings OutBoard Other data ODS data Temporary data Cube data Master data Heute mit OutBoard und ERNA Copyright Hahne Consulting GmbH SAP AG. All rights reserved. 23

24 ERNA - Features Overview Scope of Housekeeping Application (e.g. BW) System (Netweaver) n PSAs n Change Logs n Request logs & tables (RSMON* and RS*DONE) n Unused DIMIDs, Unused SIDs n Unused Masterdata n Cube & Aggregate compression n NRIV buffering n Table buffering n BI-Statistics n Process Chain Log n Errorlog for DTPs (RSBERRORLOG) n... n Application log n Batch log n IDoc tables (EDI40, EDIDS) n qrfc, trfc n Job-Tables (TBTCO, TBTCP etc.) n Change & Transportsystem n Spool data (TST03) n Table Change Protocols n Batch Input Folders n Alert Management Data (SALRT*) n... Central access for scheduling and monitoring Process Automation Automated recommendations Copyright Hahne Consulting GmbH SAP AG. All rights reserved. 24

25 Agenda SAP BW Near-line Storage (NLS) n What is Near-line Storage (NLS)? n History, Strategy, Definition, Differentations n What is the NLS Interface in SAP Business Warehouse (BW)? n Architecture, Functionality, Modeling Objects, Designtime vs. Runtime... n What is an Implementation for the SAP BW NLS Interface? n Architecture, Partner Solutions, Differentiators, License Model... n What s new with the SAP owned BW NLS Implementation for Sybase IQ? n Benefits, Realization, Availability, Piloting, Licensing, Roadmap Copyright Hahne Consulting GmbH SAP AG. All rights reserved. 25

26 Motivation Main aspects of NLS Separation of frequently accessed (hot) data from less frequently accessed (cold) data keeping the HANA in memory costs in a reasonable range Complementary to active / non-active classification within HANA Data in NLS is highly compressed in cost-efficient storage with fewer backups reducing operational costs Copyright Hahne Consulting GmbH SAP AG. All rights reserved. 26

27 Design Aspects for BW powered by HANA Near-line Storage (NLS) vs. SAP HANA DB BW on HANA RAM Temporary data data volume performance Active Data in HANA DB no restrictions, all features available hot warm Displacement Strategy Processing data Filer Persistent data active + not-active partitions in HANA no restrictions, all features available (mostly) not-active data in HANA no restrictions, all features available cold Sybase NLS IQ Persistent data NLS in Sybase IQ restricted to [BW] NLS capabilities Copyright Hahne Consulting GmbH SAP AG. All rights reserved. 27

28 SAP BW powered by HANA and Sybase IQ NLS Architecture Overview BI Clients SAP Netweaver BW 7.3x An SAP owned BW NLS Implementation for Sybase IQ offers a fully integrated solution from one provider MultiProvider SAP HANA Transient Provider PBS OEM InfoCube/DSO Near-Line SDK Sybase IQ BW NLS4IQ SAP Native new Main aspects Deliver an ABAP based implementation of the BW NLS interfaces Deliver an Sybase IQ DBSL light that covers all needs for the above mentioned NLS implementation Sybase IQ to deliver reliable and performing execution of the DBSL driven loads and queries Planned availability Q1/2013 No longer required Copyright Hahne Consulting GmbH SAP AG. All rights reserved. 28

29 BW NLS Implementation for Sybase IQ (DBSL based solution) BW BW Porting InfoProvider Nearline Adapter (ABAP based ) General Porting DBSL for Sybase ASE (/w IQ ext.) (Kernel based; Open SQL + Native IQ SQL via ADBC) NLS Interface ODBC Driver from IQ DB Client HANA DB Copyright Hahne Consulting GmbH SAP AG. All rights reserved. 29

30 Preliminary Results from ongoing Performance Tests Environment BW 7.30 on HANA Rev. 42 (Online DB) IBM M-Box Intel Xeon X Cores + HT 2,27 GHz 512 GByte RAM Sybase IQ 15.4 (Near-line Storage) Nehalem BWA-Blade Intel Xeon X Cores + HT 2,93 GHz 72 GByte RAM Setup SPO partioned by 10 InfoCubes with 10 Mio. rows each Single target table on IQ side partitioned by Archiving Request (i.e. Time-/Dataslice) Results 64 Mio. rows/h (26 GByte/h) for single relocation run (1 Cube) 238 Mio. Rows/h (97 GByte/h) for 4-fold parallel relocation run (4 Cubes) Still potential for improvements in single process mode and scalability (e.g. IQ 16.x) Copyright Hahne Consulting GmbH SAP AG. All rights reserved. 30

31 Example for 3(4)-fold Archiving Process on single Object Copyright Hahne Consulting GmbH SAP AG. All rights reserved. 31

32 Further Information SAP Public Web SAP Public Web SAP NetWeaver Business Warehouse SCN Space BW Powered by HANA Overview on saphana.com SAP NetWeaver BW 7.3 powered by SAP HANA Release Notes on SAP Help SAP NetWeaver BW 7.3 Documentation Section Using the SAP HANA Database 81F38183CF46B9B919F82D7BB7EC47/frameset.htm Copyright Hahne Consulting GmbH SAP AG. All rights reserved. 32

33 How to get started Analysis Workshop / Fitnesstest Objective: identify Quick Wins / low hanging fruits Benchmarking and Recommendations in areas of System Performance and Robustness Data Management Housekeeping ROI in a couple of months Proof of Concept Test installation Proof of Concept Project and usage Analysis and design Implementation Operation Check-up with Fitnesstest Copyright Hahne Consulting GmbH SAP AG. All rights reserved. 33

34 Thank You! Hahne Consulting GmbH Dr. Michael Hahne Managing Director