Datenmanagement für SAP Applikationen. Rudolf Munz SAP AG

Transcription

1 Datenmanagement für SAP Applikationen Rudolf Munz SAP AG

2 Agenda SAP Architecture Table Buffer Content Server livecache BI Accelerator Summary Special Data Containers Future DBMS Requirements Summary SAP AG 2007, Datenmanagement für SAP Applikationen / Rudolf Munz / 2

4 1992: SAP Introduces the 3-tier Architecture Frontend WAN-enabled, few roundtrips, data volume < 10 KB Application Server Database Server LAN required, many roundtrips, data volume about 20 KB SAP AG 2007, Datenmanagement für SAP Applikationen / Rudolf Munz / 4

5 SAP Application Server Scalability Frontends Application Server... Scalability... Database Server SAP AG 2007, Datenmanagement für SAP Applikationen / Rudolf Munz / 5

6 SAP Philosophies and Successes First in real time applications First in application integration via single shared database First in Unix and Windows First in SQL DBMS First in Graphical User Interfaces for commercial applications First in Virtual Machine concepts SAP AG 2007, Datenmanagement für SAP Applikationen / Rudolf Munz / 6

7 Size of SAP ERP Data Model (Part of Business Suite 05) tables columns views indexes rows (initial size) 57 GB disk footprint (initial size) 270 millions lines of code SAP AG 2007, Datenmanagement für SAP Applikationen / Rudolf Munz / 7

8 Published Results for SD Benchmarks SD Benchmark (three-tier): Highest number of users 200, , , , , ,000 80,000 Number of SD Benchmark Users 168, ,000 60,000 47,528 40,000 20, ,400 14,400 6,030 19,360 26,000 SAP AG 2007, Datenmanagement für SAP Applikationen / Rudolf Munz / 8

9 Typical OLTP CPU Load Distribution Rel. CPU-Usage / Dialog Step Platform & Release dependent 68% 82% 79% 81% 10% 84% 8% 9% 7% 7% 22% 8% 9% 14% 11% FI SD PP MM ATO SAP Standard Benchmarks + = Application Server = Database Server SAP AG 2007, Datenmanagement für SAP Applikationen / Rudolf Munz / 9

11 Table Buffer Design Rationale Optimize read performance of stable or nearly stable data Granules are tables or primary key ranges Only primary key or key prefix accesses supported Stored in shared memory of application server Replicated in all application servers No transactional consistency for data in table buffer Invalidation and refresh of buffered tables and key ranges Async propagation of changes to other application servers SAP AG 2007, Datenmanagement für SAP Applikationen / Rudolf Munz / 11

12 Table Buffer in Application Server Application Server Database Server Work Process DB Interface Key Table Buffer DBMS SELECT * FROM... Open SQL SQL Data SQL Data Native SQL DB Open SQL Catalog SQL Data SAP AG 2007, Datenmanagement für SAP Applikationen / Rudolf Munz / 12

13 Performance of Table Buffer vs. DBMS (PK Access) Local DBMS ABAP VM Table Buffer µs SAP AG 2007, Datenmanagement für SAP Applikationen / Rudolf Munz / 13

14 Typical OLTP Traffic Distribution Application Server Database Server Work Process DB Interface Key Table Buffer DBMS SELECT * FROM... 80% SQL Data 20% Native SQL 98% SQL Data Cache 2% Disks 70% reads 80% primary key 20% medium complex 30% writes SAP AG 2007, Datenmanagement für SAP Applikationen / Rudolf Munz / 14

16 Content Server Design Rationale Documents are attachments to SAP Business Objects Separation of document content from OLTP data Read and write traffic of documents offloaded from OLTP database Insert/update/delete of documents not recorded in OLTP database log Improved cache utilization in OLTP database server Reduced size of OLTP database Direct content delivery to SAP Frontend (Viewer) Application server sessions use two database sessions, dual session support in database abstraction layer OLTP DBMS Content Server (based on MaxDB) SAP AG 2007, Datenmanagement für SAP Applikationen / Rudolf Munz / 16

17 Content Server Architecture Frontends Ap plication Server Content Server Database Server SAP AG 2007, Datenmanagement für SAP Applikationen / Rudolf Munz / 17

18 Content Server Consistency Documents are never updated, Update = delete + insert References to documents (DocID) stored in OLTP data Two-phase commit is avoided by write discipline 1. Insert new document into Content Server and commit 2. Insert or update Business Object in OLTP DBMS and commit If step 2 succeeds, we are done Normal case If step 1 or 2 fail, garbage in the Content Server may be left Exception Steps can be repeated SAP AG 2007, Datenmanagement für SAP Applikationen / Rudolf Munz / 18

20 livecache Features Main memory-based object management system (OMS) Persistent and shared C++ objects Favors modeling of tree- or network-like complex object structures Mainly used for SAP s supply chain management (planning & optimization) Data-intensive application logic is executed as Stored Procedures Application coding and data management in same address space Navigations on shared data are nearly as fast as on private data Navigations are 50 to 100 times faster compared to SQL Reader isolation with respect to concurrent writers (multi-version read) Writer isolation with respect to concurrent writers (versioning) SAP AG 2007, Datenmanagement für SAP Applikationen / Rudolf Munz / 20

21 livecache = MaxDB + MoreDB Applications livecache Applications MaxDB (SQL) MoreDB (OMS) OMS SAP AG 2007, Datenmanagement für SAP Applikationen / Rudolf Munz / 21

22 livecache Architecture Application Server ABAP Applications SQL Packets livecache Applications Stored Procedures in C++ livecache Server SQL OMS Object Management System Record & Page Manager SAP AG 2007, Datenmanagement für SAP Applikationen / Rudolf Munz / 22

23 Shared and Private Data in livecache Session A Session B Session C Session D Transient Transient Transient Transient C++ C++ C++ C++ Objects Object Cache Objects Object Cache Objects Object Cache Objects Object Cache Private Data (Heap) Data Cache Main-memory Database (Persistent C++ Objects) Shared Data SAP AG 2007, Datenmanagement für SAP Applikationen / Rudolf Munz / 23

24 Navigational Performance (in µs) SQL Key SQL Key SP OMS Key OMS OID Object Cache C++ Pointer SAP AG 2007, Datenmanagement für SAP Applikationen / Rudolf Munz / 24

25 livecache Statistics (Customer Scenario) Database size (in GB) 110 Transactions / sec 33 livecache roundtrips / sec Object reads / sec Object writes / sec Log in KB / sec 660 Log in pages /sec 82 Average load in a 24x7 environment Peak load is factor 2 higher SAP AG 2007, Datenmanagement für SAP Applikationen / Rudolf Munz / 25

27 Business Intelligence Waves Wave 1: Reporting is part of the OLTP system (past) Unpredictable query load Reporting on flat OLTP tables No support of multi-dimensional data, no OLAP, no history Wave 2: Dedicated Data Warehouse (now) Separation of OLTP systems and Data Warehouse (DWH) Periodic extracts of OLTP data to DWH (ETL) OLAP on multi-dimensional data, history Wave 3: Realtime Analytics (future) Separation of OLTP and OLAP systems Transactional consistency between OLTP and OLAP data Driver: SOA and BI functionality as part of transactional applications SAP AG 2007, Datenmanagement für SAP Applikationen / Rudolf Munz / 27

28 BI Accelerator Query Performance Booster BI Tools or Applications Storage on disk SAP NetWeaver BI Query processing in main memory DBMS BI Accelerator SAP AG 2007, Datenmanagement für SAP Applikationen / Rudolf Munz / 28

29 Column-Wise Storage OLTP DBMS store tables row-wise BI Accelerator stores tables column-wise Row1 Row2 Att 1 Col1 Col2 Att 2 Att 2... SAP AG 2007, Datenmanagement für SAP Applikationen / Rudolf Munz / 29

30 Data Compression Column-wise Storage Dictionary Sorted array of all used values Values stored with front compression Column values Array of dictionary indexes Minimal number of bits used to represent values Column Dictionary ValueID Value 1 IBM 2 Microsoft Compression rate Factor 3-6 RowID ValueID 1 17 SAP 17 SAP Press 19 SAP SI Column Values SAP AG 2007, Datenmanagement für SAP Applikationen / Rudolf Munz / 30

31 Partitioning of Columns into Main Memory of Blades Columns Part 1 Part 2... Fact Table Fact Table Fact Table Part 1 Part 2... Part N Part N Column Part 1 Columns Part 2 Columns Part... Columns Part N Blade Server Blade Server Blade Server Blade Server Column Storage SAP AG 2007, Datenmanagement für SAP Applikationen / Rudolf Munz / 31

32 Incremental Data Loads Queries Data Loads BI Accelerator Engine Delta Part Supports fast loads Queries run against both parts Static Part Delta Part Holds data until they are merged Fast merge Merge of delta part triggered by Size Schedule Manually SAP AG 2007, Datenmanagement für SAP Applikationen / Rudolf Munz / 32

33 Query Execution Times (Sample Queries) Query DBMS (sec) BI Accelerator (sec) Improvement (factor) Rows after filtering Rows after aggregation Query Query Query Query Query Query Query Query Query Query Query Query Query Query SAP AG 2007, Datenmanagement für SAP Applikationen / Rudolf Munz / 33

35 Summary: Special Data Containers OLTP data Transactional SQL engine Row-wise storage Additional caching of stable data in main memory of application server OLAP data SQL engine without transactions and logging Column-wise storage Main memory database MPP approach (data partitioning into blades) Objects (C++, Java, ABAP) Transactional Object Management System Main memory database Data-intensive application logic executed as Stored Procedures Alternative to OLTP data management based on SQL Documents SQL DBMS Document repository Documents attached to Business Objects (DocID in OLTP data) Separation of document and OLTP workload SAP AG 2007, Datenmanagement für SAP Applikationen / Rudolf Munz / 35

36 Scenario: RFID-Tagged Products in a Supply Chain Events Products with an RFID tag are moved in a global supply chain All movements are detected as an RFID event by RFID sensors Customer specific RFID tag and event data (e. g. XML) Extremely high volume of RFIDs and associated RFID events Customer scenario 1KB / event 120 TB / year = 337 GB / day = 14 GB / hour = 3.9 MB / sec (best case, 10x peaks) Challenging write workload Challenging read workload with search capabilities on all attributes Distributed event capturing, storing, and retrieval Infrastructure to capture, store, and retrieve (RFID) events? SAP AG 2007, Datenmanagement für SAP Applikationen / Rudolf Munz / 36

38 Invisible DBMS Many DBMS instances in embedded systems Many DBMS instances in an enterprise IT landscape Central administration Central user and role management We will run short of DBAs DBA-free operation required Self-management = self-tuning + self-administration Implicit reorganization Implicit Update Statistics Implicit index tuning - Determine unused indexes - Propose / create useful indexes Workload analysis and adaptation (self-tuning) Online everything for automatic configuration changes Adapt to new system quotas SAP AG 2007, Datenmanagement für SAP Applikationen / Rudolf Munz / 38

39 Continuous Availability High Availability configurations are in place Protection against unplanned downtimes (hardware, system software, desaster) Hot-standby with failover from master to slave (OS cluster) Customers want business continuity 24 x 365 instead of 24 x 7 Applications facing customers or partners Think of Google, Ebay, Amazon,... in the consumer space Continuous Availability addresses planned downtime Configuration changes ( online everything ) Patches for the current release (rolling patch services in a cluster) Upgrades to the next application release (to be solved) Migration to new application coding and an extended persistency layer Old and new persistency layer run in parallel (during upgrade phase) Changes get propagated from old to new persistency layer Application server and DBMS involved SAP AG 2007, Datenmanagement für SAP Applikationen / Rudolf Munz / 39

40 Unlimited Scalability Nearly unlimited main memory Active part of an OLTP database can be kept in main memory Effects of all open transactions can be kept in main memory Optimization potential: dedicated main memory data structures vs. serializable data structures for logging and checkpointing Nearly unlimited CPU power (multi-cores) Eliminate reader/writer synchronization (= multi-version concurrency control) Reduce low-level writer/writer synchronization Context switches are expensive and should be avoided Fast synchronization techniques (compare and swap) Differentiate between safe and unsafe phases of changes Differentiate between extensions and structural changes Look for collision-free algorithms Cache misses determine CPU performance (level 2 cache vs. main memory) Locality of coding (profile-based optimization) Locality of main memory data (e. g. for scanning) SAP AG 2007, Datenmanagement für SAP Applikationen / Rudolf Munz / 40

41 Tenant-aware DBMS New software delivery model: Software-as-a-Service Hosted applications for many (small) tenants Interesting for small and medium enterprises with little IT skills All tenants run the same application, data are tenant-specific Data isolation between tenants is a must DBMS instance per tenant is too expensive (admin and system costs) Solution 1: Tenant-aware data model Requires discipline in application development and additional QA efforts No programming access to the persistency layer by tenants Solution 2: Tenant-aware DBMS Implicit tenant-enabling by the DBMS: DBMS virtualization Resource sharing for caches, data volumes, log Each tenants gets an own (virtual) DBMS instance No changes in the application coding Allows for tenant-specific extensions SAP AG 2007, Datenmanagement für SAP Applikationen / Rudolf Munz / 41

43 Summary Table Buffer Content Server livecache BI Accelerator? Specialized data containers for Stable OLTP data Documents Objects OLAP data Events Invisible DBMS Continuous Availability Unlimited Scalability Tenant-aware DBMS New operational requirements Eliminate DBMS administration Always up Exploit hardware trends Virtualization at the DBMS level SAP AG 2007, Datenmanagement für SAP Applikationen / Rudolf Munz / 43

44 Copyright 2007 SAP AG. All Rights Reserved No part of this publication may be reproduced or transmitted in any form or for any purpose without the express permission of SAP AG. The information contained herein may be changed without prior notice. Some software products marketed by SAP AG and its distributors contain proprietary software components of other software vendors. Microsoft, Windows, Excel, Outlook, and PowerPoint are registered trademarks of Microsoft Corporation. IBM, DB2, DB2 Universal Database, OS/2, Parallel Sysplex, MVS/ESA, AIX, S/390, AS/400, OS/390, OS/400, iseries, pseries, xseries, zseries, System i, System i5, System p, System p5, System x, System z, System z9, z/os, AFP, Intelligent Miner, WebSphere, Netfinity, Tivoli, Informix, i5/os, POWER, POWER5, POWER5+, OpenPower and PowerPC are trademarks or registered trademarks of IBM Corporation. Adobe, the Adobe logo, Acrobat, PostScript, and Reader are either trademarks or registered trademarks of Adobe Systems Incorporated in the United States and/or other countries. Oracle is a registered trademark of Oracle Corporation. UNIX, X/Open, OSF/1, and Motif are registered trademarks of the Open Group. Citrix, ICA, Program Neighborhood, MetaFrame, WinFrame, VideoFrame, and MultiWin are trademarks or registered trademarks of Citrix Systems, Inc. HTML, XML, XHTML and W3C are trademarks or registered trademarks of W3C, World Wide Web Consortium, Massachusetts Institute of Technology. Java is a registered trademark of Sun Microsystems, Inc. JavaScript is a registered trademark of Sun Microsystems, Inc., used under license for technology invented and implemented by Netscape. MaxDB is a trademark of MySQL AB, Sweden. SAP, R/3, mysap, mysap.com, xapps, xapp, SAP NetWeaver, and other SAP products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of SAP AG in Germany and in several other countries all over the world. All other product and service names mentioned are the trademarks of their respective companies. Data contained in this document serves informational purposes only. National product specifications may vary. The information in this document is proprietary to SAP. No part of this document may be reproduced, copied, or transmitted in any form or for any purpose without the express prior written permission of SAP AG. This document is a preliminary version and not subject to your license agreement or any other agreement with SAP. This document contains only intended strategies, developments, and functionalities of the SAP product and is not intended to be binding upon SAP to any particular course of business, product strategy, and/or development. Please note that this document is subject to change and may be changed by SAP at any time without notice. SAP assumes no responsibility for errors or omissions in this document. SAP does not warrant the accuracy or completeness of the information, text, graphics, links, or other items contained within this material. This document is provided without a warranty of any kind, either express or implied, including but not limited to the implied warranties of merchantability, fitness for a particular purpose, or non-infringement. SAP shall have no liability for damages of any kind including without limitation direct, special, indirect, or consequential damages that may result from the use of these materials. This limitation shall not apply in cases of intent or gross negligence. The statutory liability for personal injury and defective products is not affected. SAP has no control over the information that you may access through the use of hot links contained in these materials and does not endorse your use of third-party Web pages nor provide any warranty whatsoever relating to third-party Web pages. SAP AG 2007, Datenmanagement für SAP Applikationen / Rudolf Munz / 44