Database Platforms Best Practices and What s New Darrell Gosden Principal Consultant SAP Global Database and Technology Services
Speaker Introduction Darrell Gosden My Background Speaker The Database SAP Introduction Real-time Platforms Data Best Platform Practices (RTDP) and What s New Darrell The Outline Vision Gosden Agenda My Background Long term Sybase Principal Consultant (>20 Years) prior to SAP acquisition (in Oct 2012 in the UK). Project / Customer Engagement Delivery Financial Markets Bias. Relational Database / Data Management (including Oracle and Ingres) and associated technologies. SAP Sybase Database Technology Specialist with core skill set in SAP Sybase ASE, SAP Sybase IQ, SAP Sybase Replication Server and associated design / operational management products & tools. Contact Details: Darrell Gosden SCN Entry or you can always find me through LinkedIn! Mobile: +44 (0) 7977257074 E-Mail: darrell.gosden@sap.com Public SAP Community Blog(s) or search Darrell Gosden Running SAP Sybase Virtualized : Running SAP Sybase Data Management Products in a Virtualised World 2
Database Platforms Best Practices and What s New Outline Agenda Database Platforms Best Practices and What s New Outline Agenda 1. The SAP Real-time Data Platform (RDTP) Vision explained. 2. SAP Sybase Adaptive Server Enterprise (ASE) What is it? 3. Migrating to SAP Sybase ASE. 4. ASE Customer Success Stories. 5. SAP Sybase IQ (IQ) What is it / What s New? 6. IQ Customer Success Stories. 7. The SAP Sybase IQ Near-line Storage (NLS) Solution Overview. 8. SAP Sybase IQ NLS The Rapid Deployment Service (RDS). Please feel free to ask questions throughout the session. 3
The SAP Real-time Platform (RDTP) The Vision The Database SAP Real-time Platforms Data Best Platform Practices (RTDP) and What s New The Outline Vision Agenda 4
The SAP Real-time Platform (RDTP) Where does SAP Sybase ASE fit in? The Database SAP Real-Time Platforms Data Best Platform Practices (RTDP) and What s New Where Outline does Agenda SAP Sybase ASE fit in? 5
SAP SAP Database Sybase Sybase ASE Platforms ASE Best Practices and What s New Designed for mission-critical transactional environments Designed Outline Agenda for mission-critical transactional environments Performance Efficiency Usability Easily scale the amount of users and data Execute millions of transactions per minute with terabyte-sized databases Support rapid growth in data volumes and transaction levels Reliable and responsive even under the most punishing workloads More efficient use of servers and storage Improve efficiency by reducing administrative tasks Easy to maintain, minimized downtime Automated tools for monitoring and diagnostics Reduce operational risk by ensuring data security and system stability 6
SAP Sybase ASE A Record of Innovation Sybase IDC Database Finds ASE Platforms ASE s A Quick TCO to Best History be Practices 28% Less and Than What s Other New RDBMS Outline Agenda Heterogeneous Data Mgmt Heterogeneous Data Mgmt., Replication Distributed Query Processing Client/Server Client Server Architecture, Stored Procedures, Triggers 1980S Commodity & Low Cost Linux, Java-in-the-Database, XML 1990S 1994: Microsoft releases SQL Server 4.2 for NT, MS & Sybase Split Marketing SQL Server Certification with SAP Business Suite & NetWeaver DBMS Clustering Database Clustering, Cluster Workload Mgmt, Virtual Resource Management 2000S 2009: Leader in Forrester Wave for Enterprise Database Management Systems 2010S 2010: Best enterprise data management vendor on Wall Street four years running. 7
IDC Finds SAP Sybase ASE s TCO to be 28% Less Than Other RDBMS IDC Database Finds Platforms ASE s TCO to Best be Practices 28% Less and Than What s Other New RDBMS Outline Agenda IDC research conducted in-depth interviews with organizations to develop insights into the true costs associated with deploying ASE in comparison with other RDBMS products. Sybase ASE Five-Year Savings per 100 Users Sybase ASE installations were more efficient than the other systems, requiring 24% less storage capacity and running on 29% less server processor power. Sybase ASE required 27% fewer IT staff resources overall IDC White Paper (Dec 2011): Calculating the True Cost of RDBMS Ownership and How Sybase ASE Stacks Up: A Guide for SAP Business Suite Users 8
Migrating to SAP Sybase ASE Best Practices A (Very!) Quick Review Migrating Database Platforms to SAP Sybase Best ASE Practices and What s New Best Outline Practices Agenda Thoughts A (Very!) Quick Review Database Schema - Sybase PowerDesigner is your friend! Reverse Engineering. Schema Migration. Data Migration Approaches Big Bang Materialisation (Flat-file, ETL tool, Sybase CIS & proxy tables ). Trickle Feed / Incremental (Flat-file, ETL tool, SAP Sybase Replication Server, Sybase CIS). Continuous Data Movement SAP Sybase Replication Server. SQL / code Migration Hand craft / home brew. Tools provided as part of SAP Services packages semi automated targeted at Oracle to ASE. Parallel Running SAP Sybase Replication Server. 9
Migrating to SAP Sybase ASE Best Practices SAP Services can help you! Migrating Database Platforms to SAP Sybase Best ASE Practices and What s New Best Outline Practices Agenda SAP Services Can Help You A number of packaged SAP Services offerings are available to support your migration to SAP Sybase ASE including: RDS Rapid Database Migration to SAP Sybase ASE (inc. SAP applications). RDS Rapid Database Migration POC to SAP Sybase ASE to support a modest proof-of-concept (PoC) exercise. Try before you buy!. Database Migration of Custom Environments to SAP Sybase ASE. Please contact your SAP Account Executive for fuller details or myself during this conference if you prefer. 10
Migrating to SAP Sybase ASE Best Practices Useful Blogs Migrating Database Platforms to SAP Sybase Best ASE Practices and What s New Best Outline Practices Agenda Additional Migration Support Go to the Public SAP Community Network (SCN) SAP Services Content Library (http://scn.sap.com/docs/doc-34352) where you will find an excellent series of blogs by my college Jonathan Wesley-James on planning and executing SAP Sybase ASE Migrations: 11
Day & Zimmermann Engineering, Construction and Operations Day Database & Zimmermann Platforms Best Practices and What s New Engineering, Outline Agenda Construction and Operations As part of our technology transformation initiative we migrated our SAP applications running on the Oracle database to SAP Sybase ASE. This project virtualized 10 SAP applications consisting of 70+ servers with VMware on Windows. Significant cost savings were achieved on Sybase ASE annual maintenance costs vs. Oracle, and, using compression technology in SAP Sybase ASE, our storage requirements were reduced by 30% for additional savings. SAP provided expertise and training to ensure the migration went smoothly, and as a satisfied long-time SAP customer we believe SAP s strategic roadmap for its data management products will meet our needs well into the future. Trushar Desai, Director, SAP Technology, Day & Zimmermann 12
Redington Gulf IT distribution and support services Redington Database Platforms Gulf Best Practices and What s New IT Outline distribution Agenda and support services IN BRIEF Redington Gulf is the largest an end-to end supply chain provider for IT distribution and support services across the Middle East, Africa and Turkey. 2x less expensive than Oracle database 100% performance improvement over Oracle Redington Gulf replaces Oracle database with SAP Sybase ASE for lower TCO Business Challenges High Oracle database licensing and maintenance costs Solution Implementation Migrate SAP ERP system off Oracle to SAP Sybase ASE Deploy new SAP CRM system on SAP Sybase ASE SAP Services assisting with ERP migration Benefits 100% performance improvement over Oracle 50% more compression than Oracle DBA Cockpit provides simplified and better administration Lower database licensing and maintenance costs Lower database TCO 13
SAP Sybase IQ Real-time Data Platform (RTDP) Positioning SAP Database Sybase Platforms IQ Position Best within Practices Real-time and What s Data Platform New Near Outline Line Agenda Storage (NLS)* & many other scenarios possible IQ 16.0 * Discussed later. 14
SAP Sybase IQ In a Nutshell! SAP Database Sybase Platforms IQ Best Practices and What s New In Outline a Nutshell! Agenda Comprehensive webbased monitoring and administration Enterprise-grade RBAC authorization and LDAP authentication High performance, fully parallel bulk loading New indexing technology for increased compression and fast, incremental batch loads Re-engineered column store for extreme data volumes Web based administration and monitoring SAP Sybase IQ Engine Web-enabled analytics Communications and security Loading engine Fully parallel Role-based access control LDAP authentication Column indexing subsystem Query engine Aggressive scale out Hash partitioned tables and data affinity Text search In-database analytics Column store N-bit and tiered indexing New Generation PETABYTE SCALE store Low latency, write optimized store Storage area network Resilient Multiplex grid architecture Information lifecycle management Resilient Multiplex grid withstands network interrupts and server failures Aggressively scaled out query engine Intelligent query engine with data affinity for shared nothing performance on a flexible shared everything architecture Low latency, concurrent write optimized delta store 15
SAP Sybase IQ IQ Column Store and Compression SAP Database Sybase Platforms IQ Best Practices and What s New IQ Outline Column Agenda Store and Compression A cornerstone capability IQ stores data vertically Values of same column are stored next to each other, column values of a row are in separate locations Easy reference for fixed length values Efficient evaluation for very large number of rows with a limited set of columns Rows exist as row numbers which mark a position in each individual column vector IQ compresses values Dictionary Compression for repeating values, Storage Compression for all values IQ is optimized for very large databases IQ 16 holds the Guinness World Record for Big Data Load (> 34 TB / hr.) Massive parallel Multicore and Multinode query execution, table partitioning 16
SAP Sybase IQ 16 What s New? SAP Database Sybase Platforms IQ 16 Best Practices and What s New What s Outline New? Agenda Notable Enhancements in SAP Sybase IQ 16: Improved table concurrency multiple writers against a single table row level locking/versioning (RLV) feature NOTE: Part of REAL TIME option for IQ > uses memory based row versioning and row merge back techniques very similar to SAP HANA. Some usage restrictions apply. (Even!) better storage compression (NBit dictionary tokenization etc) targeted at VARCHAR/VARBINARY/other variable length data (Even!) faster data loading higher degrees of internal parallelism possible and data load query plans available allowing for low level tuning/optimization Breaking News!: Early like for like IQ 15.4 v. IQ 16 comparisons indicate 6-8x faster loads with load rates of 5,000,000 > 7,000,000 rows/sec easily achievable with IQ 16! Query (inc. DQP) processing improvements partition based hash algorithms & others LDAP support. Enhanced user security/authorisations based on fine grained access privileges/roles. HADOOP Integration HOT! 17
SAP Sybase IQ 16 HADOOP Integration Of particular note! SAP Database Sybase Platforms IQ 16 Best Practices and What s New HADOOP Outline Agenda Integration Of particular note! The data returned from a Hadoop analysis can be integrated into an SAP Sybase IQ database in several ways: ETL Processing Bulk load data from Hadoop data stores into SAP Sybase IQ using the open source utility SCOOP. Data Federation Expose HDFS files as tables in an SAP Sybase IQ database that participate in SQL queries. The HDFS files do not need to be loaded into SAP Sybase IQ. Query Federation Allow SQL queries in SAP Sybase IQ to execute Hadoop processes that return data that is incorporated into the SQL result set. Client-side Federation Federate queries across SAP Sybase IQ databases and Hadoop files using the TOAD SQL tool. New capabilities such as these help position SAP Sybase IQ as part of the SAP HANA Platform for Big Data next slides! 18
SAP HANA Platform for Big Data Where does SAP Sybase IQ fit in? SAP Database HANA Platforms for Best Big Practices Data and What s New Where Outline does Agenda SAP Sybase IQ fit in? SAP Enterprise data Non-SAP Enterprise data Mobile data Machine data (Sensors, SCADA, Machine Logs, Etc.) Data Sources Real-Time Replication Stream Processing Synchronization Analytics & Applications Dashboard / Reporting in Real- Time HANA In Memory Transactional Analytical Extended Storage (IQ) NLS Solution Tiered Storage (Hot-warm-cold) Smart Data Access Predictive Analysis Planning & Simulation Graph Spatial Large Low Cost Data Platform (Hadoop / IQ) Historical Data, Offline Batch Processes, Model Training etc. SAP HANA Platform for Big Data Enable real-time operations, analysis and actions Real-time data stream (Billions of events/day) Millions of events/day correlated with Enterprise Data Use Cases Energy Optimization Predictive maintenance Remote asset mgmt. Supply/demand forecast Inventory mgmt. Route optimization 19
SAP Sybase IQ In Summary SAP Database Sybase Platforms IQ Best Practices and What s New In Outline Summary Agenda SAP Sybase PowerDesigner, SAP Sybase Replication Server, SAP, ISYS, and Panopticon Bradmark, Symantec, Whitesands, Quest, and ZEND SAS, SPSS, KXEN, Fuzzy Logix, Zementis, and Visual Numerics BMMSoft, SOLIX, and PBS Ecosystem Optimized BI and EIM; model and replicate Development and administration tools Predictive analytics Packaged ILM applications App. services DBMS Hadoop and R Comprehensive ANSI SQL with OLAP Built-in fulltext search In-database analytics with MapReduce + simulator Web 2.0 APIs Big data OpnSrc APIs Most mature column store Comprehensive lifecycle tiering MPP queries + virtual marts + user scaling High-speed loads Structured + unstructured store 20
Customers Succeed with SAP Sybase IQ for Mission-Critical Applications Financial Database Services Platforms Customers Best Practices Succeed and with What s IQ for New Mission-Critical Outline Agenda Applications Customer US Securities & Exchange Commission Citigroup Morgan Stanley (Spain) ICICI Bank BNP Paribas Securities Services IQ Deployment With Sybase IQ, the SEC built a flexible, scalable and reliable data-rich warehouse that delivers faster load times and query results, and more comprehensive information views all at a lower cost. With Sybase powering their data archive, Citigroup users can now research historical data themselves, whereas in the past IT specialists were required to do so. With Sybase IQ, Morgan Stanley Spain now has access to heterogeneous system-stored critical data with single, unified delivery that has dramatically reduced analysis response times. ICICI bank has achieved improvement in system uptime and query performance over its previous Teradata implementation, in addition to the host of other benefits of the Sybase IQ data warehouse migration. Sybase IQ has improved the robustness of its online reporting system. Now the bank can compile larger amounts of data and anticipate growth in its customer base by proactively providing real-time data. 21
SAP HANA and ASE / IQ Working Together Smart Data Access A interesting new capability! SAP Database HANA Platforms and SAP Sybase Best Practices ASE / IQ and Working What s Together New Smart Outline Agenda Data Access A interesting new capability! SAP HANA SP6 was released recently and one of the key features included in this release is the Smart Data Access, which allows customers to combine SAP HANA data with other heterogeneous data sources like Hadoop, Teradata, SAP Sybase ASE and SAP Sybase IQ. With Smart Data Access (SDA) you can expose data from remote sources as virtual tables and combine these with other data in HANA physical tables using HANA models. There are few restrictions on usage of SDA currently: Data in virtual tables is read only. Virtual tables do not support BLOB/CLOB type data. Virtual tables cannot be used in multi node SAP HANA clusters. Some analytical views do not support virtual tables. 22
Near-line Storage (NLS) for Business Warehouse What is Hot and Cold Data? Near-line Database Storage Platforms (NLS) Best for Practices Business and Warehouse What s New What Outline is Agenda Hot and Cold Data? Hot Data is permanently accessed with extremely short acceptable response time. Dashboard style information continuously updated. Tactical level data. Aberrations must be recognized with minimal latency to identify disruption and react. Warm Data is occasionally accessed with a somewhat higher acceptable response time Kept as non-active data in primary BW database. Loaded into memory on demand, after which full query performance is available Cold Data continues to be relevant but is less frequently accessed or has a more relaxed response time constraint. Paper Report style information evaluated on demand. Strategic level data. Long-range evaluation for long-term decisions. 23
Near-line Storage (NLS) for Business Warehouse SAP Sybase IQ based NLS with BW on SAP HANA Near-line Database Storage Platforms (NLS) Best for Practices Business and Warehouse What s New Why Outline would Agenda you use a SAP Sybase IQ based NLS with BW on SAP HANA? SAP HANA platform can deliver the best possible Business Analytics performance. There are, however, limitations. Hard Limitations In- Memory database size is restricted by (combined) RAM size of server machine(s). Dynamic growth of data size may be faster than your reaction speed in improving hardware. Soft Limitations You may not want to cover the cost for the best possible technology for all of your Big Data. You may want to protect your system from data growth by only keeping a moving time window In- Memory. You may want to pursue not the best possible performance but the best value for money. Split your data between subsets demanding the best possible query performance and subsets less performance critical. For non-compressing DBMS, NLS will be more space efficient. 24
Near-line Storage (NLS) for Business Warehouse Solution Overview Near-line Database Storage Platforms (NLS) Best for Practices Business and Warehouse What s New Overview Outline Agenda BW on SAP HANA Example SAP NetWeaver Business Warehouse SAP HANA In-memory Database Sybase IQ High Value, Current Data Disk Based Database In-Line Storage Hot current data (- OR - OTHER SUPPORTED DBMS) Historical Data Trend Analysis Near line Storage Cold Historical data Limited size Massive volumes Rapid real-time access Slightly slower access 25
Near-line Storage (NLS) for Business Warehouse How does NLS work? Near-line Database Storage Platforms (NLS) Best for Practices Business and Warehouse What s New How Outline does Agenda NLS work? Beside the primary BW database connection, a second connection to the SAP Sybase IQ database is maintained NetWeaver maintains and executes the aging rules Aging happens along the time axis Controls data transfer to the NLS instance Cleans up data in the primary database Query federation is handled by NetWeaver Transparent to BW application logic Evaluation of InfoProviders / DSOs is directed to primary connection, NLS connection or both Intermediate results are consolidated by NetWeaver where required 26
Near-line Storage (NLS) for Business Warehouse How does NLS work? Near-line Database Storage Platforms (NLS) Best for Practices Business and Warehouse What s New How Outline does Agenda NLS work? Simple Example 3. Once the data is moved to Near-Line Storage, it is deleted from BW 1. Data partitioned by fiscal Quarter/year 2. Partitions being moved to Near-Line Storage are locked for update 27
Near-line Storage (NLS) for Business Warehouse Rapid Deployment Solution (RDS) Overview Near-line Database Storage Platforms (NLS) Best for Practices Business and Warehouse What s New Rapid How Outline does Deployment Agenda NLS work? Solution (RDS) - The Value Proposition Business benefits SAP NetWeaver BW Near-line Storage rapiddeployment solution will: Help gain a high performing near-line storage with the functionalities of reporting, ETL, lookup Reduce your total cost of ownership for SAP NetWeaver BW (powered by HANA) by implementing a clear and scheduled data aging strategy Enable faster online querying and reporting Avoid further interfaces to near-line storage as data is accessed by SAP NetWeaver BW seamlessly Decrease maintenance efforts for SAP NetWeaver BW Decrease system downtimes for SAP NetWeaver BW Measurable success Reduced SAP NetWeaver BW downtimes Reduced SAP NetWeaver BW maintenance efforts Reduced amount of data residing within SAP NetWeaver BW Faster querying and reporting Reduced total cost of ownership for SAP HANA IMPORTANT NOTE: Traditional DBMS based BW implementations are supported by the RDS. 28
Near-line Storage (NLS) for Business Warehouse Migration to NLS RDS Options Near-line Database Storage Platforms (NLS) Best for Practices Business and Warehouse What s New Rapid Outline Deployment Agenda Solution Migration to NLS Options Overview Option 1 Option 2 29
Near-line Storage (NLS) for Business Warehouse RDS - Go live in as little as 12 weeks! Database Near-line Storage Platforms (NLS) Best for Practices Business and Warehouse What s New Outline Rapid Deployment Agenda Solution - Go live in as little as 12 weeks! Service Name: Rapid deployment of SAP NetWeaver BW Near-Line Storage Implementation Duration: 12 Weeks* Core Offerings Implementation of SAP NetWeaver BW Near- Line Storage Identification of objects to be archived Installation and Configuration of SAP Sybase IQ database Installation of SAP Sybase IQ Client and configuration of SAP NetWeaver BW to Near-Line Storage Create up to 10 DAPs (Data Archiving Processes) Create up to 2 Process Chains Adjust settings for up to 10 Queries and 2 multiproviders Optional Task** Support Package Update * Based on scope ** Provided as option and impacts overall efforts of the service A special step-by-step guide describes each activity during the deployment 30
Near-line Storage (NLS) for Business Warehouse RDS SAP Service Marketplace Near-line Database Storage Platforms (NLS) Best for Practices Business and Warehouse What s New Rapid Outline Agenda Deployment Solution SAP Service Marketplace 31