contents A U T H O R : C h e t a n J a d h a v End-To-End Performance Testing For SAP HANA Abstact 2 SAP HANA: An Introduction 3 The technical components that form HANA 3 How does HANA work as a concept 4 What applications does HANA support? 4 What SAP HANA means? 4 SAP HANA: Motivation for performance testing 4 SAP HANA: Performance Testing Approach 5 Planned Performance tests by types 6 Test Preparation and Mass Data Generation 7 End-to-End Performance Testing Using HP Load runner 7 SAP HANA: Detailed performance analysis 7 Processing time of an SQL statement on SAP HANA DB 8 Analyzing SQL statements using Explain Plan 8 Analyzing SQL execution with the profile trace 8 SAP HANA Monitoring & Performance Tuning 9 Server Monitoring 9 Using Load Runners SAP Diagnostics 9 Performance Tuning 10 Conclusion 10 Appendix/References 11 About the Author 12 End-To-End Performance Testing For SAP HANA Page 1 of 12
Abstract SAP High-Performance Analytic Appliance (HANA) is a data warehouse appliance for processing high volumes of operational and transactional data in real-time. It combines SAP database software with pre-tuned sever, storage and networking hardware. It is designed to support real-time analytic and transactional processing. This critical infrastructure component requires a professional and elaborated performance testing approach. Performance, load, and stress tests are subcategories of performance testing; each intended for a different purpose. The goal is to ensure a smooth introduction and operation of SAP HANA in routine business activities at minimal costs. This paper explains the concept of SAP HANA in detail and what is the end-to-end performance testing approach for SAP HANA. End-To-End Performance Testing For SAP HANA Page 2 of 12
SAP HANA : An Introduction SAP High-Performance Analytic Appliance (HANA) is a data warehouse appliance for processing High volumes of operational and transactional data in real-time. It is designed to support real-time analytic and transactional processing. SAP HANA is designed to process structured data from relational databases, both SAP and non-sap, and applications and other systems rapidly. It is capable of using three styles of data replication depending on the source of the data - log-based, ETL-based and trigger-based. The relocated structured data is stored directly in memory. Because of this, data can be accessed quickly in real-time by the applications that use HANA. The technical components that make up HANA? The heart of SAP HANA is the SAP In-Memory Database, a massively parallel processing data store that melds row-based, column-based, and object-based storage techniques. Other components of SAP HANA include: SAP In-Memory Computing Studio, SAP Host Agent 7.2, SAPCAR 7.10, Sybase Replication Server 15, SAP HANA Load Controller 1.00, and, SAP Landscape Transformation 1 - SHC for ABA. SAP HANA runs the SUSE Linux Enterprise Server 11 SP1 operating system. It is generally delivered as an on-premise appliance and is available now. Figure 1 - SAP HANA Architectural Design / Source: SAP 2011 End-To-End Performance Testing For SAP HANA Page 3 of 12
How does HANA work as a concept? SAP HANA is designed to quickly replicate and ingest structured data from SAP and non-sap relational databases, applications, and other systems. One of three styles of data replication trigger-based, ETL-based, or log-based - is used depending on the source system and desired use-case. The replicated data is then stored in RAM rather than loaded onto the disk, the traditional form of application data storage. Because the data is stored in-memory, it can be accessed in near real-time by analytic and transactional applications that sit on top of HANA. What applications does HANA support? SAP has delivered several next-generation, targeted analytic applications designed specifically to leverage the real-time functionality offered by HANA, including SAP Smart Meter Analytics and SAP CO-PA Accelerator. It also provides support to analytics related to retail, financial, telecommunications, and other industries and to horizontal use-cases such as human capital management. HANA is optimized well in order to interface with the SAP Business Objects portfolio of reporting, dash boarding, and other analytic products. In Short SAP HANA means: Real-time decision-making by bringing all the data in your enterprise within the reach of decision-makers within seconds in an easy-to-understand and usable format, which can help businesses perform better, run smarter and faster. Enables innovative new applications, combining high-volume transactions with analytics, to dramatically improve existing processes like planning, forecasting, pricing, optimization and other processes. High performance analytics for SAP ERP Accessing data directly from SAP ERP or combine it with other data sources for real-time analytics and insight into the business. 360-Degree view of business operations Integrate and model data from virtually any data source for complete real-time insight into the entire business. Accelerate business performance while reducing Total Cost of Ownership via less hardware and maintenance. It is based on a proven technology, which is easy to implement whether delivered on-demand or via appliances. SAP HANA: Motivation for performance testing Investments in SAP HANA are costly and require extensive justification and proof that the technology reacts quickly and effectively to changing business scenarios, rapid increases in data volume, and multiple application types. Hence, it is truly a mission-critical technology that requires solid performance and high availability to deliver the level of service the business requires. Failure of such critical system introduces tremendous business risk, both to the successful execution of internal processes and to the service level agreements as established. End-To-End Performance Testing For SAP HANA Page 4 of 12
The only way to uncover performance issues before they disrupt operations is to test the system under conditions that simulate production. In SAP HANA Environment, performance testing can be performed during below situations. New implementations (e.g. the first introduction of SAP HANA) Capacity planning and sizing Consolidation activities Upgrades Implementation of Support Packages, Patches & Builds. Figure 2 Aligning performance testing in SAP HANA Lifecycle. Performance Testing of SAP HANA in the above mentioned context has the following key challenges: Complexity - SAP HANA (HANA DB/NewDB, HANA Server & corresponding user clients) is a complex construct of different engines working in concert. Change - SAP HANA (HANA DB/NewDB, HANA Server & corresponding user clients) Requires to be tested after any change without the need of having connected all application servers. Reliability - SAP HANA is a central infrastructure component, which serves as basis for critical business processes. Expectation Management - due to many different stakeholders. Performance testing of all different SAP HANA scenarios including all user clients and application servers in a manual way is not possible. Therefore, an elaborated performance testing approach including a selection of representative test cases and automation is necessary. SAP HANA Performance Testing Approach: In order to ensure a high end-to-end performance of SAP HANA, user clients and application server s appropriate performance testing approach is required. Performance testing goal must help business reduce following factors: End-to-end execution time including specific known queries Resource consumption of end-to-end scenarios including long running queries End-To-End Performance Testing For SAP HANA Page 5 of 12
CPU consumption Memory consumption Degree of parallelization Throughput : It is the amount of transactions produced over time during a test. It s also expressed as the amount of capacity that an application can handle. Also before starting a performance test it is common to have a throughput goal that the application needs to be able to handle a specific number of request per hour. Volume restrictions SAP HANA Planned Performance tests by types: Baseline Test: Baseline test will be the first test that will be carried out. Creating a baseline is the process of running a set of tests to capture performance metric data for the purpose of evaluating the effectiveness of subsequent performanceimproving changes to the HANA system or application. A critical aspect of a baseline is that all characteristics and configuration options, except those specifically being varied for comparison, must remain invariant. Once a part of the system that is not intentionally being varied for comparison to the baseline is changed, the baseline measurement is no longer a valid basis for comparison. With respect to HANA applications, you can use a baseline to determine whether performance is improving or declining and to find deviations across different builds and versions. For example, you could measure load time, the number of transactions processed per unit of time, and resource utilization such as memory usage and processor usage. Volume Test: After completion of baseline test and having results of this test in hand, scenario for volume test will be created. It will contain all the transactions and volumetric to be created during this test will be the peak load per hour mentioned in the workload model. Stress Test: Once system has successfully completed the Volume test, stress test will be executed. Stress test is conducted by gradually increasing the load from peak load to expected load (in this case 100% of peak load) and response time is captured for each section where load is being increased. This test is conducted to find out the scalability of the application, hence resource utilization measurement will be critical metrics here. Parameters like Available memory, CPU utilization, Thread pool utilization and Max connection pool count, etc will play a vital role. Based on these metrics, load will be increased gradually till 100% of peak is achieved. Endurance Test: Once system is loaded with huge amount of data due to above test conducted, we can go ahead with an extended duration test, which we call as Endurance test. Load for this test will be average load per hour as mentioned in the work load model. Duration of this test can vary from 8 to 12 hours. Main objective of this test is to find out any memory leak in the application. Hence, major metric that needs to be monitored during this test is RAM utilization at application server level and database level. End-To-End Performance Testing For SAP HANA Page 6 of 12
SAP HANA: Test Preparation and Mass Data Generation The generation of Mass Data and its proportionate distribution according to the underlying data models and queries is a key success factor for any Performance Test. Therefore, planning and provisioning of test data is an essential and fundamental preparation step. Data structures and volumes have to be aligned from a content perspective and have to fit into the data models of the testing scope. The test cases must not only be fully functional but also repeatable. This requires a sufficient amount of data. An option for data generation is the HANA JavaScript Tool, which is a Java based engine for the execution of JavaScript files for data processing in SAP HANA DB/NewDB. Another option would be to use a tool based on Eclipse with JavaScript support (JEE or JavaScript versions). With this second option, HANA Java classes are exposed as JavaScript classes for script developers. As several generators for sequences, numbers, strings, dates, UUIDs are provided; both tools can be used for in a mass data generation scenario. SAP HANA: End-to-End Performance Testing Using HP Load runner: The HP Load Runner is used as an automation tool to validate and test the performance of the SAP HANA. The process is as follows: Define the business process: The test team uses SAP Load Runner to determine the optimum hardware and software platform, based on the operations of the business process itself. Develop the script: The test team then develops one or more test scripts to walk the software through the actual screens and entries that will be used by the business process from SAP HANA environment. This script represents the activities of a virtual user. Expand the user load: The test team uses SAP Load Runner to increase the virtual user load in order to simulate the peak loading requirements for this business process. Evaluate results: SAP Load Runner measures simulated performance against the key performance indicators for the process and then recommends any necessary changes to the related enterprise software systems. Retest to validate: The team uses SAP Load Runner to re-run the tests, using the modified environment, to validate the effectiveness of the changes. SAP HANA: Detailed performance analysis After having executed an E2E Performance Test using HP Load Runner, KPI s have been collected by using the monitoring features of HP Load Runner or standard transactions for analysis such as STAD or ST5. If KPI s measured during the E2E Performance Test using HP load Runner have not been met, a detailed performance analysis of SAP HANA DB/ NewDB is required as part of the root-cause analysis. The first step of such a root-cause analysis is to End-To-End Performance Testing For SAP HANA Page 7 of 12
understand the processing time spent in the SAP HANA DB/NewDB, when executing a query. A detailed analysis is carried out as follows: Measuring the overall processing time of an SQL statement on SAP HANA DB/ NewDB : This can be achieved by either investigating on Expensive SQL statements using the corresponding view within SAP HANA Studio. SQL traces via the Diagnostics feature with SAP HANA Studio. Analyzing SQL statements using Explain Plan The EXPLAIN PLAN statement is used to evaluate the execution plan that is followed by SAP HANA DB/NewDB in order to execute an SQL statement. The result of the evaluation is stored into the EXPLAIN_PLAN_TABLE view for later user examination. Exactly this result can be used in order to understand the different execution steps and their influence on the processing time of an SQL statement on SAP HANA DB/NewDB. Analyzing SQL execution with the profile trace The profile trace is one of the most detailed trace levels that are available in the SAP HANA Studio. The profile trace is particularly helpful to understand how the various engines inside HANA work together. For e.g. E.g. the number of records that are passed from one engine to another can be examined. Using the profile trace will help to understand the SQL statement and its execution procedure, which might enable to understand the root cause of a very high processing time. End-To-End Performance Testing For SAP HANA Page 8 of 12
SAP HANA Monitoring & Performance Tuning Server Monitoring: SAP servers can be monitored either through load runner or manually executing defined transactions in SAP. As a best practice, only required parameters should be monitored. Too many parameters generate huge amount of data and log files, as a result at the end of the execution, while load runner is collating data from all injectors and servers there is a possibility of controller crash or incomplete collation of results. Parameters to be monitored: Application Server %Disk Time Page Faults/Sec(Memory) %Processor Time Pages/Sec(Memory) File Operations/Sec Processor Length Data Queue Interrupts/Sec(Memory) Oracle Database Server: Import tnsnames.ora file in controller to connect to Oracle database. DBWR transaction table writes (V$SYSSTAT 1), physical reads (V$SYSSTAT 1), physical reads direct (V$SYSSTAT 1), physical writes (V$SYSSTAT 1), physical writes direct (V$SYSSTAT 1). UNIX Server: If central instance is in UNIX. Average load (Unix Kernel Statistics), CPU Utilization (Unix Kernel Statistics), Disk Traffic (Unix Kernel Statistics), Paging rate (Unix Kernel Statistics). Monitoring via SAP Transactions: Monitor user distribution in application servers and central instance via AL08 Trace the errors via SM 21 and ST22 Monitor all RFC s via SM66 Using Load Runners SAP diagnostic: Load Runners SAP diagnostics module helps in pinpointing performance problems on SAP HANA applications. During the Performance test, it traces and troubleshoots individual transactions across the Web, application and database servers. It helps in drill down a slow end user transaction till its method or SQL statement. SAP Diagnostic during performance test enables us to : Trace the application components exercised by business processes. Rapidly isolate application components that have a significant impact on end-user experience. Provide developers with precise data on how to make performance improvements. Such precision pinpointing of performance problems directly translates into significant business value: Quicker and more efficient performance testing cycles Decreased time to problem resolution by development End-To-End Performance Testing For SAP HANA Page 9 of 12
Performance Tuning: When end-to-end performance testing reveals HANA system, or HANA application characteristics are unacceptable, our focus must shift from HANA performance testing to HANA performance tuning HANA Performance tuning involves co-ordinated efforts from all the system stakeholders viz, Database administrators, System Administrators, Network administrators Performance Tuning Process Overview Tuning follows an iterative process that is usually separate from, but not independent of, the performance testing approach. The tuning process typically involves a rapid sequence of changes and tests. This process can take exponentially more time than regular performance testing. The following is a brief overview of a typical HANA tuning process: Performance tests are conducted with the system or application deployed in a well-defined, controlled HANA application test environment in order to ensure that the configuration and test results at the start of the testing process are known and reproducible. When the tests reveal performance characteristics deemed to be unacceptable, the HANA performance testing and HANA tuning team enter a diagnosis and remediation stage (tuning) that will require changes to be applied to the test environment and/or the application. It is not uncommon to make temporary changes that are deliberately designed to magnify an issue for diagnostic purposes, or to change the test environment to see if such changes lead to better performance. HANA testing and tuning team is given full and exclusive control over the test environment in order to maximize the effectiveness of the tuning phase. Performance tests are executed, or re-executed after each change to the test environment, in order to measure the impact of a remedial change. When the tuning phase is complete, the test environment is generally reset to its initial state, the successful remedial changes are applied again, and any unsuccessful remedial changes (together with temporary instrumentation and diagnostic changes) are discarded. The HANA performance test should then be repeated to prove that the correct changes have been identified. It might also be the case that the test environment itself is changed to reflect new expectations as to the minimal required production environment. This is an unusual, but a potential outcome of the tuning effort. Conclusion: With the SAP HANA environment representing a complex infrastructure, an intense and severe performance testing is a must and a key success factor. HANA Performance testing helps in estabilishing a baseline test for future performance testing, supports in performance tuning, helps in achieving compliance goals with respect performance of the HANA system. ncluding performance testing very early in your development life cycle tends to add significant value to the project. End-To-End Performance Testing For SAP HANA Page 10 of 12
References We have taken reference for this white paper from the below mentioned official websites to represent the authentic information on this subject. SAP Help Portal: http://help.sap.com/content/hana/overview/hana_overview.htm Service Market Place: https://service.sap.com/rds-hana-erp.htm SAP HANA Testing by: Sebastian Geissler HP Sites : http://h17007.www1.hp.com/us/en/converged-infrastructure/convergedsystems/appsystems/sap-hana.aspx Security guide: http://help.sap.com/hana/hana1_imdb_sec_en.pdf End-To-End Performance Testing For SAP HANA Page 11 of 12
About the author(s) Chetan Jadhav has 10 years of diversified experience in Manual & Automated Testing in SAP functional and non-functional. He is SAP Certified consultant in SAP CRM and has good knowledge about SAP HANA, SAP IS Retail, Supply Chain, and Transportation & Logistic. He also has hands on experience in SAP testing tools like HP ALM, SAP TAO, Worksoft Certify & Solution Manager. About L&T Infotech L&T Infotech is a wholly-owned subsidiary of the USD 14 billion plus Larsen & Toubro which has a presence in construction, engineering, manufacturing and financial services. L&T Infotech is ranked by NASSCOM as the 8th largest software & services exporter from India and among the top 20 IT BPO employers. L&T Infotech provides end-to-end solutions and services in BFSI; Travel & Logistics; Media & Entertainment; Healthcare; Energy & Process; Utilities and E&C; Hi-tech & Consumer Electronics; CPG, Retail & Pharmaceuticals; Auto & Aerospace; Industrial Products; as well as Testing; Mobility; Infrastructure Management System; BI/DW; SAP; Oracle and Microsoft; Enterprise Integration; and Manufacturing Execution Systems in addition to an innovative CIOthought partnership program that provides a value-driven edge to clients. L&T Infotech s horizon is filled with the promise of new and cutting edge offerings including an end-to-end cloud computing adoption toolkit and cloud advisory consulting services; mobility solutions; big data advisory services; and in-memory computing. L&T Infotech has developed intellectual properties (IPs) in all the vertical and horizontal service lines and leverages them to provide IP-led solutions. Headquartered in Mumbai, India, the Company is differentiated by its unique three-pronged value proposition encompassing Business-to-IT Connect, Engage the Future and Execution Excellence. (www.lntinfotech.com) End-To-End Performance Testing For SAP HANA Page 12 of 12