APPLICATION PERFORMANCE TESTING IN A VIRTUAL ENVIRONMENT A REPORT FROM SAP CO-INNOVATION LAB : Joerg Nalik Shunra Software: Dave Berg HP Software: Mark Tomlinson 1.0
Table of Contents 1 Introduction Raising the Bar... 3 2 Testing in SAP Co-Innovation Lab... 4 2.1 Performance and Reliability Optimization of SAP Applications with HP LoadRunner... 5 2.2 Considering the Impact of Long-Range Networks... 5 2.3 Optimizing the WAN with Data Center Network Solutions... 6 2.4 Network Optimization Verification for Symmetrically Deployed Network Solutions... 7 3 Deployment and Test Results in the SAP Co-Innovation Lab... 9 3.1 Typical WAN Emulation Results Using Shunra... 9 3.2 Moving from Physical to Virtual Server-Based Tool Deployment... 10 4 Summary... 13 5 References... 14 Page 2
1 Introduction Raising the Bar Many say a perfect storm is hitting IT. The global economic crisis requires IT to do more with less, new cloud technologies are rapidly being deployed, and the Generation Y workforce, accustomed to using mobile applications as standard communication practice, are entering the corporate world. The bar for clarity and certainty in understanding the real-world impact of adding distributed network applications for all end users has been raised. Predeployment performance testing in a virtualized environment that encompasses all applications across the wide area network (WAN) for all end users is now standard best practice a must-do. In this paper we share an example of how this perfect storm could impact IT practices, and how we have successfully embraced today s new, virtualized environment for SAP applications using HP LoadRunner software and Shunra for HP Software for performance testing in emulated WAN environments. Let s note some thoughts: Total cost of ownership (TCO). TCO for an IT solution includes the need to integrate various IT technology stacks and applications. Successful integration requires predeployment testing and standardized test tools to increase and ensure the efficiency of test execution. Remote users. Millions of professional users of SAP applications are already working in remote locations. Expectations are that this number will increase exponentially as the number of casual users increases. We estimate adding a billion remote clients in a few short years. The WAN between application servers and end users, therefore, becomes critical. Additionally, we must also consider the SAP application servers themselves. SAP application servers and the WAN together determine overall reliability, availability, and response-time performance for end users. Testing performance, from a remote end user s perspective, is highly recommended. The cloud. We know that cloud is the new methodology for providing computing resources to applications. The natural starting point for using clouds to run business-critical applications is to move development and test systems into the cloud environment. Consequently, test tools need to be ready to follow applications into the same environments. In the SAP Co-Innovation Lab, 1 partner companies Hewlett Packard, Shunra Software, and SAP develop best practices for excellence in performance testing, analyzing, and remediation. In this joint paper, we demonstrate how the combination of HP LoadRunner 2 for performance testing of SAP solutions with Shunra for HP Software 3 for WAN emulation can dramatically increase test efficiency. The combined HP and Shunra solution performs perfectly in virtualized IT environments, the infrastructure foundation for cloud computing. Thus, the integrated HP LoadRunner and Shunra for HP Software solution is 100% cloud-ready. This paper complements a previous paper discussing the integration of Shunra s software with HP LoadRunner as it relates to testing global service-oriented architecture (SOA) deployments with proper optimization of communications across WANs. 4 Page 3
2 Testing in the SAP Co-Innovation Lab The SAP Co-Innovation Lab is a global network of labs dedicated to helping SAP partners perform proof-of-concept (POC) projects for their solutions within an SAP software environment. HP and Shunra sponsored their products for the SAP Co-Innovation Lab. Their solutions are used for various POCs, such as testing application delivery and WAN performance optimization products from various network vendors, SOA management products, 5 and more. Shunra emulates the WAN to introduce and replicate properties like network latency, bandwidth, and packet loss of an existing, planned, or nonideal network within test environments to assess performance, predict the impact of change, or otherwise optimize networked technology and business applications. Within the SAP Co-Innovation Lab environment, server, storage, and network virtualization technologies are employed, continuously evolving into the lab s own private cloud environment. The SAP Co-Innovation Lab utilizes Shunra 6 in conjunction with HP LoadRunner 9.5 for testing across global enterprises, introducing real-world network impairments into the lab. Within the SAP Co-Innovation Lab environment, the following takes place: SAP and SAP partner solution integration is easily tested and optimized in a (LAN) data center environment. Business software solutions can be evaluated without the complications of WANs. Shunra emulates a WAN to evaluate the impact of a global network infrastructure on business application solutions and application behavior. Following this performance testing, additional network solutions can be added and evaluated for countering the possible negative reliability and performance effects of a WAN infrastructure. Historically, the SAP Co-Innovation Lab used HP LoadRunner and Shunra together, but as separately deployed and operated tools. For this paper, we tested the integrated HP and Shunra solution and moved from a physical test environment to a virtual, private cloud-like environment. The test and optimization concept for SAP solutions we recommend follows four steps: 1. Optimize the application itself 2. Add WAN emulation and measure the impact on the SAP application 3. Test adding data center network optimization technologies 4. Test adding symmetric, data center, and branch office optimization technologies These four steps allow us to identify any reliability and performance issues, and to verify effectiveness of corrective solutions for any identified issues. Since the overall delivery of the application to the end user depends on the application and the delivery network together, we recommend that IT application and network teams work together for planning and execution of such tests. These four steps are described in more detail in the following sections. Page 4
2.1 Performance and Reliability Optimization of SAP Applications with HP LoadRunner Performance and reliability optimization starts with optimizing the SAP application. For the tests described here, we used the SAP NetWeaver Portal component 7.0 as the application under test. The same simple multiuser stress test scenario was used for all comparisons in this paper. Testing in this phase is performed to verify that all software is accessible and functioning as expected under ideal load conditions. 2.2 Considering the Impact of Long-Range Networks Shunra for HP Software is introduced to understand how the application will perform once deployed in a production network environment across geographical distances. The impact of WANs on SAP application delivery to end users or other remotely located applications can be measured through the Shunra integration. With this software-based solution, Shunra imports the characteristics of the production WAN (existing or planned) into the test lab. This alleviates the need for less accurate practices of testing during off-peak hours, deploying load generators in remote locations, or physically using a stopwatch in manual testing from remote locations. Time is saved and testing is conducted accurately and efficiently. Page 5
2.3 Optimizing the WAN with Data Center Network Solutions Testing with WAN emulation usually reveals problems including slow response times or high error rates for user request calls to the application. In extreme cases, it can also expose hard system capacity limits, security vulnerabilities, or total system landscape failure points. A number of network products, which are typically deployed in a data center as proxies in front of the application servers, can help control such issues. Using prior software configuration, running the SAP solution through various network proxy solutions can be verified. Page 6
2.4 Network Optimization Verification for Symmetrically Deployed Network Solutions Many WAN acceleration solutions use a symmetrical deployment of a network solution on the server and remote client sites. For symmetrically deployed net-work solution testing, the Shunra appliance 6 is recommended. The Shunra appliance can reuse HP LoadRunner scripts from previous phases and is comprised of a user interface based on Microsoft Visio coupled with one or more appliances. Page 7
One of Shunra s white papers, Best Practice for Selecting WAN Optimization Solutions: Benchmarking Performance ROI, 7 discusses the value that WAN optimization technology can provide to businesses that are highly dependent on the performance of applications over the corporate WAN. This paper also offers a best practice for selecting a WAN optimization solution that will deliver the highest return on investment to the business. Page 8
3 Deployment and Test Results in the SAP Co-Innovation Lab 3.1 Typical WAN Emulation Results Using Shunra Our test case was using the user login step to an SAP NetWeaver Portal with three different WAN emulation settings, which we refer to as LAN, DSL, and T3. The exact WAN emulation parameters are listed in the table below. The LAN case tests the performance of the SAP NetWeaver Portal as such without network obstructions. A continental DSL case is dominated by limited bandwidth, whereas the T3 case is dominated by network latency effects. Network Use Case Data center LAN: Within a data center LAN, pure SAP application server response time, 200 concurrent users Intercontinental T3: End user accessing SAP NetWeaver Portal over intercontinental distance, T3, 200 concurrent users Continental DSL: Branch office user within continental distance, U.S. East/West Coast over a low bandwidth DSL-type line, 70 concurrent users Network Emulation Parameters Bandwidth: 1Gbps Latency: no significant Packet loss: none Bandwidth: 45 Mbps Latency: 300 msec roundtrip Packet loss: 1% Bandwidth: 768 Kbps Latency: 40 msec round trip Packet loss 1% SAP NetWeaver Portal Response Time 0.57 seconds 4.58 seconds 4.53 seconds Page 9
Our results show that while the SAP NetWeaver Portal performs with excellent subsecond response time in a data center LAN, response times are severely increased through the WAN properties. Without testing with WAN emulation, this observation may not have been discovered until the portal was actually released for productive use by remote end users, who would then express dissatisfaction with the service it delivered. By following the performance engineering best practices defined in this paper, we were able to understand at what limited bandwidth the continental DSL case would be constrained, and what user capacity the link could support. Our observations found that the continental DSL case could support only 70 concurrent users, while 200 users, limited by the capacity of our SAP NetWeaver Portal, could be supported in the LAN and T3 case. An additional benefit of virtualization is that it becomes very affordable to assign a dedicated virtual server with a small physical footprint to run only HP LoadRunner. This eliminates any risk of incompatibly with other installed software components. 3.2 Moving from Physical to Virtual Server-Based Tool Deployment We wanted to know if a classical setup of HP LoadRunner deployed on a physical server coupled with the hardware-based Shunra solution for WAN emulation would yield the same results as Shunra s pure software tool integrated into HP LoadRunner, when set up on a virtual server. Our virtual server was based on VMware server virtualization on a physical HP blade server, NetApp SAN storage, and Cisco vlan networks. Page 10
The above graph is a measurement result of the login test of SAP NetWeaver Portal using the hardware-based tool environment. The network emulation parameters were T3 for bandwidth, 300 msec of latency, and 1% packet loss. The number of virtual HP LoadRunner users was ramped up over the first hour of testing. The test was then run under high constant load for another five hours. During the ramp-up period, the average user response times are slightly elevated because browser caches are empty during the first iteration of every user. This type of testing delivers two key metrics: Performance: The average response times in this case = 4.654 sec Reliability: During this measurement more than 100,000 transactions were performed over the course of the six-hour test time. Not a single error was registered. Therefore, we can estimate reliability to be better than 1 error in 100,000 transactions; excellent for WAN Web environments. Page 11
The above graph is the same measurement as before, but conducted with HP LoadRunner with Shunra software only tools and set up on virtual servers. The response-time average is 4.668 sec compared to the 4.654 sec in the previous scenario. The error rate was also less than 1 error in 100,000 transactions. Most impressive, this comparison proves that the virtual deployment of HP LoadRunner and Shunra, versus the physical deployment of HP LoadRunner and Shunra, delivers equal test results with great accuracy. Furthermore, with Shunra for HP Software deployed on a virtual server, greater efficiency gains can be realized and TCO can be lowered. Better hardware utilization: If HP LoadRunner is deployed on a physical server, it might be difficult to use the same server for other applications or software tools, especially when HP LoadRunner is in use. By using a virtual server instead of a physical one, it is easy to use physical resources only when HP LoadRunner testing is needed on-demand style. More agility: When deployed in a virtual server, the HP LoadRunner and Shunra software can be more easily relocated from one hardware environment to another, opening the opportunity to make this software a service offering within a private or public cloud environment. Page 12
4 Summary In summary, the perfect storm is already upon us with the realization of cloud computing. The unique integration of HP LoadRunner and Shunra offers SAP customers the tools they need moving forward. Specific technical benefits and best practices discussed in this paper are: Virtualization and cloud readiness: The integrated solution of HP LoadRunner and Shunra for HP Software is capable of running on virtual machines, reducing deployment and operational costs. This combination can be readily used in cloud environments. Increased test accuracy and decreased TCO: Load-testing SAP solutions with WAN emulation is a highly recommended best practice. Using Shunra to emulate real-world networks in a controlled test lab greatly increases QA testing accuracy with respect to actual end-user experience. It creates a repeatable test environment for problem finding as well as problem resolution verification, and drastically decreases TCO by removing the need for ad hoc, remotely deployed test equipment. One-stop testing of SAP application plus the WAN through the HP LoadRunner integrated with Shunra for HP Software minimizes testing overhead and changes in test configuration from one stage of performance testing to the next. Streamlined testing: By using the process outlined in this paper, SAP customers can follow a logical progression from initial solution validation through deployment. This will minimize escaped defects and help ensure reliable performance expectations. Simplified analysis: Producing credible results in a minimal amount of time is critical to any testing effort. The integrated HP and Shunra solution achieves this objective. Automated inclusion of network statistics and Shunra-specific, emulated location results grouping gives clear, credible results with no additional processing time required. Page 13
5 References 1. SAP Co-Innovation Lab Network: http://coil.sap.com. 2. Product reference: HP LoadRunner https://h10078.www1.hp.com/cda/hpms/display/main/hpms_content.jsp?zn=bto&cp=1-11- 126-17%5E8_4000_100. 3. Product reference: Shunra for HP Software www.shunra.com/shunra_ve_desktop_for_hp_software_overview.php. 4. Saha, S., & Nalik, J. (2007). Testing Secure Enterprise SOA Applications Across Wide Area Networks Without Leaving the Lab. Retrieved from SAP Developer Network (SDN) community: www.sdn.sap.com/irj/sdn/go/portal/prtroot/docs/library/uuid/00aa0f0b-3c4d-2a10-5593-a57beda851c0. 5. Lui, K. (2008, November 18). SOA Management Solutions from SAP Co-Innovation Lab. Retrieved from SAP community network blogs: www.sdn.sap.com/irj/scn/weblogs?blog=/pub/wlg/12063. 6. Product reference: Shunra appliance www.shunra.com/ve-suite-overview.php. 7. Lesser, A. (2009). Best Practice for Selecting WAN Optimization Solutions: Benchmarking Performance ROI. Retrieved from Shunra Software: www.shunra.com. For Additional Information Nalik, J. (2007, September 28). Network Infrastructure for SAP Application-based Landscapes. Retrieved from SDN community: www.sdn.sap.com/irj/scn/weblogs?blog=/pub/wlg/7447. Packard, H. (n.d.). SAP LoadRunner. Retrieved from SDN community: http://ecohub.sdn.sap.com/irj/ecohub/solutions/loadrunner. Page 14