Performance Testing of a Cloud Service Trilesh Bhurtun, Junior Consultant, Capacitas Ltd Capacitas 2012 1 Introduction Objectives Environment Tests and Results Issues Summary Agenda Capacitas 2012 2 1
Customer Over 99% of the company's revenue is earned via the ecommerce service Highly Competitive Market Business Critical Service 20% year on year growth Frequently introducing new products to keep up with the competition and to increase revenue Large ICT infrastructure mirrored across 2 data centres Capacitas 2012 3 Introduction Look at benefits arising from migrating the web and logic infrastructure that currently supports the ecommerce service to a cloud service Proposed a private cloud service with virtual machines (VMs) Some initial sizing of the number of virtual machines (VMs) required to meet business demand had been delivered by the service provider Capacitas 2012 4 2
Introduction This sizing had been achieved using assumptions around the capacity of the VM environment relative to the current production environment Customer wanted to conduct a proof of concept to assess whether the cloud service is fit for purpose from a capacity and performance perspective Capacitas 2012 5 Introduction Capacitas offers performance testing as a managed service to this customer amongst other services This includes Risk Assessment Writing Test Plan Run Performance Tests Produce Test Report Capacitas 2012 6 3
System Architecture Three tier layer Interfacing to various B2B and other 3 rd parties Capacitas 2012 7 Software Development Agile Development Process Sprint Planning Large number of developers working towards a monthly release Pre release activities include: Functional Testing Performance testing Capacitas 2012 8 4
Objectives The main objective for the customer was to understand the capacity and performance characteristics of the ecommerce application on the cloud offering compared to the existing physical infrastructure In order to do this, Capacitas planned to carry out a series of tests to measure the resource cost per transaction on both virtual servers in the cloud and the customer s physical infrastructure Capacitas 2012 9 Objectives In addition, the virtual machines were running on Windows 2008, Capacitas attempted to see if the current Windows 2003 limitation of maximum process size of 1.2 GB is removed in Windows 2008 Customer had opted to use the existing Load Runner load generation capabilities in the Test environment for this Cloud Proof of Concept project Capacitas 2012 10 5
Objectives Objective Description 1 Prove that the ecommerce web application is stable when running within a VM instance 2 Quantify the capacity throughput per web server VM in business terms 3 Quantify the capacity throughput of the web server VM, relative to the current production web server 4 Quantify the capacity throughput per Logic server VM in business terms 5 Quantify the capacity throughput of the Logic server VM, relative to the current production Logic server 6 Demonstrate that the known web server memory constraint which exists in production is alleviated on a web VM 7 Demonstrate that the CPU cost per business transaction is constant under increasing load 8 Demonstrate that the web server application memory leaks are no worse than those on the current production service, as far as can be reproduced in test 9 Prove that the operational management tools are fit for purpose from a capacity management perspective Capacitas 2012 11 Environment Cloud servers were connected to the existing physical performance testing environment with load generation capability. Capacitas 2012 12 6
Environment The Test environment needed to have the following configured: SSL turned on at the web server Compression turned off at the web server The virtual environment needed to have the following configured: SSL turned off at the load balancer Compression turned off at the load balancer Caching turned off at the load balancer SSL turned on at the web server Compression turned off at the web server Capacitas 2012 13 Environment Service Provider will be able to move the virtual servers over the two physical boxes according to the requirements of the test plan The underlying architecture was based on two clustered HP DL585- G7 4 CPU * 2.1 GHz AMD Opteron Twelve-Core, 128GB RAM servers The servers were running vsphere v4.1 Capacitas 2012 14 7
Limitations This approach had the following limitations because of the limited load generation capabilities: The logical layer was not loaded realistically during web tests There was no testing of the multiple machine logical memcached configurations under significant load however this issue exists with existing test environment There was no significant load on the underlying hardware that supported the virtual machines Capacitas 2012 15 Tests The tests consisted of the following transactions: Homepage Search Product Selection Additional Options Login Purchase Order Confirmation Order History Capacitas 2012 16 8
Tests Test LoadRunner Scripts Against Cloud Servers Setup Performance Counters Prepare Test Scenarios Capacitas 2012 17 Test 1 4 vcpu WEB Service times at the WEB server were much higher for some steps when measured on the cloud server compared to the physical server Capacitas 2012 18 9
Test 1 4 vcpu WEB Page sizes from the cloud tests were half than those measured from the baseline This indicated that compression could be turned on Capacitas 2012 19 Test 1 4 vcpu WEB Compression was enabled both at the WEB and Logic servers Capacitas 2012 20 10
Test 2 4 vcpu WEB (Compression Off) Turning Compression off did not improve the performance at the WEB server by much Capacitas 2012 21 Test 2 4 vcpu WEB (Compression Off) Page sizes were similar to the ones measured on the baseline Capacitas 2012 22 11
Test 3 8 vcpu WEB The 8 Cores WEB Server was performing worse than the 4 Cores WEB server Capacitas 2012 23 Test 3 8 vcpu WEB Further investigation into these issues showed that the web site was running in debug mode Capacitas 2012 24 12
Test 4 4 vcpu WEB (Debug Off) Service times decreased considerably but were still higher than those measured on the physical servers Capacitas 2012 25 Test 5 8 vcpu WEB (Debug Off) The 8 cores web server was still performing worse than the 4 cores one even with debugging off Capacitas 2012 26 13
Logic CPU Service times on the cloud Logic server was nearly similar to the ones measured on the physical Logic server except for the purchase step Capacitas 2012 27 VMware Counters VMware Performance Counters Collected Capacitas 2012 28 14
VMware Counters Capacitas 2012 29 VMware Counters Capacitas 2012 30 15
Transactions Per Second Response Time (s) Response Time Response Times 10 8 6 4 2 0 Home Search Purchase Physical 3.049 2.655 2.415 4 Cores 6.515 4.795 8.456 8 cores 6.775 4.514 8.924 The increase in response time had a negative impact on throughput as well Response times were worse across the main steps on the cloud servers when compared to physical servers 12 10 8 6 4 2 0 Transactions Per Second Home Search Purchase Physical 10.24 5.55 2 4 Cores 5.38 3.54 1.3 8 cores 5.29 3.56 1.49 Capacitas 2012 31 Setup Network Connectivity Configuration inconsistency Firewall Incompatible Code Issues Capacitas 2012 32 16
Conclusion Moving the ecommerce service to the private cloud was not a technically and from the discussions economical feasible proposition They would need 288 and 64 physical cores (3.0 GHz) at the web and logic server respectively, if they used 4 VCPU servers 2.1 GHz AMD Opteron) then they would need 480 and 80 cores at the web and logic servers respectively Capacitas 2012 33 Questions? Trilesh Bhurtun Junior Consultant Capacitas trileshbhurtun@capacitas.co.uk www.capacitas.co.uk Capacitas 2012 34 17