Amazon EC2 XenApp Scalability Analysis

Transcription

1 WHITE PAPER Citrix XenApp Amazon EC2 XenApp Scalability Analysis

2 Table of Contents Introduction...3 Results Summary...3 Detailed Results...4 Methods of Determining Results...4 Amazon EC2 x86 Standard Small...5 Amazon EC2 x86 High CPU Medium...6 Amazon EC2 x64 Standard Large...7 Amazon EC2 x64 Standard Extra Large...8 Amazon EC2 x64 High CPU Extra Large...9 Summary...9 In house XenServer Comparisons...10 Test methodology...11 Environment Configuration...11 Scripting User Interaction with EdgeSight for Load Testing...11 Network and Client Hardware

3 Introduction As IT organizations consider their future infrastructure management plans, cloud computing comes into focus as a method of increasing computing capacity when there is an immediate need and without the capital expenditure associated with investment in new infrastructure. Amazon s EC2 cloud closely follows the Infrastructure as a Service (IaaS) paradigm, which means it is a complete virtualized platform environment where IT customers purchase servers, software, data center space and network equipment as a fully outsourced service. In our case, we requested a fully functional Windows 2003 Operating System that could communicate directly with our existing infrastructure. The Infrastructure as a Service paradigm of cloud computing provides several benefits over a typical data center expansion: You do not need to add any new hardware, software, or network capacity you can supplement sections of your current datacenter in the cloud and move forward with increased capacity. A virtual machine instance is dynamically scalable if the server requires more resources like memory or processor power, an image can be taken and placed on a larger virtual machine instance. You have immediate access to more resources when needed. With Infrastructure as a Service steadily reaching maturity as an accepted method of delivering computing power on demand over the web, the Citrix Product Development Team set out to test the scalability of Citrix XenApp on Windows 2003 virtual machine instances within Amazon s Xen powered EC2 cloud. Our goals were to gather the specific maximum user limit for each Windows virtual machine type offered in the Amazon EC2 cloud by using EdgeSight for Load Testing to simulate user interactions and monitor performance characteristics of each simulation. Results Summary All test scenarios were constructed using Citrix EdgeSight for Load Testing automated application performance and regression testing framework. EdgeSight for Load Testing provides an efficient and cost effective method of server sizing and application load testing against a wide range of productivity scenarios for Citrix XenApp environments. Below is a synopsis of the results from each of the five scalability test runs: 3

4 Note: After analyzing the performance results for all virtual machine instances tested, we came to the following conclusions: CPU usage was the primary bottleneck Memory usage limits were never reached Through further research, we found that the best performing VM instances resided on the latest hardware using higher performance processors. This is also noted in the latest Amazon EC2 Developer Guide: Detailed Results Methods of Determining Results There are three primary methods used to determine when the limits of the machine have been reached: 1. Application launch time has doubled and in some cases, tripled. Application launch time is the time it takes to launch the published application after the session has connected. 2. Total CPU usage has reached above 85 percent. 3. EdgeSight for Load Testing has logged disconnected sessions before the test has ended. 4

5 The methods above were used together to determine the approximate user scalability limit for each virtual machine instance that was tested. In general, each method was complementary with the other in showing the user limits for each virtual machine instance. Amazon EC2 x86 Standard Small Platform Amazon ID CPU ID (Windows System Properties) RAM Users x86 Standard Small Dual Core AMD Opteron Processor 2218 HE, 2.60 GHz (1 core) 1.7 GB 7 In this graph, we see that the test reached a seven user limit. The points highlighted within the red section in the above graph highlight important pieces of information that show how we determined the user scalability limit of the x86 Standard Small virtual machine instance: 1. Processor utilization has reached up to 100 percent. 2. Application launch times for Excel, Powerpoint, and Word have increased by 100 percent to 200 percent. Note that the Word launch time started at 10 seconds, and based on the trend, would have been about 20 seconds at this point. 3. Script failures begin to occur at eight users. 5

6 Amazon EC2 x86 High CPU Medium Platform Amazon ID CPU ID (Windows System Properties) RAM Users x86 High CPU Medium Intel Xeon CPU E5345, 2.33 GHz (2 cores) 1.7 GB 21 In this graph, we see that the test reached a 21 user limit. The points highlighted within the red section in the above graph highlight important pieces of information that show how we determined the user scalability limit of the x86 High CPU Medium virtual machine instance: 1. Processor utilization has reached above 90 percent. 2. Application launch times for Excel, Powerpoint, and Word have increased by 100 percent to 200 percent. 3. Script failures begin to occur at 21 users. 6

7 Amazon EC2 x64 Standard Large Platform Amazon ID CPU ID (Windows System Properties) RAM Users x64 Standard Large Dual Core AMD Opteron Processor 2218 HE, 3.02 GHz (2 cores) 7.5 GB 17 In this graph, we see that the test reached a 17 user limit. The points highlighted within the red section in the above graph highlight important pieces of information that show how we determined the user scalability limit of the x64 Standard Large virtual machine instance: 1. Processor utilization has reached 90 percent. 2. Application launch times for Excel, Powerpoint, and Word have increased by 100 percent to 200 percent. 3. Script failures begin to occur at 19 users. 7

8 Amazon EC2 x64 Standard Extra Large Platform Amazon ID CPU ID (Windows System Properties) RAM Users x64 Standard Extra Large Dual Core AMD Opteron Processor 2218 HE, 2.99 GHz (4 cores) 15 GB 24 In this graph, we see that the test reached a 24 user limit. The points highlighted within the red section in the above graph highlight important pieces of information that show we determined the user scalability limit of the x64 Standard Extra Large virtual machine instance: 1. Processor utilization has reached 90 percent. 2. Application launch times for Excel, Powerpoint, and Word have increased by 100 percent to 200 percent. 3. Script failures begin to occur at 24 users. 8

9 Amazon EC2 x64 High CPU Extra Large Platform Amazon ID CPU ID (Windows System Properties) RAM Users x64 High CPU Extra Large Intel Xeon CPU 2.33 GHz, 2.55 GHz (8 cores) 7 GB 60 In this graph, we see that the test reached a 60 user limit. The points highlighted within the red section in the above graph highlight important pieces of information as to how we determined the user scalability limit of the x64 High CPU Extra Large virtual machine instance: 1. Processor utilization has reached 90 percent. 2. Application launch times for Excel, Powerpoint, and Word have increased by 100 percent to 200 percent. 3. Script failures begin to occur at around 61 users. Summary The most notable point taken from the tests is the large performance increase that was recorded when using the High CPU instances offered in the Amazon EC2 cloud. The x64 High CPU instance more than doubled the scalability offered by the next highest performing virtual machine instance. 9

10 For all of the graphs, once processor utilization reached 90 percent, application launch times in some cases tripled and EdgeSight for Load Testing recorded such degradation in user experience that scripts began to fail. In house XenServer Comparisons After the exploration of the scalability for each of Amazon s VM types, we decided to take the analysis one step further by attempting to answer the following question: How does Amazon s EC2 solution compare with what we already have? We proceeded to compare Citrix XenApp in the Amazon cloud with a typical in house Citrix XenServer setup that best matched two of the 32 bit and 64 bit instances available in the Amazon EC2 cloud: Note: The Citrix in house XenServer Enterprise numbers were gathered using a similar server with a 2.99 GigaHertz AMD Opteron Processor, model 2218, in a simulated WAN environment. Although the effort was made to match the hardware (CPU) and virtual resources (CPU and RAM) exactly, there are several factors that could possibly weigh into the differences between the Amazon EC2 and Citrix in house scalability results that were gathered: 10

11 Amazon uses Red Hat paravirtualization drivers, created from Xen version 3.x.x, while the Citrix in house environment used the latest Citrix Xen paravirtualization drivers. Although the processor models were matched, supplementary internal hardware including L1 and L2 cache and motherboard could be different. On the in house server, although the virtual machine specs (CPU and memory) were matched exactly, we only ran one VM at a time. Assuming that Amazon EC2 XenServers are running multiple VMs on the same machine at the time the tests were run, this could possibly figure in to performance differences. Test methodology The information that follows explains various characteristics of the test environment and the test scripts used to form our test methodology. Environment Configuration Each of the virtual machine instances were setup exactly the same to ensure that the only variable that changed between tests was the virtual machine type. Two Amazon Machine Images (AMIs) were created in Amazon EC2 one for each of the 32 bit and 64 bit operating systems. The 32 bit AMI was used for both of the x86 instance types, and the 64 bit AMI was used for each of the x64 instances. Two elastic IPs (Amazon s name for a static public IP address) were used; one for each of the server instances. The instance was kept in a workgroup in order to remove variability in login times associated with domain logins. One hundred local user accounts were created on each machine and logged in prior to running the tests in order to create the necessary local user profiles. This limits variability in login times associated with copying the default profile to the new user profile for new users. Citrix XenApp 5.0 with Hotfix Rollup Pack 3 was installed on each virtual machine AMI. Three Office applications were published on the XenApp server. Scripting User Interaction with EdgeSight for Load Testing By using EdgeSight for Load Testing, the Citrix test team was able to simulate real user interaction over the wire to the XenApp Servers running on the Amazon EC2 cloud. The applications tested were Microsoft Word 2007, Powerpoint 2007, and Excel The scalability test simulated user logins at a constant rate of three users every ten minutes, each user launching either Word, Powerpoint, or Excel. Once logged in, each user runs through a series of interactions with the application that loop forever until 11

12 it is determined that the server has reached its limit of logged in users. The following points provide a bit more detail into what the EdgeSight for Load Testing scripts do: Excel 2007 script: The user launches Excel, opens a worksheet, creates two large tables of data, runs through several menus to format the tables, runs through some printing and page setup options, closes the worksheet, and loops all of the steps until failure. Word 2007 script: The user launches Word, opens a new document, sets some menu options, types approximately two pages of text, inserts clipart, runs through printing and page setup options, opens help, closes the document, and loops all of the steps until failure. Powerpoint 2007 script: The user launches Powerpoint, opens a new presentation, creates a nine slide presentation using advanced slide features such as smartart and clipart, runs through a slide show, closes the presentation, and loops all of the steps until failure. Script failures will occur when the application does not register key combinations that are supposed to bring up a new window with a specific title. This typically happens when the server is under heavy load. When a failure occurs, EdgeSight for Load Testing disconnects the user. Network and Client Hardware For Amazon EC2 testing, all clients were located within the Citrix datacenter, and made connections across the internet to the Amazon EC2 cloud. Network latency was measured on average to be anywhere from 30 milliseconds to 60 milliseconds. The client hardware used was an HP DL360 G5, Dual Quad Core, 16GB RAM, with Windows 2003 x64 server installed. The EdgeSight for Load Testing launcher was installed on it and used to simulate all user logins. For in house testing, all clients were located within the Citrix datacenter, but were placed behind a WAN emulator set to simulate network latency between 30 milliseconds and 60 milliseconds. The XenApp servers were installed on Windows 2003 virtual machines running on Citrix XenServers (using Citrix paravirtualization drivers). The in house servers were an exact match for processor, AMD Opteron model 2218, 2.99 GigaHertz. 12