Build Your Own Performance Test Lab in the Cloud Leslie Segal Testware Associate, Inc. leslie.segal@testwareinc.com
Problem(s) I was trying to Solve Need more VUs Don t have Tool Times up reschedule test Use Vendor Cloud Production Issue need to run test now Use Vendor Cloud Crap! How fast can Crap! I should just I build my own? Can t run test need dll for login build my own Schedule Test Times up reschedule test Can t run true test can t spoof IPs Times up reschedule test Can t run test Tool version different 2
Issues with Vendor Cloud Test Lab Must schedule in advance Limit on number of Load Generators (LG) you can use (1 LG per 1,000 vu) Can t install anything on cloud machine Permissions typically locked down Different version of tool software Accessing your application under test (open your firewall, open cloud firewall) 3
Advantages of Setting up your Own Cloud-Based Performance Test Lab Cost (hardware & tool) Need to look at all costs to determine if truly cost effective Time (don't have to wait for long approval processes for HW & SW) Control when (right-now), where (US, CAN, EU) and how many Load Generators Easy way to try vendor tools Test tool upgrades 4
Why Test from Different Locations? Different Users in different locations will have different experiences Those closest to your servers typically have the best experience Users inside and outside the firewall will have different experiences Using a Content Delivery Network (CDN)? Need to test that time sensitive data is consistent throughout Are you mobile? Huge variety of mobile networks throughout the world 5
Performance Test Architecture AUT Local Area Network Load Generators Controller Analysis 6
Tool Components HP LoadRunner Controllers individually licensed Multiple types of Virtual User protocols Have cloud offering (uses Amazon Cloud) Can supplement existing licenses with cloud licenses or use standalone Need license server Separate component for scripting Borland SilkPerformer Unlimited controllers 3 Levels of Virtual User Protocols Have cloud offering (uses Amazon Cloud) Can supplement existing licenses with cloud licenses or use standalone Need license server Integrated scripting component 7
Load Generators Number and size depends on several factors Number of virtual users (VU) you want to generate Type of VU Web service, DB, Web, Mobile Type of Application Thin client, fat client, amount of network traffic Distribution of Load and Transactions 50% of normal load from West Coast/20% Canada/30% East Coast 10% of users download content from your site 90% of users outside the firewall 8
Load Testing on Physical HW vs. VM Cloud machines are typically Virtual Machines Differences Between Physical & VM VM is a slice of a physical machine VM needs to perform an extra step to talk to actual hardware Excessive requests can skew results VM can handle about 80% of load of same VU on physical machine Need to actively monitor CPU, Memory, I/O, Network Utilization of LG while test is running 9
Cloud Cost Components Machine Size Storage 10
Machine Types Options On Demand Reserved Size Standard High Memory High CPU High I/O Operating Systems Linux Windows Ubuntu DB Oracle SQL Server Can start small and resize any instance 11
Amazon Cloud Sample Pricing 12
Rackspace Pricing (Windows) 13
Netel Cloud (CAN $) 14
Use Calculators to Estimate 15
Calculator - Amazon Note Large = 7.5GB RAM 16
Cost for Data Transfer Amazon Rackspace 17
Throughput limitations Rackspace 18
IP Addresses (IPv4) Amazon Each AWS Cloud Server comes with a non-persistent private IP address NATed to a public IP address AWS allows 5 Elastic IP addresses per account Cost for not used IP addresses - $0.005 per Elastic IP address not associated with a running instance per hour on a pro rata basis (~$44/yr) Rackspace Each Rackspace Cloud Server comes with a dedicated and persistent public IP address get same IP if shut down and restart instance Note that the server must remain in Active state to retain IP and there is a charge for an Active Server that is powered off Can purchase 5 public IP addresses for $2/month/IP 19
Locations (as of Jan 2013) Amazon US East (N. Virginia) US West (Oregon) US West (Northern California) EU (Ireland) Asia Pacific (Singapore) Asia Pacific (Tokyo) Asia Pacific (Sydney) South America (Sao Paulo) Rackspace US (Dallas) US (Chicago) UK (but separate account required) 20
Addressing Security Issues Rackspace Isolated networking Windows Firewall Dedicated Servers Private Cloud (software) Amazon Dedicated servers Windows Firewall Private Cloud VPN Gateway 21
Other Considerations Image Backups Can be created at any time Minimal cost to store images (~$.10/GB/month) File Level Backups Backup tools Can set up separate drive and clone it Where are you writing your scripts? Do you need to spin up a cloud machine to create scripts or just for execution? Managed service level and/or monitoring agreements for additional cost Software to remind you if you leave instances running (or auto shut them down) 22
Road Map to Set up your Own Performance Test Lab in 24 Hours or Less Sign up for Account Some magic here Modify Firewall, User Access & install 3 rd party SW Verify Connectively to Controller and Image LG Launch Controller Server Install LG Software Spawn N copies of LG Image Install Controller Software (and license) Launch Load Generator Server Start Testing 23
Step 1: Start in the Right Place 24
Step 2: Know What you Need 25
Step 3: Pick Good Name and Enough Space Windows O/S needs 30GB space 26
Step 4: Wait Depending on size and number of instances will take between 5-15 minutes before the instances are available for use Best to create 1 instance, set it up and then clone it Setup includes User access (default only Admin) Set Admin Password Configure Firewall Install software In addition to testing software will also need browsers (only IE with no add-ins), add-ins for browsers (java, flash, etc.) General use software such as adobe reader, zip file handler, excel, etc. 27
Step 5: Use (and Stop using) You will be charged for usage as soon as the machine starts spinning up You will be charged for any portion of an hour If you are not using a machine, remember to stop it 23 cents/hr. is nothing until you accidently leave 20 machines running over the weekend Consider Saving results & doing analysis locally Can also have separate drive that you can un-mount from cloud machine, turn off machine and access data from local machine Work on scripts locally or use smaller instance in cloud for script development Consider having license server in cloud (on small instance that is always on) 28