PorposalPPP Q: Gomez is standlone web application testing tool? Gomez provides an on demand platform that you can use for both testing and monitoring your Web applications from the outside in across your users, browsers, mobile devices and geographies using a global network of 100,000+ locations, 168+ countries and 2,500+ local ISPs. Reality Load is Gomez s web load and performance testing solution and you can reuse the same scripts for both load testing and production monitoring Q: What is the difference between the other load testing tools which enables the wan emulation, location based load testing and Gomez load testing? A few key differences are the scale of the Gomez offering, the number of locations that you can test from and the fact that it is not emulating a WAN. In terms of scale, the Gomez Last Mile network includes 100,000+ peers located in countries on every continent. (Reality Load runs on real user s machines on the local ISPs and connection speeds available to them). The Gomez network is already deployed and requires no configuration other than to choose which computers (we called them peers) you want to use for load testing or monitoring purposes.. This gives you an unprecedented view into user experience that cannot be gained by simply emulating different configurations. If you are interested in how your real users would experience your site under load you need to realistically test under the same conditions, and the Gomez Last Mile is the only way to do that. For example with Gomez Last Mile you can find geographical response time discrepancies that may surface only under load What will be the impact on the business if average response time stays under 4 seconds (your target goals), but users from key markets are seeing response time of 10+ seconds? Q: How would we use Gomez for the internal applications...not the external app? Reality Load, and the rest of the Gomez platform, is designed to test and monitor your applications across the internet, from an outside in customer perspective. But testing internal applications is a common request. At present there are a couple of options. One would be to allow the Gomez test machines access to the internal system, either by allowing requests from our IPs or by indentifying our agent string. Another option currently available for our Monitoring offerings, though not our load testing, is to deploy a Gomez private peer behind the firewall from which the performance measures could be taken. Q: Also what will be the approach of Gomez while testing any web based application running on a mobile phone? For testing mobile applications Gomez has 2 complimentary offerings, Reality Load (discussed in our webinar) and Active Mobile for monitoring mobile web applications. Users could run high volume load test using Reality Load, including device specific user agent strings and HTTP headers, while utilizing Active Mobile for the most accurate measure of end user experience. Active Mobile emulates devices across major mobile carriers in the US (Chicago, Boston, Seattle), the UK, Germany and China, giving an accurate view of the impact of the increasing load on the mobile user experience. Q: So the agent actually mimics browser behavior, is not actually running the scripts inside browsers? The browsers used by Gomez s Reality Load are real Web browsers. That means that they are actually running scripts and processing the pages, not just playing back HTTP type requests. Q: does it support Virtual private network usage? VPNs are not supported for load testing at this time. Gomez, Inc. 10 Maguire Road, Suite 330 Lexington, MA 02421 3110 Main: +1 781.778.2700 Fax: +1 781.778.2799
Q: Also does this tool monitors servers during the test run and co relate the bottle necks? Today most of our RL customers (like Jim from Dollar Thrifty mentioned during the Webinar) are using existing in house system mgmt tools to correlate performance inside and outside the firewall during a load test. The Gomez platform has an open integration philosophy, and we will continue to integrate with any EMS tools that you may be using. Now that Gomez is a Compuware Division we are looking into offering an integrated view of application performance under load by connecting the Gomez platform and Compuware Vantage behindthe firewall infrastructure monitoring capabilities, so you can quickly correlate bottlenecks and performance issues across both the enterprise and the Internet. Q: Do you use a 3rd party load test tool behind the GOMEZ Reality load tool? Something like HP LR, RPT or a combination of it? The Gomez load testing tool uses our own tool set. It uses the same script recorder and test agent as the rest of the Gomez Platform. This makes it very easy to move between load testing and 24/7 monitoring and vice versa. Q: Can the raw data be used for the CompuWare Load testing toolsets? The raw data is exported in CSV files, so it accessible to a wide range of toolsets. Q: Gomez software is opensource? Is there a trial version? it runs in any OS or web explorer? The Gomez platform is a SaaS platform accessible on demand through a web browser that means that you don t need to allocate hardware or install any software products, and relies on. The Gomez network 100k+ testing locations, 168+ countries, 2,500+ ISPs. It is not opensource, but if you are interested in a trial just let us know and we can set you up. Q: On your interface, when and why would you check the checkboxes, Exclude JS, CSS or Images? This was a feature requested by a number of our customers. Basically, it can serve two purposes, one is around test methodology and the other around cost. Methodology wise, we have customers who prefer to start their testing with simple steps, focusing very specifically on pieces of their applications. This focused approach starts with hitting just the basic pages and not any of the ancillary content like images, JS, CSS that may not be needed. Here they are focusing just on the impact of the requests for their application pages on the server under load. The second reason is cost. Many of our customers use CDNs that charge by the amount of content served. When running a lot of load tests the costs can get pretty high. So they will exclude the objects from their early tests and only include them for a final test or two. Also, just wanted to highlight that we see a lot of issues with third party providers and external components performing poorly, especially under load, and regional discrepancies as well, so don t forget to include test cases that cover external components and geographical testing into your plans We also allow users the option to exclude requests by host. Q: Are you taking care of that pages may be cached in a transaction? If not, can you be sure that the front end is not getting too much load? Our load test tool performs as a real user s browser would perform for each transaction. So if images are cached for instance, each iteration of each user run would request the object once, but not again within a transaction. 2
Q: If you are testing in the PRODUCTION environment during off peak hours, you still cannot know how much USER traffic is hitting your site. So you cannot isolate your results to your test. How do you account for UNKNOWN load during your test? This is definitely a concern for testing of production systems. Here it is important to monitor the utilization of the production system prior to the load test, during the load test and after the load test. This can give you a profile of the live traffic and assist in understanding the utilization on the servers. Also, by monitoring things like firewall bandwidth (at the network level), and comparing the overall total with the load reported for the test users (from the test tool) the load from live users can be extrapolated. For instance, if Reality Load was showing that the test was consuming 80 Megabits per second of bandwidth, and the firewall statistics were showing 90 Megabits per second of overall traffic, then you can get a sense of how much live traffic was on the site (in this case, 10 mbps worth of traffic). The same could be said for page request rates or other measures with the difference between the load tool measures and the basic system measures providing the sense of live traffic. The best way to ensure quality of experiences and that an application scales properly under load is to realistically test your application all the elements of the web application delivery chain from the outside in, using real users and desktops worldwide to access your production environment Q: How do you calculate transactions per second? One of the Reality Load metrics is transactions per minute, I assume this is what the question refers to. This is calculated as the sum of the transactions (complete runs of the test scripts) complete in each minute. Q: Do you help customers identifying workload profile, if yes what is your approach? Most of our customers operate in a self service model and utilize their own test methodologies. However we do offer collateral around best practices for things like workload profiling as well as offering professional services assistance where needed or requested.. In general though, the approach is to evaluate the traffic the web application is currently seeing with focus on three key areas: the most heavily hit pages/functionality, the most business critical pages/ functionality and any pages/functions known to cause problems. Q: Can you terminate a test run when things start to break? Does the initiator of the test have access to the test admin console? Can they terminate a test in a timely way to avoid crashing the application? The Gomez tool is a self service platform, so users have access to the controls to schedule a test, stop it, view real time results, etc. Tests can be stopped within a few seconds just by clicking a stop button. In addition, Reality has a Max Failure Rate settings (Reality Load checks that the % of users encountering errors at any given minute remains below the Max Failure Rate value) and it can automatically reduce the load level looking for essentially a last known good state if needed. Q: We've got a staging environment that matches production and it has a web server in the DMZ and is accessible from the Internet. We use Loadrunner to test the system. What is the advantage of using Gomez? Traditional inside the firewall testing tools, like LoadRunner, are built on the philosophy of generating load and measuring performance behind the firewall. This approach is valuable, but will only tell you part of the story, since these tools are focused on testing internal systems. However, your end users don't live in data centers they are located at the end of a long and complex web application delivery chain. When you generate load and measure performance behind the firewall you have no visibility into end user experience, or how 3rd party 3
components (ISPs, CDNS, etc) will scale under load. And all components and services need to perform optimally in order to deliver quality web experiences across all users and regions. Some examples of problems that you could be missing are: Inconsistent geo performance especially under load, CDN configuration issues or oversubscribed POPs, major ISPs outages, SMS routing, latency issues, etc Q: Did you support flex applications? Yes, Gomez supports flex applications. Q: Who does the scripting? Do customers write own scripts or do you write the scripts/scenarios for the customer? Specifically scripting web services The Gomez Platform, including Reality Load is a self service platform, and so most of our customers are building their own scripts for load testing and monitoring. However, there is a service desk option for free assistance with script creation if our customers run into any difficulties. Q: What language are scripts written in? The scripts for the Gomez Platform are recorder through a visual interface. Users click through their application and the Gomez recorders recorder the users interactions with the web application. When the scripts are played back, the agent builds out the page (document object model, javascript and CSS processing etc) and then replays the recorded actions. Behind the scenes, the scripts are saved as XML, and can be extended using javascript, but there is no scripting language per se. Q: How do you create the scripts (clicking through the site)? Yes, scripts are recorder by clicking through the site. Q: How easy is to script Q: How difficult is it to create testing scripts and what tools are used for that task? Scripting is relatively simple and uses the Gomez Script recording tools for script creation. It involves just clicking through the web site while the Gomez recorder captures your actions. When the scripts are played back the agent starts at the first URL and then executes the recorded actions. Q: Can we capture screens of application at predefined points in the script...? Reality Load allows users to collect screen captures on error (SCoE) in load tests. Q: Do you have any facility for testing client side rendering using javascript or ajax? The test users are executing the javascript, ajax calls, etc during the loadtesting, so those metrics are already included in the data. Q: Raw data exported is going to be cumulative of the ser responses how often it has been collected (or) for every user for every request until the end of the test? Reality Load allows users to export with the aggregated (minute by minute) data, or the full raw data set. The full raw data set includes data on every hit of every page request of every transaction run in a load test. Q: Is the summary report export format in XML? The summary report for Reality Load is exported as a PDF, though all of the data behind the reports can be exported in CSV files. 4
Q: How did you insure that you correctly backed out the scripted transactions without losing any real customer traffic data. Gomez transactions are easily identifiable by text in the request header. This text can be used to ensure that traffic monitoring tools do not count the load test requests in their reports. Q: what was the size of Page 0 (the megabits/s seem to grow really quickly is this representative of real life use?) Page 0 in the demo was a 250KB page. This is pretty common for a homepage, and in fact smaller than many I have seen recently. Q: Can the results point to database transaction bottlenecks? Interpreting load test results is a combination of effective test planning, collecting the right metrics and proper analysis. Looking at the user experience metrics, and combining that with data collected from the back end systems can point to a the location and likely cause of a bottleneck. For example, if only certain pages in a transaction are seeing performance degradation the first step would be to ask what they have in common. If they all make database hits then check the CPU of the database server. Is it within tolerable parameters or was it under duress. This correlating of functions exercised, user experience changes and server metrics can be highly effective in isolating performance bottlenecks. Q: Our test environment is significantly slower than our production environment. How do you load test in that scenario and insure that your production environment can actually handle the load? In scenarios where a test system and production system are not identical (and this is often the case) it is important to understand the 'factor of scalability', or the degree of difference between the two systems. This may take a little testing, or some estimation. In this way you could execute your test on your staging system and project them out to the production capacity. However, unless you actually run a test against your production system you'll never know for sure. I've seen many situations over the years where tests on staging go well, and then production fails under live load because of a missed configuration, unexpected change, etc. So despite the difficulties of testing against a production environment, I believe it s critical to run at least some tests against it before the customers do... Q: General load test question: So would starting with 50 users for 15 minutes and then running 100 for 15 minutes and so forth to see where the throughput breaks down as far as number of users...would this be a viable test My preference in load testing is to bring users on slowly over time, so that when a problem is encountered, you'll know exactly the load level that can be supported. In the example presented here, if the site could handle 75 users you wouldn't know that for sure. You'd have a 'yes' for 50 users, but only a no for '100'. I would start at 1 user and ramp to 50 over 10 minutes, then hold at 50 for 5 minutes (or until some transactions can complete at this load level), then ramp to 100 over another 10 minutes and hold again at 100. That way you would see the impact of increasing load as well as the flat load. Q: Is throughput test is the same name as "Stress test" for you? There is some overlap in the terms, but the goal of a throughput test isn t necessarily to push a system to the breaking point. The goal of a throughput test is to measure the system s ability to deliver pages, hits, bandwidth, orders etc (things that are typically measured in terms of metric per second/minute/hour), independently of how many users are running. So typically in a throughput test a small number of users run through transactions very quickly (no think time delays). This can uncover bottlenecks or validate capacity 5
with fewer users and less time than larger tests (what we would call concurrency tests), which are focused on measuring performance in terms of the number of users the system can support. Q: Does this product only test pure http traffic or can it handle new technologies like flex, java scripts, ajax etc and be able to emulate the client side processing as well? Reality Load uses the same transaction engine as the rest of the Gomez platform. This agent can handle flex, java script, ajax, etc and client side processing Q: Do your browsers include mobile devices such as the iphone and Blackberrys (on simulated 3G connections)? Presently Gomez supports mobile simulation for Monitoring, but not for load testing. The monitoring is simulating connections through modems on a all major carriers in the US (Chicago, Boston, Seattle), and in UK, Germany and China. Stay tuned for our Reality Load 2010 plans. Q: Also does this supports Citrix testing? Citrix is not supported at this time. Q: Does support security sites like https? Yes, HTTPS is supported by all of the Gomez products including Reality Load 6