Five Strategies for Performance Testing Mobile Applications

Transcription

1 Five Strategies for Performance Testing Mobile Applications Mobile applications are now mainstream business. Businesses ranging from healthcare providers to retailers and airlines are creating applications for mobile devices including iphones, ipads and Android phones. In fact, many new businesses are starting with a mobile application first, and then adding web applications later. In Q the number of smartphones shipped exceeded the number of PCs, for the first time. In January 2011, Apple reached 10 billion downloads from the AppStore. Few businesses can afford to ignore mobile users. In April 2010, Mary Meeker at Morgan Stanley estimated that the number of mobile users would surpass the number of desktops by the beginning of That tipping point may in fact come earlier, as mobile adoption has continued accelerating since she made that prediction. Mobile technology is simply ramping up faster than other technologies have in the past it may well be the fastest technology adoption curve in history. And this has important implications for your business model. You've got to be fast to market, but also prepared for rapid adoption. If your application performs poorly, or fails under load, multiple competitors are ready to take your place the barriers to entry are low. Performance testing is essential Mobile users have high performance expectations, whether using a mobile application or accessing a website from a mobile browser. According to Equation Research, 58% of mobile users expect sites to load roughly as quickly on their mobile phones as on their desktops at home. According to the Mozilla Foundation, shaving 2.2 seconds off load time increased download conversion by 15.4%, or 60M downloads a year. Sluggish performance, errors or crashes will send many users to a competitor, and may affect their willingness to do business with you at all, even from fixed workstations. The mobile app marketplaces are filled with competitive offerings;; if you want a spot on a customer's smart phone, you have to earn it. At the same time, you cannot afford to spend months, and millions of dollars, testing mobile apps before you deploy them. There are too many competitors, and the market is growing too quickly, for you to sit on the sidelines. So you need to carefully test your mobile apps, without slowing down your deployment cycle. And merely testing for functionality, while essential, is not enough. You've got to confirm how your apps will perform in the real world, ensuring that your infrastructure delivers the performance that mobile users expect, even during peak loads. 1

2 Five performance testing strategies for mobile If you want to make sure that your mobile app will function reliably and consistently even under heavy load, then use the following strategies when designing and implementing performance testing: 1.Test in the production environment 2.Test at 2-3 times anticipated peak load 3.Test accurately for the mobile environment 4.Test across geographies 5.Test for real time results 1. Test in the production environment Performance testing used to be fairly straightforward. Set up a lab environment, spin up a few load generators, watch how it performs and extrapolate upwards to your expected load. With the complexity of today's web infrastructure and mobile users scattered around the globe, extrapolating performance based on a limited lab test is dangerous. Testing in the lab or a staging environment is helpful, but is unlikely to reveal the performance and reliability problems that appear in production, at scale. To really test the performance of your mobile application, you have to extend your testing efforts beyond the application itself to the broader back-end infrastructure and application ecosystem. This means testing in the actual production environment. Production testing does not necessarily mean putting your production application at risk. There are ways to test in the production environment that limit your exposure. For example: Test during maintenance windows Test while the site is live, but during a low-usage time, so the potential impact of performance testing is limited, and be prepared do stop if needed Test in production but before the product is announced/released Test on a slice of the production infrastructure Only by testing in the production infrastructure can you take into account all of the components of your application ecosystem, including those outside the application itself. These may include: Shopping cart applications Content delivery network performance and file placement You can only discover problems with these third party application integrations by testing in the production environment. And you only uncover problems with your own larger infrastructure (load balancers, DNS routing, firewalls) by testing from across the web rather than a tightly controlled test environment.

3 2. Test for 2-3 times expected capacity What's the peak capacity that you expect for your application? Can you hazard a guess? No one is really good at predicting peak loads, we're all just guessing. Even companies that have been in the business for years and have history to look back on make the mistake of assuming that past performance predicts the future. That's a dangerous assumption in today's online world particularly with mobile users who can start downloading a new app or music album by simply pulling out a phone or tablet. For example, Internet giant Amazon was blindsided when it offered a one-day promotion for a new Lady Gaga album with its Cloud Drive and Cloud Player services. With more than 10 million Twitter followers and 32 million Facebook fans, Lady Gaga mobilizes a virtual army of fans on fixed and mobile networks. The response exceeded expectations and, in the words of the director of Music for Amazon, "melted" some servers. Amazon tried a 'do-over' the next day. Its business can certainly sustain the negative impact of this kind of public meltdown. Your business might not be so lucky. Many factors can affect your traffic in unanticipated ways: Your marketing team runs a brilliant social media campaign that takes off beyond their expectations A prominent blogger raves about your app External events conspire to send people your way Because of this uncertainty, we recommend that you estimate your peak traffic, and then test at two to three times that peak capacity. Testing at scale like this gives you confidence that your application infrastructure can handle an unexpected load without melting down. 3. Test accurately for the mobile environment Make sure that your performance tests can model the actual behavior and environment of your mobile users. Certain test platforms inherently restrict testing parameters and models. For example, some test platforms require the use of browser plug-ins to record web traffic to create the tests essentially limiting testing to those specific browsers. In the diverse mobile application environment, you need to be able to accurately reflect the user environment, including web services calls from games or other apps as well as mobile browsers. For mobile web pages, testing should include multiple mobile platforms, including Blackberry devices, WebOS and Windows Mobile. This is only achieved by emulating the traffic just as the phone or tablet would communicate with your application infrastructure. Once you have created accurate scripts from different mobile devices and user behavior profiles, you can then combine, parameterize or manipulate them to create the complex usage patterns and high traffic volumes needed for an accurate load test. Ideally, you should be able to correlate end-user performance on real mobile devices with performance metrics from your supporting back-end infrastructure, to discover how infrastructure issues impact the end user performance.

4 4. Test across geographies By definition, mobile users don't stay in one place. And the Internet defies geographical borders, so it is important to test across geographies for global reach. Send traffic from different parts of the country, over different Internet backbones, and from different countries and continents. Make sure that the application performs consistently and reliably across the entire infrastructure. 5. Test for real time results With fast-moving development cycles and rapid mobile adoption cycles, you cannot take weeks or months to test. You need to see results in real time, so you can make changes on the fly and test those changes as part of an iterative performance test. When testing in the production environment, real-time insight also helps you shut down testing that might be affecting actual customers or application performance. Sound too difficult? Use the cloud. A few years ago the guidelines in this paper would have sounded impossible and impractical. Who can afford to deploy a global testing infrastructure, or ramp up thousands or tens of thousand of virtual users in a few hours? Today, however, the idea of deploying testing applications across geographies, on a giant scale, is not unreasonable. In fact, it's surprisingly easy and affordable, with multiple cloud ecosystems like Amazon EC2 and Windows Azure offering the possibility of instant scale-up with a global reach. When testing mobile apps at scale, you do not actually need to deploy legions of mobile users around the globe on iphones, ipads, Android phones or other devices. Instead, use a proxy to capture all HTTP or HTTPS from each tested device performing the actions you want to test. You can use load generators to simulate the behavior of those devices. Creating these test scripts should not require complex scripting.

5 New technologies also offer the fast test deployment and real-time insight necessary to get the testing done without delaying deployment or adversely affecting production performance. If you are deploying mobile applications, you really have two options: performance test yourself to find and correct the problems created by running at scale in the real environment, or leave the performance testing to your customers, and use reactive crisis management to minimize the damage to your reputation and market share. The first alternative is a better approach from a business perspective, and with today's technologies it is easily within reach. About SOASTA SOASTA's CloudTest products and services leverage the power of cloud computing to quickly and affordably test consumer-facing Web and mobile applications at scale, providing customers with the confidence that their business-critical applications will be reliable and scalable, even under extreme loads. SOASTA s customers include many of today s most successful brands, including American Girl, Chegg, Gilt Groupe, Hallmark, Intuit, Microsoft and Netflix. For more information about SOASTA, please visit SOASTA, Inc SOASTA, SOASTA CloudTest, and SOASTA CloudTest On-Demand, are trademarks or registered trademarks of SOASTA, Inc. All other trademarks are the property of their respective owners.