Mobile Performance Testing Approaches and Challenges

Transcription

1 NOUS INFOSYSTEMS LEVERAGING INTELLECT Mobile Performance Testing Approaches and Challenges

2 ABSTRACT Mobile devices are playing a key role in daily business functions as mobile devices are adopted by most enterprises as a business tool. A smooth functioning mobile apps are more critical today. Issues such as performance or downtime are a greater concern, as they are linked with revenue or brand image. This has made Mobile Performance Testing an important stage during development In the fast changing world, it is essential to address mobile performance testing approach and challenges; this is especially true when it comes to mobile where end users now expect the same level of performance from mobile apps as they do from web apps. In the real world, applications will be accessed by many simultaneous users with different devices and various network conditions. In order to understand the complete mobile user experience for an application, testers must be able to replicate a realistic load on the back end of an application while measuring the performance of the application front end on real devices. In this paper we addressing, mobile performance testing approach and challenges, which helps to performance testing team to drilldown the performance issues which quick turnaround time INTRODUCTION In this upwardly mobile and fast paced world, it has become essential to be constantly connected with the customers even as they go about their daily lives. It is obvious that mobile usage has drastically increased in the past few years, as people are using their mobiles to browse websites and ultimately make a product purchase. According to a recent research report, internet usage on mobile phones in July 2013, accounted for 17% of global internet usage. Another study estimates that the total number of mobile phone users will reach 5 billion by In light of these statistics, businesses need to connect with the user on their mobile. Mobile Performance Testing Approaches and Challenges 2

3 However, mobile phones have their limitations, some of these include the varying display size amongst mobiles; market is shared across multiple operating systems; RAM and CPU have constraints and network infrastructure offers much lower bandwidth. The main challenge businesses encounter is delivering a fast functioning, high performing and engaging mobile app across all devices, including mobiles with lower hardware specifications; while retaining their end-customer s attention. This makes performance testing for the mobile a critical activity as compared to performance testing for the desktop. The performance testing for the mobile should be in-depth focused on back-end performance but the front-end performance as well. The below figure shows how mobile users affect a mobile app as compared to web users: With growing number of mobile internet users, the expectation for seamless mobile web experience has gone up as well. There are few such studies that indicate this trend; Mobile users will wait for a maximum of five seconds for a website or app to load 74% will leave a mobile website if it doesn t load in five seconds 50% will exit an app if it doesn t load within five seconds 71% of global mobile web users expect websites to load almost as quickly or faster on their mobile compared to desktop 1/3rd of the mobile users will go to a competitor s site if the website or app is slow 80% of mobile web users don t want to try a website or application more than two times Figure 1: How mobile users affect our application CHANGING MOBILE SCENARIO AND TYPES OF APPS The Internet usage on mobile phones has increased drastically over the last few years. According to a recent research report, internet usage on mobile phones in July 2013, accounted for 17% of global internet usage. Another study estimates that the total number of mobile phone users will reach 5 billion by Given this scenario, mobile performance testing needs to be customized based on the type of application. We need to effectively target the processes and components involved in these different types of applications. Mobile Performance Testing Approaches and Challenges 3

4 Predominantly we have three types of apps: Types of Apps Browser Based Apps Native Apps Hybrid Apps Due to the differences in bandwidth, network data plan limitations, device configurations a desktop browser-based mobile applications might perform differently. Most of the mobile browser based application looks like, m. example.com. While, doing load test for a website, it is equally important to test the mobile browser apps also. By installing the app directly on to mobile devices, native mobile applications allow users to access the applications quickly. Native apps are platform dependent and due to this the development cost is also high. This app can be used with or without Internet. Hybrid applications can be called as a combination of both browser as well as native apps. Hybrid apps are developed using web technology and wrapped with platform specific shell, allows the app to be installed like Native app. Due to this, the development cost is higher than others. Mobile (Mobile Network (2G/3G/4G etc...) Server Figure 2: Key Components Governing Mobile App Performance The performance of a typical mobile application depends on three key factors: Back-end processing time: This is the total time taken by the request to reach the server, response generation time by server and the time to deliver the response back to the client. This time is affected by the total load on the server, network bandwidth, data transferred and the external component (if any). Network Emulation: Networks have a wide range of variability when it comes to mobile internet usage and this makes network emulation important for mobile performance testing. We need to effectively emulate the network condition to replicate real-time traffic during performance testing. MOBILE PERFORMANCE TESTING DRIVERS The performance test of the mobile app should target every component in the app architecture. The key components for a generic mobile application are represented in the figure below. Content rendering time at the device: The time it takes to display the response delivered to the client on the device is measured. The platform type, configuration of the device and other hardware configurations affects this time. This time should be handled using an on-device monitoring tool. Mobile Performance Testing Approaches and Challenges 4

5 CHALLENGES FACED DURING PERFORMANCE TESTING Load Testing Tool Few challenges that impacts the performance testing process in mobile world are as follows Virtual Mobile User on cloud Firewalls Dashboard Real Devices Virtual Mobile User on cloud Figure 3: Recommended approach to Mobile Performance Testing PHASES FOR MOBILE PERFORMANCE TESTING: THE RECOMMENDED MOBILE PERFORMANCE TESTING APPROACH The recommended approach to testing the performance of a mobile application is as follows. The recommended approach provided here, utilizes Neotys Neo- Load, HP LoadRunner as examples. Planning, Develop Load Test Scripts, Execution Reporting Planning A well planned performance test methodology/strategy drives the success of a performance testing project and its execution. While designing a performance test plan, engineers need to forecast for the end user trends, apply right patterns else the load simulations will not generate results. Load the application servers using traditional load testing solutions In parallel, the mobile performance will be checked with a minimal feasible number of users leveraging real devices or mobile cloud solutions. There are a few key parameters we need keep in mind when the mobile performance testing is conducted: Type of application (Hybrid, native, browser based) Type of Network (broadband, LAN, mobile, wireless) Browser simulation (Android, Blackberry, Symbian, Windows, iphone, ipad, Tablets-Android) Bandwidth simulation Users load/patterns Mobile Performance Testing Approaches and Challenges 5

6 Develop Load Test Scripts With the increasing importance of mobile as a platform, several leading performance tools have started supporting mobile performance testing. Mobile applications can be recorded in many ways: i. Recording directly from the device itself a. Using the proxy mode as available with Neotys NeoLoad and other tools b. Using the tunnel mode as available with Neotys NeoLoad c. Using a Native app (recording application) like HP LoadRunner VuGen mobile app ii. Recoding with emulators iii. Recording on a desktop web browser (for pure HTML applications) a. Proxy Mode Most load testing tools support recording the user actions though proxies. We need to set up a proxy into the device or on an external machine and it records all the communication done on the device afterwards. Mobile User Activites Recording in tunnel mode WebServer Figure 5: Recording in Neotys NeoLoad Tunnel Mode To make use of Tunnel Mode it is necessary to activate the Tunnel Mode parameters. This can be done by applying the below settings. Also Neotys NeoLoad has an option to record desktop browser as a browser on a specific mobile devices. Edit > Preferences > General settings > HTTP Recorder > Tunnel Mode Recorder settings Proxy Server Mobile User Activites WebServer Figure 4: Recording in Proxy Mode b. Recording in Tunnel Mode If your application cannot be used via proxy or the mobile device doesn t support proxy, Neotys NeoLoad has recording capabilities using tunnel mode. Tunnel mode makes it possible to record any application, either web or native, standard or customized, for any platform in any version. Figure 6: Neotys NeoLoad Settings for Tunnel Mode Source: Neotys NeoLoad c. Recording using Native Recorder Costly performance testing tool has launched its own mobile testing application using LoadRunner Mobile Recorder, to record directly on to the mobile. For this, the device must be rooted to record. Though this option we can measure the end user experience and also create a load on the servers. d. Recoding with emulators During performance testing of mobile applications, we need to simulate different networks (e.g. 2G, 3G, Edge, WiFi etc.) to reach the real-time traffic. We can enter the real time bandwidth, latency and packet loss values using WAN Emulation. Mobile Performance Testing Approaches and Challenges 6

7 e. Recording using Desktop Browser Neotys NeoLoad gives you an option to emulate desktop browser as a browser on a specific mobile device. Figure 8: NeoLoad integration with Perfecto integration using Custom Action Execution Once performance testing team complete building an automation perfecto test script and NeoLoad load creation scripts, you can apply realistic traffic conditions such as browser, load mix, bandwidth, and networks conditions to generate a realistic load on the server. Figure 7: Recording using Desktop Browser Source: Neotys NeoLoad While application under load performance testing team can monitor server s resource utilizations and real devices metrics on a single dashboard, which helps to analyze the performance bottlenecks in quick turnaround time. Performance testing team will able to see how the DEVICES +SERVERS utilizations at various load levels on various devices (Android, ios, Windows Phone, BlackBerry etc.). 2. Perfecto Test Script Perfecto mobile is a cloud service provider, it is an Independent from hardware platforms, OS and browser, Perfecto Mobile enables creating and executing scripts on a wide range of physical and online virtual hardware. Pick most frequently used device which you intended to measure transaction response times. Which helps performance testing team to measure /monitor real time mobile end user experience by running Perfecto Test Scripts along with NeoLoad test scripts. Possible test profiles Running one script on several mobiles Running several scripts on one mobile Running several scripts on several mobiles Figure 9: NeoLoad Runtime dashboard Mobile Performance Testing Approaches and Challenges 7

8 Reporting Unlike traditional performance test metrics few parameters specific to mobile need to be understood clearly when results are presented. This includes devices level statistics namely: 1. Memory Free (KB) 2. Memory Cache (KB) 3. Memory Used (KB) 4. Bytes In-Mobile 5. Battery Level (%) 6. Bytes Out -Mobile 7. CPU User (%) 8. Bytes Out -WLAN 9. Bytes In WLAN The below graph show the average response times for the key business operation with different load levels vs real devices Figure 10: Sample test report with different load levels vs mobile devices MOBILE APPS PERFORMANCE OPTIMIZATION TECHNIQUE FOR DEVELOPERS Hopefully, you now understand a lot more about the challenges associated with mobile testing of native and web applications. Our recommendations for optimizing mobile performance are listed below. Use less number of HTTP requests between the client and server; this will greatly improve the page loading speed. Maintain a constant HTML code for all devices; to change the user interface for different devices. Use CSS media queries to specify which CSS rules should apply for the browser displaying the page. Use HTML5 web storage for browser caching, which is an alternative of browser caching. By in-lining resources instead of using them as linked references it can greatly increase the page loading time. Bandwidth consumption can be lowered by compressing and reducing page resources to make it faster. By resizing and optimizing images for the mobile screen, we can not only reduce the page size but also improve the processing power. Key Points A Quick Recap Mobile performance testing approach and challenges Phases of mobile performance testing approach Methods to build load test scripts and perfecto test script It is essential to consider the mobile end users real time conditions to simulate a load The mobile performance with single user should be monitored in parallel with the load on the server. Leveraging one script solution for both purposes loading the server and single user mobile rendering could be time saving. Mobile performance optimization techniques for developers Conclusion The popularity of mobile applications has grown rapidly and is increasing continuously therefore there is the growing need for performance testing, to remain competitive in the market and have engaging mobile applications. This will help deliver a better user experience, which remains most important factor to attract and retain mobile users. Load testing on Mobile apps is similar to the traditional desktop apps. However, Performance Engineers need to know the all types of architecture and the fundamental differences between Mobile and desktop application to test all parameters successfully. AUTHORS Sahadevaiah Kola, Senior Test Lead Sachin Goyal, Senior Test Engineer Testree is the Independent Verification and Validation (IV&V) division of Nous Infosystems. Testree has strong expertise in functional, non-functional testing, migration and consulting and advisory services on testing. Our deep domain knowledge lies in Banking & Finance, Insurance, Healthcare & Life Sciences and Retail. We work with industry proven proprietary frameworks and commercial tools to provide precise testing solutions, applying some of the emerging technologies. Testree s working models include onsite testing, offshore testing, combination model and through dedicated test centers or through domain focused test labs. With over 18+ years of testing experience and success of over 100 testing engagements, Testree offers all-inclusive testing solutions with critical focus in automation, performance and mobility testing. For more information, please visit and Nous Infosystems Pvt. Ltd. All rights reserved.