Performance Testing- Is Your Mobile Application Ready for Scale?

Transcription

1 Hexaware & Experitest Co-webinar: Performance Testing- Is Your Mobile Application Ready for Scale? April 2015

2 Speakers Guy Arieli CTO Experitest Vinayak Gaur Head of Mobile Application Testing Hexaware

3 Agenda 1 Why performance testing is important? 5 Demo 2 Optimal testing strategy 6 Case Study 3 Zoom into Mobile Network Virtualization 7 Q & A Importance Basic terms Challenges Experitest Solution

4 Mobile (App) Experience 60% of mobile users will abandon your app if it doesn t respond within three seconds or provide the desired user experience in any circumstance (Google) >65% Over 65% Of the apps on App Store and Play Store have poor ratings (2 or less stars) Brand A poor mobile experience can have a negative impact on the transaction; and in the long term, it could be even more detrimental to a brand s reputation.

5 Composite Testing Strategy = Optimal Testing Strategy Is this the new Trend? Functional Usability / Platform Security Device level Performance Server Level Performance Compatibility Regression Test Automation Compliance Network Virtualization > Segregation of Horizontal and vertical Test types > Combination of 2 or more test types > Implementation Strategy defines the parameters for combining Test types > Network Virtualization provides critical coverage across test types

6 Network Virtualization Testing Why is it Important for your testing Strategy? Fragmentation Performance Functionality Network Virtualization 30% of bugs found by end users were caused by running the application in "non optimal" network conditions

7 Drilling Down Into Network Virtualization

8 Network Virtualization- Basic Terms If your network is a water pipe your data is the water Throughput The width of the pipe, or how match water it can move in a second. Latency The length of the pipe, or how much time it will take for a drop of water to path the pipe.

9 What have Changed When We Moved to Mobile? Built in latency Built in disconnection Operator/Provider Wi Fi LTE GSM Different Connectivity Technologies Different physical geography conditions In the mobile age the connection is dynamic.

10 Lab Network Conditions Your lab device is connected to a high speed wi-fi connection The device connects through the Wi fi to the application server Wi fi Throughput=~30,000 Kb/sec Latency= <2 ms

11 Real Life Network Conditions Your customers are using a slow dynamic GSM network. The client device is on a different network in India So the difference is huge! The servers are in the US. GSM Throughput=~50 Kb Latency= ~250 ms

12 Why does my application care about the network condition? Your application assumes the data requested from the server will be delivered in a reasonable amount of time. A part of the quality of your application is related to tuning the timeouts and verifying the error message are appropriate. Other aspects of your application quality related to the user experience. If your application is not responding after a few seconds your user is likely to delete it.

13 SeeTest Network Virtualization Architecture a b c e N d

14 End-to-End Product Suite - Manual, Automation, Monitoring, Load, Network Virtualization, Manager G SeeTestManager U SeeTestAutomation Automation tool Same test runs on different devices Plugs into all testing & CI environments: Mobile Add-On for UFT Full & complete integration into HP UFT: Record a test, edit it, run it, and view results Integration of object repository and more SeeTestManual Manual testing tool Mini-recorder for automating short, repetitive, manual scenarios Report with screenshots of each step Manual to Automation Script Convertor Free Tool! N L SeeTestCloud Device Lab located onsite (in the customer VPN) Devices can be reserved and accessed by any remote user Used in conjunction with SeeTestAutomation/Manual for testing of remote devices SeeTestNetworkVirtualization Device performance under varying network conditions Simulating mobile network bandwidth latency and reception while on the move SeeTestLoad Real device generated mobile load Full integration to LoadRunner, Jmeter, etc.

15 Demo

16 Call for action Providing this information to the developers will enable them to use few technics to solve the issues: Reduce the data size Use CDM storage where part of the data (images) will be served dependent on your user device location. Cache the data Reduce the number of round-trips

17 Percentage of total Testing Analyzing the Completeness of a Mobile Testing Strategy 50% 40% 30% 20% 10% 0% Iteration 1 Iteration 2 Iteration 3 Iteration 4 Functionality Compatibility Usability Compliance Performance-Server Level Performance-Device Level Security Change in Distribution of Test Types across Sprints Performance Server Level Compliance Security > Functionality > Compatibility > Usability > Performance Device Level

18 Challenges Including Network Virtualization in your Testing Strategy Test Strategy Components Impacted Test Infrastructure Test Planning & Implementation 1 Distribution of Testing teams X X Efficiency in Test Execution Test Coverage 2 Support for multiple Mobile Platforms versions, devices, configurations X X X X 3 Organizational Info-security constraints X 4 Need for network coverage across regions X X 5 Application availability restricted to client devices X 6 7 Cost of infrastructure setup and sustenance; Tools & Licenses X X X Lack of adequate support from Mobile Test tools X X

19 Case Study

20 Case Study: End User Performance Testing of a Mobile Financial App CUSTOMER Leading global financial institution (140+ countries) BACKGROUND TESTING IMPERATIVES End to End Testing of Banker Application on Mobile Platform (aimed at providing the end user with information to conduct banker business activities) Business need: Improve Time to Market & Exceptional User Experience Web <_>Mobile cliet synchronization Validation of user level access controls Sync Testing User Security Testing Functional Testing Platform Testing Support of ipad features in conjunction with the App Performance Testing Server & Device level performance testing Speedy Delivery of the Application to the business Users in limited time frame User Experience surpassing the business needs for users in different countries Supported ipad & network configurations Adherence to human interface principles for ipad platform Compatibility Testing Usability Testing ipad Banker Application Device Build Management Test Data Management Built distribution strategy on mobile devices Setup & creation of business specific test data for different levels of testing SDLC Custom Agile SCRUM

21 Case Study: End User Performance Testing of a Mobile Financial App SAMPLE SCENARIOS App performance in different network configurations WiFi, 3G, 4G Mobile workloads during various business hours Performance characteristics based on geographical domain specific latencies CHALLENGES Validating app user experience in varying network bandwidth & conditions Performance benchmarking for real life usage scenarios (driving, low signal, etc.) WORKAROUNDS / TACTICAL SOLUTION Usage of Performance Testing NV integrated solution for for Server Performance testing Distribution of testing between Nearshore, Onsite & Offshore at different locations Manually vary parameters to enable repetitive testing in varying network conditions support from test automation

22 Thank you