STeP-IN SUMMIT June 18 21, 2013 at Bangalore, INDIA. Enhancing Performance Test Strategy for Mobile Applications

Transcription

1 STeP-IN SUMMIT th International Conference on Software Testing June 18 21, 2013 at Bangalore, INDIA Enhancing Performance Test Strategy for Mobile Applications by Nikita Kakaraddi, Technical Lead, Honeywell Technology Solutions Copyright: STeP-IN Forum and Quality Solutions for Information Technology Pvt. Ltd. Published with permission for restricted use in STeP-IN SUMMIT 2013 in agreement with full copyrights from owner(s) / author(s) of material. All rights reserved. No part of this publication may be reproduced, stored in a retrieval system or transmitted in any form or by any means, electronic, mechanical, photocopying, recording or otherwise without the prior consent of the owner(s) / author(s). This edition is manufactured in India and is authorized for distribution only during STeP-IN SUMMIT 2013 as per the applicable conditions. Practices Experience Knowledge Automation Produced By Hosted By

2 1 1 Why Performance Test for Mobile apps? 1 out of 5 mobile application users would delete the application immediately from their device if it froze for just 5 seconds. 2

3 Abstract Abstract: With mobile application capabilities and increase in usage, the application functions that are performed on a mobile device are also becoming more complex in functionality and more critical to businesses. The mobile mode of service delivery being closest to end users and needs to be always available hence attention needs to be paid towards mobile application performance testing. The load tool must support the latest technologies and we need to construct a good strategy in order to effectively address mobile apps which are developed and deployed rapidly using a variety of cutting edge protocols Key Takeaway s: A complete technical guide to carry out end to end performance testing of mobile applications Performance Test Approach for each type of Mobile application Browser based Native apps Hybrid Application Strategy to test at each mobile application performance driver to recreate real world scenarios Device Network Server Performance Metrics that need to captured at each component level Assurance of 24x7 availability of Mobile applications Generic guidelines to experience better performance of an mobile application Target Audience would include performance test engineers & development team who carry experience in developing and testing the mobile applications cut across different technologies, devices and platforms. Audience should know the basic & advanced concepts of performance testing of web & mobile applications. 3 Speaker Profile Nikita Kakaraddi, a Software Test Engineer carrying 8+ yrs of extensive experience in System, Integration & performance testing of web application and core engineering products. Currently working as Tech Lead & I am part of Performance COE in Honeywell Technology Solutions, Bangalore, India. My key responsibilities are to drive the performance test engagement from end to end right from requirements phase, design, testing till execution, analysis and reporting. I am mainly responsible to carry out all the POC s (Proof of Concepts) for the applications and products developed using the upcoming technologies & complex solutions. Recently I was involved to conduct POC s for performance evaluation of mobile web and native applications, simulation of mobile networks, various metrics that needs to be captured at each component to measure the performance of an application. My areas of interest include extensive Workload Modeling, extensive coding, Performance Testing (Includes baselining, benchmarking, Data volume, load, stress & Longevity testing), Analysis, Profiling, Capacity Planning and bottleneck identification. I love to work in a challenging environment which synergies my conceptual & technical skills coupled with hard work, dedication & determination. Karthick Palanichamy, a Software Test Engineer carrying 2+ yrs of extensive experience in performance testing of web application and core engineering products. Currently working as Performance Engineer & I am part of Performance COE in Honeywell Technology Solutions, Bangalore, India. Recently I was involved to conduct POC s for performance evaluation of mobile web, native & Hybrid applications, simulation of mobile networks, various metrics that needs to be captured at each component to measure the performance of an application. My areas of interest include extensive Workload Modeling, extensive coding, Performance Testing, Analysis and bottleneck identification. I love to take up any challenging work & explore all possible solutions to address the problems in effective way, guide the team towards advanced level scripting and have good working experience in couple of industry well known performance test tools. 4

4 Flow Of Contents Objective Of Performance Evaluation of Mobile Applications High level Performance Test Approach for Mobile Apps Workload Modeling for Mobile Applications Recreating Real World Conditions Challenges in Testing for Mobile Apps Tools Catalog for Mobile Apps Generic Scripting & Test Approach for Mobile Applications Standard Pointers to Test Mobile Applications Insight into Performance Evaluation of Mobile Apps Case Study for Different Mobile Applications 24x7 Availability of Mobile Applications Generic Guidelines to Experience Better Performance of Mobile Application 5 Main Objective of Performance Evaluation for Mobile Applications An ideal objective of mobile application performance test should target to recreate real world conditions by evaluating each performance driver Simulating real world device conditions Simulating real world network conditions Simulating real world server conditions Recreating real world Device conditions Recreating real world Network conditions Recreating real world Server conditions Gauge the performance on each component 6

5 Performance Test Approach for Mobile Applications Requirements Analysis Planning & Design Infrastructure Setup Scripting Requirements from Stakeholders Analyze access logs Derive Workload Device & Code profiling Define strategy for mobile apps Condition to Load mobile Server Condition for Network Simulations Identify device / emulator/ tool for load simulation Enable monitors on mobile servers Install apps to capture device statistics Sniff traffic from the device Script using mobile protocol Simulate web & web service calls Automate device operations Tools used Loadrunner, VSTS, Fiddler, QTP, Muex Testing Mimic web & web service calls Simulate mobile networks Simulate mobile interrupts Gather Metrics Capture device statistics Responsiveness of business scenarios Mobile App Availability Metrics 7 Workload Modeling for Mobile Applications For a startup application, workload modeling inputs to be derived from stakeholders discussion For an existing application in production perform user info log analysis using the industry wide standard tools like Google Analytics Infrastructure Details Response time SLAs Network, bandwidth, network by geography Type of application, device, platform, parallel applications or OS components running in device Max expected load, Critical user operations with user (Mobile & Web) & Geographic distribution 8

6 Workload Modeling for Mobile Applications contd Scenario Device Type N/W Type Packet Loss Signal Latency Upstream Speed Downstream Speed Region % Of Users Search IOS 2G 2% 3 ms 64 kbps 512 kbps APAC 40% Add IOS 3G 1% 3ms 64 kbps 512 kbps EMEA 10% Navigate IOS EDGE 5% 5ms 94 kbps 512 kbps US 50% 9 Recreating Real World Conditions Simulating real world Device conditions Conditions to recreate Application Type(thick or thin client) Platform(Android, IOS etc..) Parallel applications or OS process running like Antivirus/GPS etc.. Parameters to monitor Battery, processor & memory consumption Response times Page component performance Method Level Profiling Simulating real world Network conditions Network Quality Network Load (percent of bandwidth) Network Fluctuation like jitter, packet loss, latency Network Variability (2G to 3G, 3G to Wi-Fi etc..) Network by Geography like Airtel 3G in Bangalore, AT&T 3G in New York Packets and bytes sent/received Average delay Packet drops Bandwidth Utilization Segment Size TCP Window Simulating real world Server conditions Workload Type (desktop or mobile) Load on Server interms of number of users from different workload types Specifying and obtaining servers with required network specifications. Processor& memory utilization Disk performance Database query performance Each process level processor & memory utilization Packets sent /received Method Level Profiling 10

7 Challenges in Testing for Mobile Apps The unique challenges in performance testing of mobile applications arise from Diversity of the Device Environment new mobile devices hitting the market everyday Hardware Configuration & Simulating Network Bandwidth & Speed Rapid Application Development (RAD) Methodologies Users Growth Exponential growth in mobile users, hence it is important to predict the business growth by forecasting in advance The performance test CTQ s for evaluation of an Mobile Application User experience Scalability Analysis Availability of the device Device level statistics Bottleneck Analysis Periodic Performance Assessment Interrupt Testing 11 Tools Catalog for Mobile Apps To Test Mobile Web Apps To Test Mobile Native Apps Mobile Network Simulations MobileDevice Remote Connectivity Mobile Apps 24x7 Availability Check Mobile Apps Profiling Android Loadrunner Soasta Soasta Monkey Talk Robotium Monkey runner Shunrawith Loadrunner VNC Viewer Jenkins Plugin along with Soasta BAC Gomez Deviceanywhere Solarwinds Eclipse MAT (Memory Analyzer Tool) DDMS Trace view IOS Loadrunner Soasta Soasta Monkey Talk Shunrawith Loadrunner VNC Viewer Jenkins Plugin along with Soasta BAC Gomez Deviceanywhere Solarwinds IOS Instruments Blackberry Loadrunner Shunrawith Loadrunner Tiny Controller Pocket Pro BAC Gomez Deviceanywhere Solarwinds Blackberry JDE Windows Loadrunner Shunrawith Loadrunner Pocket Pro M-eux BAC Gomez Deviceanywhere Solarwinds Dot Trace Profiler 12

8 Generic Scripting & Test Approach for Any Mobile Application Mobile Connected to Wi-Fi 1 Proxy Server to Capture the traffic 2 Proxy Tool 3 Scripting Tool 4 A Mobile Servers 3 Test Agents(Geographically Distributed) Test Controller 2 1 A 4 Scripts Developed 13 Standard Pointers for Testing Mobile Applications Approach Load users from Web Mobile browsers Actual device Webservice Calls Prerequisites Work Load Geographic Distribution Mobile Browser to simulate Mobile device to simulate Network bandwidth specifications Challenges Simulating the bandwidth fluctuation Network variability test Number of devices required to simulate the load. Simulate WebService Calls to load the server. Key Outcomes Response time Metrics Data Usage from Mobile. Device level battery & memory utilizations Server level resource utilization Metrics. 14

9 Insight Into Performance Evaluation of Mobile Apps Continuous Integration On Mobile Platforms Script using mobile protocol Change the user agents Tools used: Loadrunner, VSTS Mimic web /web service calls Requirement Analysis Access Log Analysis Strategy for Mobile apps Condition to load web server Condition for Device operations Condition for network simulations Sniff traffic from device Simulate web service calls only Automate device operations Tools used: Fiddler, QTP, Muex Simulate mobile network Simulate mobile interrupts Capture device statistics Responsiveness of business scenario 15 Case Study for Different Mobile Applications.Net Native / Hybrid Mobile Application on Windows 1. Objective was to identify break point in terms of supported number of users and Device performance on continuous usage by the field engineers 2. Network Simulated 2G & 3G 3. Components for simulating Mobile & Web Browser requests, Web Service Calls invoked by the native application, UI activity on the device Outcome of the test: 1. Responsiveness of the identified business scenarios at browser level & device level. 2. Device level statistics like memory, battery consumption & Disk.Net Browser based Mobile Application on IOS 1. Objective was to identify response time of the identified business scenario under specified network bandwidth. 2. Mobile Browsers simulated - Safari 5 & 6 3. Network Simulated - 2G, 3G and Edge Outcome of the test: 1. Responsiveness of the identified business scenarios 2. Performance impact with varied browser versions 16

10 24x7 Availability of Mobile applications Alerting System triggers notification to application owners & stakeholders in case application is not responding or in case of server resource utilization reaching beyond 80%. Configure Alerting System for notifications & execute the business critical scripts 24x7 Advantage: Ensures availability of mission critical applications and databases during planned and unplanned downtime by monitoring the status of the application Provides faster application failover in the event of an outage or disaster. 17 Generic Guidelines to Experience Better Performance of an Mobile Application Generic techniques for optimizing the mobile apps performance Reduce number of requests between client and server. In-lining resources and sharing it across multiple pages. Utilize browser caching and HTML5 web storage features effectively. Reduce Bandwidth consumption by compressing and reducing page resources. Resize images according to mobile screen. Techniques to ensure end user experience is less dependent on device network vulnerabilities: Enable data compression before transfer Avoid redirects Minimize data content size Reduce number and length of cookies Remove white spaces and comments from code Organize the delivery for incremental rendering, aggregating requests and using PUSH behaviors 18

11 THANK YOU Nikita Kakaraddi Karthick Palanisamy - [email protected] 19