Perfect Your Mobile App with Load Testing and Test Automation

Transcription

1 Wipro & Experitest Co-webinar: Perfect Your Mobile App with Load Testing and Test Automation June 2015

2 Speakers Guy Arieli CTO Experitest Sudheer Mohan Director - Mobility Certification & Automation Wipro Technologies June 2015

3 Agenda 1 2 Non Functional Testing for Mobile Challenges in Mobile Load Testing Demo- See Test Integration with LoadRunner Mobility Automation Wipro s Automation Framework Case Studies- Major Bank and US Retailer

4 Non Functional Testing Real world conditions Device Manufacturers Platforms & platform versions Service Providers Enterprise Challenges Performance issues (Slow, freeze, crash) Heavy mobile resource utilization Uses heavy network bandwidth Accessibility compliance Low application ratings Unsatisfied customers & Potential loss of Business WIPRO LTD CONFIDENTIAL

5 Wipro Non-Functional Test Solution 3rd Party Integration Social media Crash analytic tools Analytic tools Network condition Handover situations Data transfer Sensitive data validation Simulate mobile user load* Load testing * Stress testing * Security testing Mobile Apps Mobile network Mobile Platform Enterprise Services * Covered as part of SeeTest load testing solutions App profiling Interruptions App response time Exploratory testing Accessibility testing Load testing* Stress testing* Security testing Focused non-functional testing WIPRO LTD CONFIDENTIAL

6 Challenges in Mobile Load Testing Mobile generates more load

7 Mobile generates more load The number of mobile users already exceeds the number of desktop users. Mobile users use your application more extensively

8 Challenges in Mobile Load Testing Mobile generates more load The market shifts from web to mobile

9 Thin-Thick Client Mobile applications are replacing Web Application The mobile enterprise business application market is projected to double by 2018 Gartner

10 Challenges in Mobile Load Testing Mobile generate more load The Market shifts from web to Mobile Mobile Apps are relatively more complex

11 Thin-Thick Client Mobile apps are relatively complex Protocol Data Communication Concurrency Security Transaction Web Single Protocol- HTTP HTML Synchronized request / response Single threaded app Unified security browser based Single request / single response - Submit / next page Mobile JSon over http, JSon over TCP, XML-RPC, Soap, REST, RMI Json, XML, Free Text Both synchronized and nonsynchronized Multiple threads Unlimited methods multiple requests

12 Challenges in Mobile Load Testing Mobile generate more load The Market shifts from web to Mobile Mobile App is relatively more complex Mobile load creates a larger effect on your servers

13 Mobile load creates a bigger effect on your servers One of the main causes of load on your server is the number of concurrent sessions Mobile sessions take longer to be processed due to mobile network conditions. It results in an overload of your servers The same number of users from mobile compared to desktop can result in over 20% more load on your servers.

14 Challenges in Mobile Load Testing Mobile generate more load The Market shifts from web to Mobile Mobile App is relatively more complex Mobile load create bigger effect on your servers Traditional Load Testing is Synthetic

15 Traditional Load Testing is Synthetic In traditional load testing the application under test is being removed and a synthetic network traffic is being generated How does your application traffic look like? Your application open TCP connection to the server (SYN-> SYN,ACK<- ACK->) Your application sends a request. The server sends a response. Your application is closing the connection. What would you find in your application request? Key value property data structure includes information like: Request information Request parameters Session ID Transaction ID Typical requests can have dozen or even hundreds of fields. One of the big challenges of a load engineer is to understand the nature of all of those fields.

16 Challenges in Mobile Load Testing Mobile generate more load The Market shifts from web to Mobile Mobile App is relatively more complex Mobile load create bigger effect on your servers Traditional Load Testing is Synthetic syntactic load is Risky

17 Syntactic Load is Risky 2 type of correlations: User parameters to request fields Server response to user request Why do I need to modify the request fields? To simulate different user inputs i.e different user name / password or different UI form population To support internal server protocol, a very common server protocol scenario is that after the login request, the server will return a session ID. The client needs to use this session ID in any request that HAs been performed.

18 Syntactic Load is Risky It will create no effect on the server What will happen if I will modify the request field? The server will ignore this field The server will ignore the entire request Main Risk The request will cause a server internal error With Synthetic load you don't really know what will happen, it is a trial and error process.

19 Challenges in Mobile Load Testing Mobile generate more load The Market shifts from web to Mobile Mobile App is relatively more complex Mobile load create bigger effect on your servers Traditional Load Testing is Synthetic Syntactic Load is Risky Hard to Validate a Syntactic Load

20 Validate Syntactic Load Performance teams need to adopt their methodologies to validate that the generated load is valid. A few approaches can be: Database / UI validation to verify that the transaction generated has the expected effect Looking at the server log files to find transaction completion log messages. Monitor the server vitals to verify the load effect on the servers CPU, memory, number of threads

21 Challenges in Mobile Load Testing Mobile generate more load The Market shifts from web to Mobile Mobile App is relatively more complex Mobile load create bigger effect on your servers Traditional Load Testing is Synthetic syntactic load is Risky Hard to validate syntactic load

22 Monitoring your load using a real device transaction As you generate the synthetic load, you would like to generate few interactions with the servers based on real devices running your real application. Benefits Provide validation to the syntactic load effect. Provide response time reference (when executed without the syntactic load) Provide real transaction response time.

23 End-to-End Product Suite - Manual, Automation, Monitoring, Load, Network Virtualization, Manager G SeeTestManager U SeeTestAutomation Automation tool Same test runs on different devices Plugs into all testing & CI environments: Mobile Add-On for UFT Full & complete integration into HP UFT: Record a test, edit it, run it, and view results Integration of object repository and more SeeTestManual Productivity tool for Manual testers Report with screenshots of each step Used in conjunction with SeeTestCloud Allow remote debugging Free Tool! N L SeeTestCloud Device Lab located onsite (in the customer VPN) Devices can be reserved and accessed by any remote user Used in conjunction with SeeTestAutomation/Manual for testing of remote devices SeeTestNetworkVirtualization Device performance under varying network conditions Simulating mobile network bandwidth latency and reception while on the move SeeTestLoad Real device generated mobile load Full integration to LoadRunner, Jmeter, etc. Mobile Add-On for LoadRunner Full integration to LoadRunner through SeeTestAutomation

24 Demo

25 Mobility Automation WIPRO LTD CONFIDENTIAL

26 In Mobility Automation is the Key!! Why is automation a must? More releases in a year Parallel Tracks- Phone/ Tablet/ Web Less time to test Numerous Devices/ Platforms/ Versions Not just UI Testing- Services/ Backend End to End Testing WIPRO LTD CONFIDENTIAL

27 Wipro Automation Solutions Overview 1. Early Automation Support Agile & BDD projects Faster Testing by in- sprint automation Optimize testing cost by expanding automation coverage 3. End to End Automation Services & Back end validation Test in various environments by pointing to different systems behind the scenes High quality apps 5. Parallel Execution Faster test execution Shorter test cycles & more release support 2. Platform Agnostic Multi platform and device support Reusable libraries Jump start automated testing in new projects Reduce testing costs 4. Unattended Testing DevOps adaptation Fully Automated Integration with Test Management & CI solutions Wipro Automation Framework WIPRO LTD CONFIDENTIAL

28 Early Automation Approach Dev-ops or CI/CD In-sprint automation Accelerated Testing 1 3 Analyze Wireframes Application Flow Objects in each screen Automatable scenarios Execution Ready Map actual Objects to Object repository 5 Re Engineer Do modifications to the scripts to rectify the issues found in dry run and make it stable 2 Develop Automation Scripts Develop Framework Build reusable functions/ pseudo scripts Object repository mapping table ( Key->Value pair) 4 Dry Run Ensure modified scripts are running fine 6 Execution Start test case execution on test devices WIPRO LTD CONFIDENTIAL

29 Un-Attended Testing DevOps support Quick testing Integration with existing test management solutions Checks in Code Automated Build Automated Deployment Automated Reporting Automated Testing WIPRO LTD CONFIDENTIAL

30 Demo Framework Highlights Demo In brief Build on Java, leveraging SeeTest Supports keyword driven and data driven approach Supports parallel execution for faster testing Easy to configure and use even by a manual tester Supports early automation approach Supports integration with 3 rd party test management tools Supports un-attended testing Configure the environments via environment sheet Write the test cases using Excel based template using keywords Demonstrate un-attended test execution using Jenkins by automatic deployment of build Demonstrate parallel execution using SeeTest on one Android and ios device Integration with HP ALM Updating results Demonstrate standalone reports WIPRO LTD CONFIDENTIAL

31 Case in Point: Complete Mobile Automation- Leading Bank in US! Problem Statement Testing of Bank s mobile application on ios and Android devices A flexible and highly reusable automation framework which supports unattended testing and integration with existing test management tool (HP ALM) Backend mainframe validation for transactions from mobile devices Reduce testing cycles and improved ROI for mobile automation Wipro Solution Leveraged and customized Wipro mobile automation framework which supports: Platform agonistic scripting Data driven Highly reusable Parallel automation execution Leveraged SeeTest partnership to build a Mainframe validation component and incorporated in to Wipro framework Integrated Wipro automation framework with Jenkins and HP ALM for un attended testing with SeeTest as the automation tool Implemented in-sprint automation using Wipro automation framework Implemented Parallel automation test execution using SeeTest for faster testing cycles Benefits Up to 40%* productivity gains with automation in comparison with manual testing Continuous integration using Jenkins and enhanced reporting capability by automated results being updated in HP ALM Reduce testing cycles using early automation and parallel execution Shift-left automation script development by improving ROI for mobile automation Automated backend validation for each transaction Easily extendable to other OS platforms if needed WIPRO LTD CONFIDENTIAL * Based on 60% automation and early automation approach

32 Case in Point: Automation Productivity Enhancement- US Retailer! Problem Statement Testing on ios and Android devices High automation script reusability across customer mobile applications Build an automation framework which is easy to use and allows non-technical testers to do automation Faster testing cycles and improved ROI for mobile automation Enhanced automated test reports based on customer requirement Reduce overall testing cost by reducing as much as commercial tools Flexibility to run automation across various test environments Wipro Solution Developed a Java based automation framework which allows writing of test cases using Excel sheets which enables even non-technical testers to do automation Core technical team writes new keyword functions Framework supports parallel execution by configuring the devices in the configuration file Custom made reports for holistic view of test execution across all devices in a single report Integrated with Jenkins for Continuous Integration support Implemented Parallel automation test execution using SeeTest for faster testing cycles Benefits Jump start automation by enabling non technical testers to do automation using a simple excel spreadsheet Improved automation development productivity using excel based framework Flexible and highly reusable automation framework which is completely configurable via excel sheets Shift-left automation script development by improving ROI for mobile automation WIPRO LTD CONFIDENTIAL

33 ! Case in Point: Enhancing Customer Experience for a Leading US Retailer Problem Statement Reduce production defects reported by end users ( Performance issues, app freezing, no-product found) Wipro Solution Tested app under various network conditions and identified the real world issues faced by real users Conducted usability testing to analyze the application from an end user perspective and identified the usability issues Benefits Improved app ratings from 3 to 4.5 (ios) & 1 to 4.3 (Android) Improve app rating of existing mobile applications Ensure seamless customer satisfaction Ensure latest/upcoming features & technologies like ibeacon, barcode scanning, iwatch, etc. Conducted app profiling to find out the resource utilization of mobile application Conducted network monitoring to ensure optimum network data transfer Conducted field/in-store testing to ensure seamless end user experience Simulate network load and backend server load and identified the issues faced by user in peak load times Conducted app comparisons to identify the issues and benchmark the app against other competitors Rated as one of the fastest growing retail mobile apps Stable application under real world scenarios WIPRO LTD CONFIDENTIAL

34 Thank you Sudheer Mohan WIPRO LTD CONFIDENTIAL