Total Quality in Mobile Apps -Continuous Integration & Performance Testing

Transcription

1 Infosys & Experitest Co-webinar: Total Quality in Mobile Apps -Continuous Integration & Performance Testing Nov 2014

2 Part I How To Use Continuous Integration for Mobile Speaker: Guy Arieli, Experitest CTO

3 Agenda Continuous Integration is especially important for mobile 1 Why is it important? 2 How is it different from regular Continuous Integration? 3 What is the recommended flow for Continuous Integration in mobile? 4 Case study: major bank 5 Demo: mobile continuous integration Jenkins

4 What Is Continuous Integration? Developers Source Code Repository CI Server YOUR APP Software development practice where members of a team integrate their work frequently Each person integrates at least daily leading to multiple integrations per day Significantly reduces integration problems and allows a team to develop cohesive software rapidly CI significantly reduces integration problems and allows a team to develop cohesive software rapidly

5 Why is CI especially important for mobile apps? Fast app release Mobile app development is faster End-user expectation for fast response time Frequent external environment changes New OS versions Huge device matrix Higher ROI because of mega matrix Many (hundreds) devices to be tested Several mobile OS to be tested New devices High adoption of Agile methodology in R&D

6 How is mobile CI different? Complex management of Continuous Integration Smart device allocation Dual application mode (instrumented, non-instrumented) Parallel execution

7 Continuous Integration For Mobile Developers x Test Results Source Code Repository Execute Test On Devices CI Server Executer Agent YOUR APP

8 Recommended process for mobile Continuous Integration View and analyze reports Prepare the application for automation Reserve the devices to be used Deploy application and device preparation Tests Execution Build the application

9 Build the Application YOUR APP Build the Application Source Code Repository CI Server New Build of Your App For example: If it s an Android build, you will get a URL for downloading this APK file Trigger the entire execution

11 Prepare the Application (optional step) Application under test may have two modes: YOUR APP YOUR APP Instrumented more control and more capabilities over the application Tips for Implementation Test automation tool should enable: 1 Unattended, on-the-fly instrumentation 2 Perform the instrumentation on the same machine that is being used for automation New Build of Your App YOUR APP Non- instrumented 3 Windows machine to perform all instrumentation (inc. on-the-fly) (also in the case of an ios app)

12 Dynamic, smart reservation of the Devices to be used Execute your regression of tests on as many devices as you can Cloud of Devices Smart Device Reservation Dynamic, smart android > 4.2 htc

14 Deploy Application and Device Preparation Have full control of the device and of the application installation Working with the Settings Uninstallation of applications Device reboot Installation of applications Application clear data Launch of applications **These operations should be available and should be able to run entirely unattended.

16 Tests Execution Actions for making Test execution efficient: Capability to run the tests in parallel on different devices Decupling between the CI server and the execution machine Queue of tests that are waiting for devices the be available T1 T2 T2 Execute the Devices T2 Tn CI Server Executer Agent Remote Mobile device Lab

18 Recommended process for mobile Continuous Integration

19 Demo How to use Jenkins for mobile Continuous integration

20 Case Study

21 Customer Profile: Multinational banking and financial services company Major global financial services provider engaged in personal banking, credit cards, corporate and investment banking and wealth and investment management. Headquartered in APAC Spans over 50 countries across Europe, America, Africa & Asia ~200,000 employees Mobile applications in tens of languages Ranked in top 10 largest banks worldwide Revenue more than $30 Bn.

22 The Need Testing end users' apps The bank s Personal and Business accounts management apps Wallet apps Trading apps Mortgages apps BankCard Many more Testing the Internal bank Employees Apps Applications used by The bank employees providing services to customers such as: Loan qualification form Mortgage requirements form Etc.

23 The Requirements On-premise devices, all within the bank VPN Cross-platform testing on all OS: Android, ios, WindowsPhone8 & BlackBerry </> No need for access to application source code for testing (on-the-fly instrumentation) Support for multi-field form filling including scrolling in lists, invisible objects, etc. Ability to test on devices located in one of the bank facility by testers located in other locations

24 Implementation Years working with the tool: 3 Over 100 SeeTestAutomation stations Number of Automation engineers working on the project: geographical locations around the world (US, Europe, Africa and China) Number of tests before: 0 Number of tests today: ~20,000 Number of executed tests: ~200,000 (each test can run on more than one device) Onsite deployment with integration into the existing test environments QTP, Junit, Eclipse QC and Jenkins Parallel execution of tests

25 Coverage Total coverage: ~90%. Number of executions of the full cycle: Once a day Number of apps tested: +20, (different application in different regions) Coverage of both customer s apps and internal systems apps Full app test coverage for all OS-Cross platform testing of Android ios WindowsPhone8 BlackBerry Full end-to-end coverage of test scenario Support for multiple languages (including English, Spanish, French)

26 ROI What if we did it manually? A single QA engineer: 56 man years to finish the cycle (lifetime). For one day we would need 20,000 trained manual QA engineers. Another Advantage The R&D engineers can work faster and take more risks According to the bank R&D managers the development speed is 3-4 times faster!

27

28 The Tools Used SeeTestAutomation Automation tool Same test runs on different devices Plugs into all testing & CI environments: UFT/Selenium/ Microsoft Visual Studio/ RFT/Java/C#/TestComplete//Perl/Python/Ruby QC/TFS/Jenkins/TeamCity/RQM Manual to Automation Script Convertor SeeTestManual Free Tool! Manual testing tool Mini-recorder for automating short, repetitive, manual scenarios Report with screenshots of each step SeeTestCloud Device Lab located onsite (in the customer VPN) Devices can be reserved and accessed by any remote user Used in conjunction with SeeTestAutomation/Manual for testing of remote devices

29 Thank you

30 Infosys Point of View (POV) on Mobile Application Performance Testing 17 December, 2014

31 Performance is key, for a Mobile user Frequent crashes and glitches Slow loading times High battery drain Unreliability Inefficient start and exits Analysis reveal that majority of 1-star reviews are related to performance issues, proving that Quality is key for an end-user. Other Poor App Store ratings are often linked with poor app performance % of Critical, 1-Star Reviews For Top ios Apps 48% 52% Performance Related CRITICAL, 1-STAR REVIEWS 0 Lagging, Performance, Hot, Battery Drain Slow, Long Load, Load Fails Crash, Bug 9% 15% 76% PERFORMANCE RELATED, CRITICAL, 1-STAR REVIEWS Source: appurify Forrester: Mobile apps live and die by their ratings in an App Store When the rating suffers, customer adoption suffers. IMS, Goldsmiths: "Users experience a lot of negative emotions and frustrations when trying to complete some digital tasks and apps are slow to load and respond" Apigee: 44% of users delete a mobile app immediately if it didn t perform as expected; Performance is reported to be especially important for banking (74%) apps realbusiness: Consumers are unforgiving towards slow and poor performing apps.

32 If left unaddressed, the impacts are significant Lost transactions not only result in lost revenue for banks, but have a detrimental impact on customer loyalty and brand reputation it's a lose-lose situation. 38 percent try another app when faced with a problem, 34 percent stop using the app, and 19 percent complain to friends and family Majority users delete or uninstall mobile app because of performance problems. About 18% delete the app immediately if it froze for just 5 sec. 38% delete the app if it freezes for longer than 30 seconds 30% of users would spend more money with an organization that had a good mobile app 29 percent would pay more for a product or service if the organization s app performed better than its competitors Nearly one third of smartphone and tablet owners would change banks if a mobile app wasn t up to par

33 Addressing Customer Reviews on Mobile apps.. 33 One in six customers who switch banks indicate a poor mobile banking experience prompted the switch Between 51% and 76% of mobile users use their mobile apps to check balances, transfer money, pay bills, and look up branch locations and hours. Hence reduced performance leads to greater & immediate customer dissatisfaction Problem Cause Solution Application is buggy (doesn t open, crashes, freezes often). Sloppy code, varying internet connection, poor memory management. Thorough testing of the app under multiple conditions and variables to ensure that an app is devoid of issues Infosys Performance Testing Solution Client Application Performance Server Side Performance Network Impact on Performance

34 Device Impact Testing / Client Application Performance 34 Key Measurement Parameters: Battery Consumption Device CPU and Memory constraints Device I/O Assess speed for read and write to devices Tools: - Xcode Instruments (memory allocation, leaks, CPU usage ) - Android ADB (battery consumption, CPU usage / processing speed) - Android DDMS ( memory usage, device I/O, track running thread info..) - Quadrant (only for Android) - Neoload

35 Server Side Performance / Load Testing 35 Key Measurement Parameters: Response Time Error Rate Throughput Server CPU utilization Server Memory utilization Tools: - LoadRunner - Neoload - CloudTest Mobile Performance Tool Recommendation: Ability to support multiple devices Ability to support native applications as well as browser-based applications Ability to record through the device using agents, or through emulators, as well as directly from a browser to reduce complexity Ease of use and fast turnaround time

36 Network Impact Testing 36 Network Emulation Options: Network Coverage - Occasional disconnection of the network. Packet Loss & Duplication Packet loss & duplication for an application session by dropping packets based on preset values Delay and Jitter - Delay of data originating from app. Multi-Network testing (Roaming) Switching from one type of network to another type of network e.g. (3G, 4G, LTE) Key Measurement Parameters: Response Time Exception Handling UI Issues Consistency Tools: - Infosys Network Impact Testing Solution (inits) - Shunra - Neoload Mobile Performance Tool Recommendation: Ability to dynamically emulate various network conditions Ability to generate and or emulate load from various locations, leveraging the cloud Ability to test regardless of the protocol the application uses

37 Some of the Performance Testing Best Practices for Mobile Apps 37 Test early: Do quick check on mobile web application performance, using page analysis tools like PageSpeed Insights, Yslow etc. For native apps, developers can perform regular checks with Instruments / DDMS tools Verify & validate the performance on a real device Perform both modular and system performance tests Correlate server-side metrics with client-side response times. It helps nailing down the root cause May look at reusing the functional scripts for load Testing (e.g. SeeTestLoad -LoadRunner) Look for consistency in test results, during multiple test iterations Ensure complete performance test coverage, executing Load, Endurance, Stress and Single-user test cycles

38 Thank You 2013 Infosys Limited, Bangalore, India. All Rights Reserved. Infosys believes the information in this document is accurate as of its publication date; such information is subject to change withoutnotice. Infosys acknowledges theproprietary rightsofother companies to thetrademarks, productnames and such other intellectual property rights mentioned in this document. Except asexpresslypermitted, neitherthisdocumentation noranypartofitmaybe reproduced, stored in aretrieval system, ortransmitted in anyformorbyany means, electronic, mechanical, printing, photocopying,recording orotherwise, withouttheprior permission ofinfosyslimitedand/oranynamed intellectualpropertyrightsholdersunderthisdocument.