Mobile Test Automation: Lessons Learned in the Trenches

Transcription

1 T4 Track 10/4/2012 9:45:00 AM Mobile Test Automation: Lessons Learned in the Trenches Presented by: Manish Mathuria, InfoStretch Sha Mohammed, Sabre Airline Solutions Brought to you by: 340 Corporate Way, Suite 300, Orange Park, FL

2 Manish Mathuria InfoStretch Manish Mathuria has more than fifteen years of professional and management consulting services experience. As the CTO and leader of the mobile testing practice at InfoStretch, Manish leads the team in developing architecture, scalability, performance, and IT strategies for customers, especially in the mobile domain. InfoStretch provides certification and testing services to some of the leading national and international app stores. Manish and his team help optimize major global app stores testing and certification processes. They have tested and certified more than 50,000 mobile applications in the InfoStretch Mobile Lab. Manish speaks regularly at leading quality and software engineering conferences worldwide.

3 Sha Mohammed Sabre Airline Solutions Mr. Sha Mohammed, Director Software Quality Engineering at Sabre Airline Solutions, has over 13 years of experience in Software/IT industry in both product development and consulting roles. Sha has deep knowledge and experience in software testing and successfully delivering high volume, high availability ecommerce applications for air travel and finance industries. He carries an excellent track record in building and sustaining high performing teams and supporting infrastructure and logistics in large and complex multi-national corporations. Sha has also had remarkable success in finding the right mix between deploying proven traditional methods as well as innovative cutting edge technology, tools and processes to enable faster Time-To-Market with great quality at the right cost. Prior employers include Fidelity Investments, Ariba, American Express & EMC2.

4 1 1 Mobile Test Automation Lessons from the Trenches Manish Mathuria CTO InfoStretch Corporation Mohammad Sha Director of QA Sabre Airline Solutions [email protected] Lakeside Drive, Ste 200, Santa 2 AGENDA Mobile Landscape and Testing Challenges 1 Mobile Automation Best Practices QUESTIONS Mobile Test Automation Tool Categories Case Study Sabre Airline Solutions [email protected] Lakeside Drive, Ste 200, Santa 1

5 MOBILE LANDSCAPE MOBILE TESTING CHALLENGES 3 4 Mobile on Fast Track Do you know how many babies are born each day? 371,000 DAILY Impressive, until you realize the number iphones sold 377,900 DAILY Or the number of Android phones activated. 700,000 DAILY And letʼsnot forget the growth of tablets. 191,000 DAILY [email protected] Lakeside Drive, Ste 200, Santa 2

6 Mobile Shorter Life Cycle Risk Gap from requirements Short Cycle for Mobile Traditional Mobile Mobile Dev. Traditional SW/PC Short Cycle Mobile dev. Time SW Cycles Mobile ALM Process Continuous QA Market 6 years Browsers Product Version Product Release timeline 6 years 3

7 Market Product Version Product Version 1 year 2011 Product Version Product Version Android V2.3 V2.3.3 V3 V2.3.4 V3.1 V2.3.5 V3.2 V2.3.7 V2.3.6 V4 Product Version Product Release timeline 1 year MOBILE TESTING IS DIFFERENT 8 4

8 THE CHANGE FROM TRADITIONAL TESTING Many platforms (Blackberry, Android, ios, S60, WP7, etc ) Many form factors (touch, keyboard, slide, landscape, etc ) Many networks (3G, 4G, Wi-Fi, Wi-Max, etc ) Mobile is fragmented More platform changes (8 major version changes in 2010) Handset life is short (6-18 months) Device Capability? How to test location based services? Dynamic Carriers/Networks/Devices? Mobile is dynamic Many handsets Actual networks Accounts, SIM, Access - expensive, time consuming and resource intensive Usability, network connectivity, call interruptions, and device providers Mobile requires logistics 9 NEW NEEDS + CHALLENGES FROM TRADITIONAL TESTING Testing Challenges How to identify which devices to test on to support your user base? How to identify device limitations and characteristics for feature testing? How to report observations during testing on devices to developers? How many versions of Mobile OS tests need to be repeated? How to find out unique features of each device? Offline-Online Content and caching? Connectivity to native devices peripherals Software Development Cycle Mobile SDLC is in its nascent stages No set quality and development process; Agility required Location based, orientation, social media integration When to do full test vspartial test? Different methods to deploy - making testing more complex Ad-hoc testing beyond what is documented Enterprise Testing Challenges Difficult to test in the field No VPN on old devices How does the app deal with the backend system changes? How to push updates and test them? No security standards Minimum vulnerability database for apps as well as mobile browsers No standard privacy policies and laws set 10 5

9 TESTING LIFECYCLE ENTERPRISE MOBILE TESTING LIFECYCLE Manage Requirements Optimize Quality Optimize Performance + Monitoring Business Process Definitions Business Requirements Platforms, networks And UX requirements Test Requirements Test Strategy And Plan Define Use Cases Test Analysis DEFECTS Capacity Planning Platform Functional Test Exec Regression Manual Platforms Platforms + UX Testing Capacity Planning Diagnostics ISSUES Test Creation and Maintenance Performance Testing Timing Networks Load Platforms App Develop Collaborate Mobile Specific Web Specific Mobile + Web Specific MOBILE AUTOMATION MOBILE AUTOMATION TOOL CATEGORIES Mobile HTML5 Based Automation Native Platform Automation Technologies Platform Independent Mobile Automation 12 6

10 TOOL CATEGORIES MOBILE AUTOMATION TOOL CATEGORIES Mobile HTML5 Based Automation Native Platform Automation Technologies Platform Independent Mobile Automation What Drives HTML and JS constructs RecognizesWeb Controls Drives GUI widgets and Controlson the Native UI Layer Leverages screencoordinates, images and OCR to drive automation Pros Cross device platforms Cross browsers Object aware automation Non Intrusive to App Ultimatecontrol as good object level UI layer access Crossplatform scripting, one script many devices. Has access to the whole device Cons Tools Limitedto Web and Cross Platform HTML5 Apps Selenium Webdriver QUnit Intrusive, often requires code access Written specifically for a device platform Limited to the context of the application Robotium Frank UIAutomation MonkeyTalk Limited object awareness, reliance on image and OCR Less effective for complex automation PerfectoMobile See Test (Experitest) Device Anywhere MOBILE AUTOMATION LESSONS FROM THE TRENCHES 14 7

11 15 Lessons Learnt How to Select Test Cases and Devices for Mobile Automation? 1 Dealing with Fragmentatio n Deal with Scripting Challenges Test Execution Challenges [email protected] Lakeside Drive, Ste 200, Santa TEST CASE/Devices SELECTION How do you identify the test cases for Mobile Automation? Complexity Medium complexity test cases Provide coverage for device compatibility tests Devices Selection Jail broken or not? Market relevant Screen resolution coverage OS Coverage Tool and Resource Capability System Interaction Peripheral Interaction Interaction between multiple apps/ OS-App /Multi Domain Location aware, peripheral, battery life, system alerts etc. Stability/Change UI components that will change less. Clear understood business processes Best Practices for other GUI Automation apply 16 8

12 Scripting Challenges 17 Widgets will render Different even on same OS Android 2.3 Left: Samsung Right: Sony Ericson 9

13 Form Factor Challenges Droid X2 Droid Charge iphone 4 Test on the actual target devices Desktop Browser Droid Charge 10

14 Dealing with Synchronization Image Scan & Analysis Yes Needle Found? No No Timeout Exceeded? Yes Exit With Success Exit With Failure 21 Dealing with Fragmentation Design your test cases for exceptions! Browser Browser Go Open To Sign On Make Payment Text Verify Confirm Payment Sign Off QTP/Selenium Make Payment Test case Address ATM Search Browser Open Select Acct View Schedule Payment Sync Text Quick Links Nick Name Browser Go To Sign On Sign Off ZIP ATM Search Make Payment Sync Image Confirm Payment Framework Virtual Function Library ios Android HTC Aria Android HTC Droid BB Torch BB 8520 LG CU620 LG VX9000 Devices Device Implementations for Sign On (10+) Device Implementations QTP Testcases Virtual Functions 22 11

15 Execution Best Practices 23 Invest in Continuous Integration (CI) Automating the Automation What is CI? Automate execution of test cases Integrated automated reporting, and notifications Prominent tools Jenkins, Microsoft TFS Invest in a Test Automation lab of devices 24 12

16 Our CI Integration Test Management Tool Automating the Automation Test Framework e.g. QMetry / HP-QC Continuous Integration Check-in Test Runner Setup Launch Reporting Teardown e.g. TestNG Test Assets Optimize Report Test Build Test Reports e.g. Hudson / Jenkins e.g. ReportNG 25 Develop Strong Test Recovery Automating the Automation Automation Tests will break due to Device characteristics Test script errors Automation tool instability Device instability Data instability Invest in Robust Test recovery system Develop agents that can restart text components on devices Enhance test case recovery, capability to restart test case

17 Sabre Case Study 27 Business Problem Automated a mobile web based application that enables the ability for passengers to book and manage travel on their ios and Android mobile devices through a simple and intuitive process flow. Development and Manual Testers defined 33 user scenarios test cases for automation to detect regressions. Test cases are categorized by a. Book My Flight b. Manage My Flight c. Check Flight Status d. Feature Function Test integrated testing with the above 3 categories Automation starts as soon as Manual tests verify each functionality. Application compatibility testing with system notifications are manually tested. 14

18 Test Environment Automation Tool Selected: Selenium Remote Android and IOS web driver. Automated tests limited to the Sabre web application only. Selenium based solution does not support automated application compatibility i.e. system alerts phone calls while application is running and Blackberry devices -Perfecto Mobile test automation can be used in this case. Selenium Test Suites Selenium Test Suites Test on Firefox first as baseline, then move to mobile devices Webdriver installed on each mobile devices Sabre Test Code Repository Jenkins CI Selenium Test Suites Monitor for regression Some Technical Challenges Faced Element ID Discovery Test code dependent on locator id changes and UI flow changes especially in agile scrum development environment. Colocation is critical to promote communication. Xpathnot supported on all mobile devices. Selenium Web driver connection reset when executing whole test suites on ios and Android devices on a single run. Web driver not responsive at times causing timeouts. Still maturing -monthly releases On Android Android Web driver delay in populating in list box values. Problems with launching web applications on Android emulator. Compatibility across various devices i.e. On Android 2.3 OS -Samsung Galaxy and Sony Ericson have different list-box implementations (see next slide) On IOS ios Web driver does not always reset after test suite run and need device restart. Installing Web driver on ios devices requires jail broken device and valid provision profile on device. 15

19 Lessons Learned Automation team to be co-located with development team especially with agile scrum teams. Establish policy to ensure locator ids backward compatibility between sprints. Questions? Due to the variability of devices and web driver behaviors, architect your automation with this theme when things change, change it in ONE place. Automation scripts should coordinate the highest level abstraction, (libraries) whenever possible to maintain agility and robustness across devices. Agreement on list of devices / vendors to be supported. Not all devices will be have the same way. Selenium ios and Android web drivers are still evolving -expect to do debugging to workaround web driver issues. 16