Fast Feedback: Jenkins + Functional and Non-Functional Mobile App Testing Without Pulling Your Hair

Transcription

1 Fast Feedback: Jenkins + Functional and Non-Functional Mobile App Testing Without Pulling Your Hair Uzi Elion, Tecnology Director Carlo Cadet, Director, Technical Evangelists

2 Tuning the Process for Mobile Proprietary & Confidential

3 Overview Objectives Mobile changes test planning Describe CI challenges with mobile Describe a working pattern Share some examples Share best practices for extending CI to mobile projects

4 Introducing Perfecto Mobile Enabling manual and automated mobile application testing as well as application monitoring for increasingly agile delivery processes Getting Noticed Perfecto Mobile marks shift in mobile apps QA with Continuous Quality Lab Good Friends At a glance Founded in late 2006, Business HQ in Boston and R&D in Israel Proven management team with over 200 employees globally Funded by leading VCs, Total - $57.8M

5 Easily Applied for Browsers UI Integration Unit Testing Mobile Create Challenges

6 90 Minute Commute Check Facebook Skip that OS update Watch YouTube videos Clear easy s Start expense reports Answer the phone Finish expense reports Order a gift on online Send several text messages Accept LinkedIn invite

7 Mobile Requires Shifting the Plan OS OS versions Form factors CPU & Memory Manufacturers Wearables IOT Target Devices Performance UI Security Integration Unit Testing Target Env. Networks Orientation Location Notifications Events Device events

8 What Why How CI Blockers: Adding Real Devices Struggling to add real devices to the CI process Uncertain device availability Test readiness and configuration control of real devices Managed lab Devices SLA

9 The answer a Lab that 1. SLA assured devices availability 2. Parallel execution capability 3. Manage by attributes capability

10 Best Practice: Select Devices By Attribute Select targets by attribute

11 What Why How CI Blockers: Inefficient Test Development Developing UI automation takes too long Automation requires constant maintenance Scripting per platform Skillset - shifting from scripting to coding Easy object identification Develop framework Reuse predefined code functions Write cross platform test code

12 Best Practice: Collaborate for Test Friendly Code Catalog objects Object Properties Inherit Properties Perform Actions

13 Best Practice: Cross Platform Test Code Cross platform execution

14 What Why How CI Blockers: Excessive Execution Failures > 25% execution failures and errors Unexpected notifications Environment readiness Device errors Separate test code & execution mgt Perform pre-test readiness checks Add Error /Event handling

15 Best Practice: Graceful Test Recovery Manage Mobile Events to Reduce False Negatives System level control Application control Device control

16 What Why How CI Blockers: Poor Execution Performance UI Automated test execution exceeds available test window All the tests wait for the device (resources issue ) Lack of execution management Hardcoded Waits Apply automation best practices Parallel execution Use sync mechanism

17 Error handling Remove CI/Automation Blockers NativeDriver.manage().timeouts().implicitlyWait(15, TimeUnit.SECONDS); NativeDriver.manage().timeouts().pageLoadTimeout(15, TimeUnit.SECONDS); Implement dynamic waits & Parallel Execution

18 Manage the Time Budget Build Frequency Test Duration Test Scope Unit System Smoke Regression Performance Device Coverage Hourly Min ±4 3-4 times a day Min ±6 Nightly 2-7 Hours ±10 Weekend Hours ±20

19 Our Working Cookbook Automation Test Framework Manual Where Must Apply Real User Conditions Early Non-Functional Align QA tools to Dev Shared Lab Continuous Integration

20 Recommendation's 1. Share tools 2. Maintain test code guidelines 3. POM maintenance 4. Error & event handling 1. Readiness checks 2. Externalize Resource mgt 3. Optimize execution 4. Multi-threaded execution

21 By the Way

22 Continuous Quality Lab for a full App Delivery Lifecycle Continuous Feedback and Reporting Dev Team Bug Tracking Source Control Parallel Test Execution Functional Testing Integration Testing Performance Testing Automated Deployment on Real Devices Reporting Testing and Monitoring Srvcs Wind Tunnel MobileCloud Platform The Continuous Quality Lab CI Server Launch Perfecto Job (Release Package)

23 Nightly CI builds Full regression (ios & Android) - Future BATs (IOS & Android) Daily / 5:30 AM / across 4 VMs/cradles Jenkins App Builds (5 AM) copied to Perfecto Cloud / Post Builds Perfecto Connectivity Daily / 4:45 AM Web Service Regression Daily / 4:30 AM & On-Demand Web Service Connectivity Daily / hourly & Post Deploys Majority of Business Logic is within the Web Services Slide 23 Test Automation Engineering Group Paychex 2010 All Rights Reserved

24 Continuous Quality (to-be) Continuously Check-in Build Deploy to TST Test Deploy to ACC Acceptance Production Done Check-in code & review Code review Check-out & Build application Unit Test Quality code analysis Security/Perf. code analysis Deploy to TST Smoke test on TST Regression Test System Test Portability Test Pre-PAT Performance Test Exploratory test Deploy to ACC Smoke test on ACC PAT Chain Test Security UAT Schedule P- deployment Deploy to PROD PROD verification Automated step Deploy stories when done, not per sprint schedule Manual step

25 Uzi Eilon Director Technology Carlo Cadet Dir, Technical Evangelists