Mobile App Development: The CD Recipe Jenkins + Functional and Non-functional Testing + Real Devices. Carlo Cadet, Director, Technical Evangelists

Transcription

1 Mobile App Development: The CD Recipe Jenkins + Functional and Non-functional Testing + Real Devices Carlo Cadet, Director, Technical Evangelists

2 Introducing Perfecto Mobile Enabling manual and automated mobile application testing as well as application monitoring for increasingly agile delivery processes Getting Noticed Perfecto Mobile marks shift in mobile apps QA with Continuous Quality Lab Good Friends At a glance Founded in late 2006, Business HQ in Boston and R&D in Israel Proven management team with over 200 employees globally Funded by leading VCs, Total - $57.8M

3 Overview Objectives Mobile changes test planning Describe CI challenges with mobile Describe a working pattern Share some examples Share best practices for extending CI to mobile projects

4 Time to Market Value Agility Innovation

5 Time to Market Value Agility Innovation

6 RaboBank Digital Portfolio Current online presence - Website for desktop and for mobile - Native app for ios, Android and Windows Phone Platform #users # login / month Desktop 3,6M 18,6M App ios 1,2M 30M App Android 1,2M 30M App Windows Phone 0,02M 0,3M Development - 1 new hybrid environment that will replace all native apps and the website for desktop created by 10 scrum teams - Total of 18 scrum teams, 200 FTE - Core of a team: Development, PO, QA, Operations - Extended team: BA, Appl Eng, project mngt

7 Support programming Automation Strategy Automated Functional tests (system (integration) & chain, 200 portability testing 10% x 10 devices) Automated Unit testing Unit integration testing Business facing Technology facing Regression testing is cross-category Exploratory testing User acceptance testing Accessibility Visuals (partially) Manual Non-functional testing (performance, security, production acceptance testing) Manual/automated Critical projects

8 Easily Applied for Browsers UI Integration Unit Testing Mobile Create Challenges

9 90 Minute Commute Check Facebook Skip that OS update Watch YouTube videos Clear easy s Start expense reports Answer the phone Finish expense reports Order a gift on online Send several text messages Accept LinkedIn invite

10 Mobile Requires Shifting the Plan OS OS versions Form factors CPU & Memory Manufacturers Wearables IOT Target Devices Performance UI Security Integration Unit Testing Target Env. Networks Orientation Location Notifications Events Device events

11 Our Working Cookbook Continuous Feedback Align QA tools to Dev Shared Lab Continuous Integration

12 What Why How CI Blockers: Adding Real Devices Struggling to add real devices to the CI process Uncertain device availability Test readiness and configuration control of real devices Managed lab Devices SLA

13 A Lab that provides 1. SLA assured devices availability 2. Parallel execution capability 3. Resource abstraction

14 Best Practice: Select Devices By Attribute Select targets by attribute

15 What Why How CI Blockers: Inefficient Test Development Developing UI automation takes too long Automation requires constant maintenance Scripting per platform Skillset - shifting from scripting to coding Easy object identification Develop framework Reuse predefined code functions Write cross platform test code

16 Best Practice: Collaborate for Test Friendly Code Catalog objects Object Properties Inherit Properties Perform Actions

17 Best Practice: Cross Platform Test Code Cross platform execution

18 What Why How CI Blockers: Excessive Execution Failures > 25% execution failures and errors Unexpected notifications Environment readiness Device errors Separate test code & execution mgt Perform pre-test readiness checks Add Error /Event handling

19 Best Practice: Graceful Test Recovery Manage Mobile Events to Reduce False Negatives System level control Application control Device control

20 What Why How CI Blockers: Poor Execution Performance UI Automated test execution exceeds available test window All the tests wait for the device (resources issue ) Lack of execution management Hardcoded Waits Apply automation best practices Parallel execution Use sync mechanism

21 Best Practice: Parallel Execution NativeDriver.manage().timeouts().implicitlyWait(15, TimeUnit.SECONDS); NativeDriver.manage().timeouts().pageLoadTimeout(15, TimeUnit.SECONDS); Implement dynamic waits & Parallel Execution

22 Manage the Time Budget Build Frequency Test Duration Test Scope Unit System Smoke Regression Performance Device Coverage Hourly Min ±4 3-4 times a day Min ±6 Nightly 2-7 Hours ±10 Weekend Hours ±20

23 Recommendation's Automation requires an available lab Maintain test code guidelines Error & event handling Framework maintenance Readiness checks Externalize Resource mgt Optimize execution Multi-threaded execution

24 By the Way

25 Uzi Eilon Director Technology Carlo Cadet Dir, Technical Evangelists

26 Continuous Quality Lab for a full App Delivery Lifecycle Continuous Feedback and Reporting Dev Team Bug Tracking Source Control Parallel Test Execution Functional Testing Integration Testing Performance Testing Automated Deployment on Real Devices Reporting Testing and Monitoring Srvcs Wind Tunnel MobileCloud Platform The Continuous Quality Lab CI Server Launch Perfecto Job (Release Package)

27 Nightly CI builds Full regression (ios & Android) - Future BATs (IOS & Android) Daily / 5:30 AM / across 4 VMs/cradles Jenkins App Builds (5 AM) copied to Perfecto Cloud / Post Builds Perfecto Connectivity Daily / 4:45 AM Web Service Regression Daily / 4:30 AM & On-Demand Web Service Connectivity Daily / hourly & Post Deploys Majority of Business Logic is within the Web Services Slide 27 Test Automation Engineering Group Paychex 2010 All Rights Reserved

28 Continuous Quality (to-be) Continuously Check-in Build Deploy to TST Test Deploy to ACC Acceptance Production Done Check-in code & review Code review Check-out & Build application Unit Test Quality code analysis Security/Perf. code analysis Deploy to TST Smoke test on TST Regression Test System Test Portability Test Pre-PAT Performance Test Exploratory test Deploy to ACC Smoke test on ACC PAT Chain Test Security UAT Schedule P- deployment Deploy to PROD PROD verification Automated step Deploy stories when done, not per sprint schedule Manual step

29 Tuning the Process for Mobile Proprietary & Confidential