Fast Feedback: Jenkins + Functional and Non-Functional Mobile App Testing Without Pulling Your Hair Uzi Elion, Tecnology Director Carlo Cadet, Director, Technical Evangelists
Tuning the Process for Mobile Proprietary & Confidential
Overview Objectives Mobile changes test planning Describe CI challenges with mobile Describe a working pattern Share some examples Share best practices for extending CI to mobile projects
Introducing Perfecto Mobile Enabling manual and automated mobile application testing as well as application monitoring for increasingly agile delivery processes Getting Noticed Perfecto Mobile marks shift in mobile apps QA with Continuous Quality Lab Good Friends At a glance Founded in late 2006, Business HQ in Boston and R&D in Israel Proven management team with over 200 employees globally Funded by leading VCs, Total - $57.8M
Easily Applied for Browsers UI Integration Unit Testing Mobile Create Challenges
90 Minute Commute Check Facebook Skip that OS update Watch YouTube videos Clear easy emails Start expense reports Answer the phone Finish expense reports Order a gift on online Send several text messages Accept LinkedIn invite
Mobile Requires Shifting the Plan OS OS versions Form factors CPU & Memory Manufacturers Wearables IOT Target Devices Performance UI Security Integration Unit Testing Target Env. Networks Orientation Location Notifications Events Device events
What Why How CI Blockers: Adding Real Devices Struggling to add real devices to the CI process Uncertain device availability Test readiness and configuration control of real devices Managed lab Devices SLA
The answer a Lab that 1. SLA assured devices availability 2. Parallel execution capability 3. Manage by attributes capability
Best Practice: Select Devices By Attribute Select targets by attribute
What Why How CI Blockers: Inefficient Test Development Developing UI automation takes too long Automation requires constant maintenance Scripting per platform Skillset - shifting from scripting to coding Easy object identification Develop framework Reuse predefined code functions Write cross platform test code
Best Practice: Collaborate for Test Friendly Code Catalog objects Object Properties Inherit Properties Perform Actions
Best Practice: Cross Platform Test Code Cross platform execution
What Why How CI Blockers: Excessive Execution Failures > 25% execution failures and errors Unexpected notifications Environment readiness Device errors Separate test code & execution mgt Perform pre-test readiness checks Add Error /Event handling
Best Practice: Graceful Test Recovery Manage Mobile Events to Reduce False Negatives System level control Application control Device control
What Why How CI Blockers: Poor Execution Performance UI Automated test execution exceeds available test window All the tests wait for the device (resources issue ) Lack of execution management Hardcoded Waits Apply automation best practices Parallel execution Use sync mechanism
Error handling Remove CI/Automation Blockers NativeDriver.manage().timeouts().implicitlyWait(15, TimeUnit.SECONDS); NativeDriver.manage().timeouts().pageLoadTimeout(15, TimeUnit.SECONDS); Implement dynamic waits & Parallel Execution
Manage the Time Budget Build Frequency Test Duration Test Scope Unit System Smoke Regression Performance Device Coverage Hourly 20-40 Min ±4 3-4 times a day 30-60 Min ±6 Nightly 2-7 Hours ±10 Weekend 10-48 Hours ±20
Our Working Cookbook Automation Test Framework Manual Where Must Apply Real User Conditions Early Non-Functional Align QA tools to Dev Shared Lab Continuous Integration
Recommendation's 1. Share tools 2. Maintain test code guidelines 3. POM maintenance 4. Error & event handling 1. Readiness checks 2. Externalize Resource mgt 3. Optimize execution 4. Multi-threaded execution
By the Way
Continuous Quality Lab for a full App Delivery Lifecycle Continuous Feedback and Reporting Dev Team Bug Tracking Source Control Parallel Test Execution Functional Testing Integration Testing Performance Testing Automated Deployment on Real Devices Reporting Testing and Monitoring Srvcs Wind Tunnel MobileCloud Platform The Continuous Quality Lab CI Server Launch Perfecto Job (Release Package)
Nightly CI builds Full regression (ios & Android) - Future BATs (IOS & Android) Daily / 5:30 AM / across 4 VMs/cradles Jenkins App Builds (5 AM) copied to Perfecto Cloud / Post Builds Perfecto Connectivity Daily / 4:45 AM Web Service Regression Daily / 4:30 AM & On-Demand Web Service Connectivity Daily / hourly & Post Deploys Majority of Business Logic is within the Web Services Slide 23 Test Automation Engineering Group Paychex 2010 All Rights Reserved
Continuous Quality (to-be) Continuously Check-in Build Deploy to TST Test Deploy to ACC Acceptance Production Done Check-in code & review Code review Check-out & Build application Unit Test Quality code analysis Security/Perf. code analysis Deploy to TST Smoke test on TST Regression Test System Test Portability Test Pre-PAT Performance Test Exploratory test Deploy to ACC Smoke test on ACC PAT Chain Test Security UAT Schedule P- deployment Deploy to PROD PROD verification Automated step Deploy stories when done, not per sprint schedule Manual step
Uzi Eilon Director Technology uzie@perfectomobile.com Carlo Cadet Dir, Technical Evangelists carloc@perfectomobile.com