Integrate Testing into Continuous Delivery Needs of Phonak combined with solution from Kalistick Marc Rambert and Aliaksei Tsitovich @ Swiss Testing Day, March 19, 2014, Zürich
Agenda 2 Fitting Software Development at Phonak Challenge: Integrate Testing in Continuous Delivery at Phonak Solution: Test Coverage Intelligence Experience report from Phonak perspective Q&A
What is Fitting? 13.03.201Page 4 4 4 Gardening Potential for Sunflower Beautiful Sunflower Fitting Potential for Hearing Performance Happy Customer
5
Fitting Software Codebase is relatively big ~4 * 10 6 LOC of C# (mainly) 5 teams, 70 Software Engineers, 10 Test Engineers, 3 SETs TeamCity as CI, Polarion as ALM Internal release every morning Release every 6 month with certification
Continuous integration hides many untested behaviors
#1 Bugs stay unnoticed in uncovered behaviors Behaviors explored manually Your Behaviors executed by automatic tests code Zero probability to notice a bug
#2 Bugs are introduced by last-minute changes 10 Change request implemented Functional Regression set #1 Bug because of Last minute effect
We need feedback on uncovered lacunas 12 Automatic instantly with CI Combined build and manual test counted Detailed on modifications regressions should be highlighted Test Coverage Intelligence
Test Coverage Intelligence extracts data to improve testing 13 Application(s) 1 2 Learning System Scanner Modifications ++ ++ + Test Footprints CI Automated Tests ++ 3 Smart Engine Test Scoring Testing Holes
Detailed Impact Analysis for a whole system Test Team Test 3 Server WebService Test 1 Manual & automated tests Test 2 B. Extension Kalistick Agent - Kalistick Change detection (scanner) Agent - Kalistick Test 2 Test 1 Test 3 Test Holes Test Scoring : 4 2 1 14
Aggregate Test Footprints to find Test Holes & Redundancies 15 Application change Unit testing Integration testing Functional testing Exploratory testing User experience
Integration in a Continuous Delivery process 16 version scan build test footprint test footprint test footprint test footprint to the Cockpit TU release candidate Integration SCM Compile Deploy Acceptance Staging Continuous Integration Continuous Delivery production release Introduction Verification @ Phonak
Test Coverage Intelligence assembled at Phonak 17 A B C D Continuous Integration Nightly Scans + Build Test Functional Tests ALM Integration
Every internal release goes through scanning and testing 18 Scan of EXE & DLLs Run+Scan Unit & Integration tests Footprints are online Build + Installer 12am Continuous Integration Test Engineer #1 Deploy Installer Run Functional tests Test Engineer #9 Deploy Installer Run Functional tests 4pm
Coverage Intelligence was used in the last release cycle: October 2013 to March 2014 19 Hurdles: - Bugs in 3 rd party code are headache - Profiling may impact performance - New technology takes time to get adopted by people
We got the release data we wanted! 20
Important lessons learned in after-the-release analysis 21 1. Some uncovered code is a big surprise for both developers and test engineers. 2. Exploratory testing boosts combined coverage. 3. Modification coverage is useful towards the end of the release.
Continuous Delivery + Test Coverage Intelligence = Continuous Feedback Feedback! Feedback! Feedback!
Test Engineers, Managers and SETs are main consumers Test Engineers: - come up with new test cases to cover lacunas in the beginning of release cycle; - check to re-test status for already executed tests; - use uncovered areas to guide exploratory sessions in last phases.
Test Engineers, Managers and SETs are main consumers Test Managers: - review and re-prioritize functional tests; - identify redundancy in test cases; - investigate lacunas to focus exploratory testing and development of new test cases.
Test Engineers, Managers and SETs are main consumers Software Engineers in Test: - check uncovered areas to identify under-tested implementations (once it is available); - review lacunas and implement new integration tests (fits well into SCRUM sprints). All TEs, TMs and SETs are monitoring the effect of last minute changes before release.
Continuous feedback helps to keep fitting software healthy Untested parts are identified. It depends on context if automated or manual test cover it. 50% -> 70% of combined coverage Impact analysis of last-minute fixes highlights what to re-test. Better control over with late commits Positive effect of CI is available not only for developers but for test engineers as well.