Integrate Testing into Continuous Delivery

Size: px

Start display at page:

Download "Integrate Testing into Continuous Delivery"

Edmund Briggs
8 years ago
Views:

1 Integrate Testing into Continuous Delivery Needs of Phonak combined with solution from Kalistick Marc Rambert and Aliaksei Swiss Testing Day, March 19, 2014, Zürich

2 Agenda 2 Fitting Software Development at Phonak Challenge: Integrate Testing in Continuous Delivery at Phonak Solution: Test Coverage Intelligence Experience report from Phonak perspective Q&A

4 What is Fitting? Page Gardening Potential for Sunflower Beautiful Sunflower Fitting Potential for Hearing Performance Happy Customer

5 5

6 Fitting Software Codebase is relatively big ~4 * 10 6 LOC of C# (mainly) 5 teams, 70 Software Engineers, 10 Test Engineers, 3 SETs TeamCity as CI, Polarion as ALM Internal release every morning Release every 6 month with certification

Engineers, 3 SETs TeamCity as CI, Polarion as ALM Internal

8 Continuous integration hides many untested behaviors

9 #1 Bugs stay unnoticed in uncovered behaviors Behaviors explored manually Your Behaviors executed by automatic tests code Zero probability to notice a bug

10 #2 Bugs are introduced by last-minute changes 10 Change request implemented Functional Regression set #1 Bug because of Last minute effect

12 We need feedback on uncovered lacunas 12 Automatic instantly with CI Combined build and manual test counted Detailed on modifications regressions should be highlighted Test Coverage Intelligence

13 Test Coverage Intelligence extracts data to improve testing 13 Application(s) 1 2 Learning System Scanner Modifications Test Footprints CI Automated Tests ++ 3 Smart Engine Test Scoring Testing Holes

14 Detailed Impact Analysis for a whole system Test Team Test 3 Server WebService Test 1 Manual & automated tests Test 2 B. Extension Kalistick Agent - Kalistick Change detection (scanner) Agent - Kalistick Test 2 Test 1 Test 3 Test Holes Test Scoring :

Extension Kalistick Agent - Kalistick Change detection (scanner)

15 Aggregate Test Footprints to find Test Holes & Redundancies 15 Application change Unit testing Integration testing Functional testing Exploratory testing User experience

Integration in a Continuous Delivery process 16 version scan build test footprint test footprint test footprint test footprint to the Cockpit TU release

16 Integration in a Continuous Delivery process 16 version scan build test footprint test footprint test footprint test footprint to the Cockpit TU release candidate Integration SCM Compile Deploy Acceptance Staging Continuous Integration Continuous Delivery production release Introduction Phonak

release candidate Integration SCM Compile Deploy Acceptance Staging

17 Test Coverage Intelligence assembled at Phonak 17 A B C D Continuous Integration Nightly Scans + Build Test Functional Tests ALM Integration

18 Every internal release goes through scanning and testing 18 Scan of EXE & DLLs Run+Scan Unit & Integration tests Footprints are online Build + Installer 12am Continuous Integration Test Engineer #1 Deploy Installer Run Functional tests Test Engineer #9 Deploy Installer Run Functional tests 4pm

Installer 12am Continuous Integration Test Engineer #1 Deploy Installer

19 Coverage Intelligence was used in the last release cycle: October 2013 to March Hurdles: - Bugs in 3 rd party code are headache - Profiling may impact performance - New technology takes time to get adopted by people

3 rd party code are headache - Profiling may impact

20 We got the release data we wanted! 20

21 Important lessons learned in after-the-release analysis Some uncovered code is a big surprise for both developers and test engineers. 2. Exploratory testing boosts combined coverage. 3. Modification coverage is useful towards the end of the release.

22 Continuous Delivery + Test Coverage Intelligence = Continuous Feedback Feedback! Feedback! Feedback!

23 Test Engineers, Managers and SETs are main consumers Test Engineers: - come up with new test cases to cover lacunas in the beginning of release cycle; - check to re-test status for already executed tests; - use uncovered areas to guide exploratory sessions in last phases.

24 Test Engineers, Managers and SETs are main consumers Test Managers: - review and re-prioritize functional tests; - identify redundancy in test cases; - investigate lacunas to focus exploratory testing and development of new test cases.

25 Test Engineers, Managers and SETs are main consumers Software Engineers in Test: - check uncovered areas to identify under-tested implementations (once it is available); - review lacunas and implement new integration tests (fits well into SCRUM sprints). All TEs, TMs and SETs are monitoring the effect of last minute changes before release.

26 Continuous feedback helps to keep fitting software healthy Untested parts are identified. It depends on context if automated or manual test cover it. 50% -> 70% of combined coverage Impact analysis of last-minute fixes highlights what to re-test. Better control over with late commits Positive effect of CI is available not only for developers but for test engineers as well.

Agile SPL-SCM: Agile Software Product Line Configuration and Release Management

Agile SPL-SCM: Agile Software Product Line Configuration and Release Management APLE 2006 Workshop SPLC 2006, Baltimore, MD Reto.Kurmann@phonak.com Phonak Hearing Systems Presentation Roadmap 1. Introduction