The Definition of Metrics for Continuous Integration in SCRUM. How Continuous Is Our Continuous Integration?

Transcription

1 The Definition of Metrics for Continuous Integration in SCRUM How Continuous Is Our Continuous Integration? Christian Facchi University of Applied Sciences Ingolstadt Jochen Wessel Nokia Siemens Networks SMEF 2010

2 Outline 1 SCRUM and Continuous Integration 2 Definition of Metrics for Continuous Integration 3 Implementation 4 Case Study 5 Conclusion & Future Work Christian Facchi, Jochen Wessel SMEF 2010 Slide 2

3 Agile Manifesto: Individuals and interactions over processes and tools Working software over comprehensive documentation Customer collaboration over contract negotiation Respond to change over following a plan small feedback cycles focus on product /deliveries How can metrics be used? Christian Facchi, Jochen Wessel SMEF 2010 Slide 3

4 Basics SCRUM (Framework) Christian Facchi, Jochen Wessel SMEF 2010 Slide 4

5 Some basic principles i of SCRUM: continuous feedback loop every backlog item should produce deliverable code Supported by continuous integration: complete build and test after every modification (checkin) if build or test fails problems are solved early detection of integration problems (interfaces) no big bang at the end of the project/increment t Christian Facchi, Jochen Wessel SMEF 2010 Slide 5

6 Requirements for Continuous Integration: ti automated build and test automated configuration management Available toolsupport (visualisation): Cruise control Hudson Christian Facchi, Jochen Wessel SMEF 2010 Slide 6

7 Project Landscape Software Development UMTS/LTE Telecommunication System UMTS / LTE RAT spe ecific Applica ation CP and UP SW WCDMA SW LTE SW GSM SW Common SW BTS BTS Common BTS Commo on Platform SW Application S W BTS Site Mgr TRSW Common Application SW Common Platform Services SW RFSW PC/ Workstation Flexi BTS System Module HW FR/AAS Module HW Christian Facchi, Jochen Wessel SMEF 2010 Slide 7

8 Why metrics for Continuous Integration ti no objective input from SCRUM teams only feelings prioritisation of impediments among different SCRUM teams these metrics can be used not only in SCRUM requirement: automated t build process Christian Facchi, Jochen Wessel SMEF 2010 Slide 8

9 1 SCRUM and Continuous Integration 2 Definition of Metrics for Continuous Integration 3 Implementation 4 Case Study 5 Conclusion & Future Work Christian Facchi, Jochen Wessel SMEF 2010 Slide 9

10 Pulse of Continuous Integration ti CI frequency: = number of successful integrations per day heartbeat of the SCRUM team target is CI frequency 1* developers only successful integrations are measured (correct delivery of a product); not erroneous checkin (error correction phase) Christian Facchi, Jochen Wessel SMEF 2010 Slide 10

11 Quality of Continuous Integration ti #CI {OK,ERR} are defined as number of correct or erroneous checkins ErrorRatio has been determined for each phase (build, unittest, t systemtest, t t ) indicator for: process problems (before checkin a successful test is required) interface problems (caused by parallel running tests) Christian Facchi, Jochen Wessel SMEF 2010 Slide 11

12 Duration of Continuous Integration ti defined as maximal value of an automated tool completion time per day for a special development phase of a component if e.g. the build time is to long the number of integration problems will grow, due to parallel checkins as target limit 10 minutes for a finer statistical analysis (variance, ) result (OK, FAIL) has to be included, because failures are faster than successful tests Christian Facchi, Jochen Wessel SMEF 2010 Slide 12

13 Availabillity of Continuos Integration ti uptime <phase> := time where system in <phase> is up and running downtime <phase> := time where system in <phase> is not running availabillity should be very high (basic requirement for development tasks low availability is a clear indicator for severe problems Christian Facchi, Jochen Wessel SMEF 2010 Slide 13

14 1 SCRUM and Continuous Integration 2 Definition of Metrics for Continuous Integration 3 Implementation 4 Case Study 5 Conclusion & Future Work Christian Facchi, Jochen Wessel SMEF 2010 Slide 14

15 Implementation: ti fully automated test and build environment (UNIX scripts) extension of scripts to write metrics information (component name, phase, state, timestamp, ) in logfiles METRICS:<component><phase><state><time> < h t t ><ti > state = {started ok failed} PERL scripts to analyze logfiles spreadsheet program to generate statistical information Christian Facchi, Jochen Wessel SMEF 2010 Slide 15

16 1 SCRUM and Continuous Integration 2 Definition of Metrics for Continuous Integration 3 Implementation 4 Case Study 5 Conclusion & Future Work Christian Facchi, Jochen Wessel SMEF 2010 Slide 16

17 Case study: 2 SCRUM teams one development site one team in bugfix phase and one team in development phase Observations: low pulse of integration: less than 1 per team member and day; reason: already known task split (developer, tester) constant pulse of integration; no rush hour at the end of a sprint; ok slow development cycles: long maximal build time (up to 1.5 hours) long maximal test time (up to 2 hours) Reason: build process can be optimized (makefile generation) Christian Facchi, Jochen Wessel SMEF 2010 Slide 17

18 Actions: buildprocess has been optimized immediately reduced to 50%, further improvement scheduled remove tasksplitt between team members (tester, developer); areas of competence destabilized long term issue (education) Advantages for SCRUM teams: improved buildtime leads to reduce cycle time and to less integration problems (not reported) higher prioritisation of already suspected impediments Christian Facchi, Jochen Wessel SMEF 2010 Slide 18

19 Results of the case study: proposed metrics are very helpful for SCRUM teams. Despite the fact that metrics are suspected in general analyzing metrics needs time and deep insight (wrong conclusions can be drawn very fast) objective results to achieve improvements (measureable) less is more (concentrate on some valuable able metrics) better visualisation should help SCRUM teams (see burn down chart) let the SCRUM team select/define metrics Christian Facchi, Jochen Wessel SMEF 2010 Slide 19

20 1 SCRUM and Continuous Integration 2 Definition of Metrics for Continuous Integration 3 Implementation 4 Case Study 5 Conclusion & Future Work Christian Facchi, Jochen Wessel SMEF 2010 Slide 20

21 Conclusion Summary metrics in an agile environment are useful careful selection of metrics careful interpretation of results to much metrics leads to control by management (contradiction to SCRUM) metrics should help SCRUM teams, so the teams should define the metrics best practice: Continuous Integration can be used in every SW development process: Try it Christian Facchi, Jochen Wessel SMEF 2010 Slide 21

22 Future Work What remains to be done? application to other areas (plan driven projects; other agile methods) better visualization (Integration in Continuous Integration tooling); feedback to SCRUM teams extension to determine the quality of modularization Christian Facchi, Jochen Wessel SMEF 2010 Slide 22

23 Thank you for your attention Questions? University of Applied Sciences Ingolstadt Hochschule Ingolst adt Prof. Dr. Christian Facchi SW Engineering and Distributed Systems Esplanade Ingolstadt Phone: Fax: www:

24 SCRUM at LTE C-Plane SW in Nokia Siemens Networks (environment): development of a part of enodeb telecommunication SW (LTE C-Plane) ~90 software developers 2 development sites 8 SCRUM teams embedded in bigger frame program (using waterfall model)

25 Special setup of SCRUM at LTE C-Plane SW in Nokia Siemens Networks: Feature Expert Team: interface to traditional (waterfall) environment (frame process) continuity of development introduction of features to SCRUM teams support of Product Owner to build up backlog Feature Expert Team has been divided furthermore: Subject Matter Experts one central Product Owner and one proxy Product owner per site weekly SCRUM of SCRUM