Agile Software Development: Theory and Reality EuroSPI 2006 Tutorial October 11 th, 2006, 09 00-13 00 By Geir K. Hanssen, SINTEF /Norway (ghanssen@sintef.no) (This material may be reused but please put in a reference) 1
About me Geir Kjetil Hanssen Industrial experience as developer and project leader Researcher at SINTEF in Trondheim/Norway Studying agile software development in four companies PhD student at NTNU, Trondheim About you Name and work-place Work tasks/responsibilities/interests Why this course? 2
The goal of this tutorial I would like you to understand the basic ideas of Agile Software Development be interested but skeptical contribute with your opinions, views and questions 3
Schedule Introduction and presentation of us (approx. 20 m) Part 1: Theory (approx. 60 m. including 15 m. break) Explain the idea of agile software development Task1: Likes and dislikes (approx. 20 m.) Break 30 m. Part 2: Reality (approx. 60 m. including 15 m. break) Look into challenges and potential problems Task 2: Open issues/questions to be resolved (approx. 20 m.) Directions for future learning (approx. 10 m.) // Warning: This is just a plan // 4
Learning tools Input from me to you (lecture) Questions and comments from you to me and the others (corrections and focus) Tasks to make you think and have an opinion Discussions 5
Part 1: Theory Scrum Pair programming FDD Refactoring Evo Agile Modeling DSDM Crystal Lean TDD Extreme Programming 6
Part 1: Agenda The term agile The waterfall model Process flavours Software complexity and failures The motivation for a change Comparing agile and traditional approaches The core practices of agile software development The origin of the ideas The believers, the sceptics and the scientists view An example: Scrum Task 1: likes and dislikes Software projects trade-offs Software project context What do we know? 7
Agile In the dictionary: easy, quick, flexible, nimble A more mature version of light weight Agile methods is a group of related software development methodologies Extreme Programming, Scrum, Crystal, Lean etc A reaction to heavy, formal, document-driven, bureaucratic processes Extensive planning early in a project and often based on too low knowledge and experience (on both sides) Locked to plans and contract low flexibility Hard to focus on user needs (don t see what we need until we see it ) No space for creativity (?) 8
The waterfall model (By M.C. Escher) 9
The traditional approach Plan-based/driven processes Big Design Up-Front (BDUF) Big Planning Up-Front The waterfall-model 1 The idea All requirements and work is planned in advance A project follows the plan 100% predictability 100% control Not according to reality 1) W. W. Royce. - Proceedings of IEEE WESCON, 1970 10
The plan 11
12
Royce continued Royce identifies problems with the sequential approach Risky and invites failure Testing is the first point of feedback on the analysis The solution Opportunity to redo steps based on experience Iterative process! (5-600 citations according to Google Scholar the majority is probably incorrect ) 13
Alternative development processes Strategy Define all req. first? Several cycles? Distribute increments? Waterfall Yes No No req. design code test deliver Incremental Yes Yes Maybe req. design design code code test test deliver deliver design code Evolutionary No Yes Yes req. req. design req. deliver code test Source: Tore Dybå, Torgeir Dingsøyr and Nils Brede Moe, Process Improvement in Practice - A Handbook for IT Companies. Boston: Kluwer, 2004, ISBN 1-4020-7869-2 deliver test 14
The agile life-cycle Priotitation Dev./test Release Start-up Evaluation Termination 2-14 days n * 1-4 weeks a few days 15
Software growth A typical cell-phone now contains two million lines of software code; by 2010 it will likely have 10 times as many General Motors Corp. estimates that by then, more than 50% of the development costs of their cars will be due to software (Charette, R. N. Why Software Fails. IEEE Spectrum, Sept., (2005)) KLOC in Avionics System 10000 A3xx in KLOC 1000 100 10 1 A310 A300FF A300B A320 A330/340 60% of software projects are completed late and that the average cost overrun is 30-40% A substantial proportion of the late and underbudgeted software projects are outright failures (K. Moløkken, M. Jørgensen, S.S. Tanilkan, H. Gallis, A.C. Lien and S.E. Hove (2004) A Survey on Software Estimation in the Norwegian Industry, Proc. 10th Int. Software Metrics Symposium (Metrics 2004), Chicago, USA, pp. 208-219) Airplane type 16
ANNUAL PRODUCTIVITY GROWTH 1998-2003 Source: http://www.economy.com 17
Motivation Faster Increased demand for faster development and delivery? Early release of working software? Better Better understanding of explicit and implicit requirements? Software with fewer errors and higher usability? Cheaper Low documentation costs? Pragmatic administration? Reduced risk of delays? 18
Traditional and agile - compared Traditional development Agile development Fundamental assumption Management style Systems are fully specifiable, predictable, and can be built through meticulous and extensive planning. Command-and-control High-quality adaptive software can be developed by small teams using the principles of continuous design improvement and testing based on rapid feedback and change. Leadership-and-collaboration Knowledge management Communication Development model Desired organizational form/ structure Explicit Formal Life cycle model (waterfall, spiral or some variation) Mechanistic (bureaucratic with high formalization), aimed at large organizations Tacit Informal The evolutionary-delivery model Organic (flexible and participative encouraging cooperative social action), aimed at small and medium sized organizations Quality control Heavy planning and strict control. Late, heavy testing Continuous control of requirements, design and solutions. Continuous testing Table taken from: Nerur, S., Mahapatra, R. and Mangalaraj, G.., Challenges of migrating to agile methodologies, Communications of the ACM, May 2005, pp. 72 78. 19
A reaction to heavy processes The agile manifesto (www.agilemanifesto.org) says: Individuals and interactions over processes and tools Direct, verbal communication Giving the developers more responsibility and freedom Simplicity Working software over comprehensive documentation Frequent delivery of working software is the best proof of progress The customer gets early practical experience Customer collaboration over contract negotiation The customer is given a practical role Inside information Responding to change over following a plan Planning is close to impossible(!), optimal handling of change is a better approach Change DO occur don t resist, deal with it 20
Core practices investigated (1/3) Agile practices from the manifesto 1) Our highest priority is to satisfy the customer through early and continuous delivery of valuable software. 2) Welcome changing requirements, even late in development. Agile processes harness change for the customer's competitive advantage. 3) Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale. Believers view - Satisfied customer - Clearly shows progress - Responsive process - Open to new ideas - No need to plan in detail - Clearly shows progress - Always have a working version - Wrong design found early Sceptics view - How to deliver so frequent? - Why? - Is it really interesting? - What about the cost of rework? - When to know when finished? - Unhealthy pressure/stress? - No room for extensive design 4) Business people and developers must work together daily throughout the project. - Inside domain information - Dynamic requirements management - Reduce unclearity - Does the customer have time for this? - Does this disturb the developers? 21
Core practices investigated (2/3) Agile practices from the manifesto 5) Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done. 6) The most efficient and effective method of conveying information to and within a development team is face-to-face conversation. 7) Working software is the primary measure of progress. 8) Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely. Believers view - Fun and motivating - Trusting good people - No need of detailed management - Quick and direct communication - Maximum visibility - No stress or overtime - No scamped work Sceptics view - Can people be trusted? - No management? - Works only with the very best? - What about tracability? - Large, distributed teams? - What about completeness? - What about dealing with surprises? 22
Core practices investigated (3/3) Agile practices from the manifesto 9) Continuous attention to technical excellence and good design enhances agility. Believers view - Technically good solutions Sceptics view - Easy to say hard to do? 10) Simplicity--the art of maximizing the amount of work not done--is essential. - Does not invest effort in unnessesary work - What to decide what s (really) simple? 11) The best architectures, requirements, and designs emerge from self-organizing teams. - Optimal architecture - How come? 12) At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly. - Learn and improve from experience - Obstacles are removed during the project - No sceptic view on this one this is a good idea! 23
Method map (From Craig Larman) 24
An interesting anecdote Ivar Jacobson now says*: the most well-known variant of the Unified Process has grown its knowledge base beyond manageable limits. The Unified Process became too heavy Promotes a new process: Essential Unified Process An agile process *) Jacobson, I., Ng, P. W. and Spence, I., The Essential Unified Process a Fresh New Start. 2006. (download from www.ivarjacobson.com) 25
Methods and focus From: Abrahamsson, P., Warsta, J., Siponen, M. T. and Ronkainen, J. New Directions on Agile Methods: A Comparative Analysis. In Proceedings of International Conference on Software Engineering (ICSE) 2003). 26
The origin of the ideas From: Abrahamsson, P., Warsta, J., Siponen, M. T. and Ronkainen, J. New Directions on Agile Methods: A Comparative Analysis. In Proceedings of International Conference on Software Engineering (ICSE) 2003). 27
The terms used Thousands of projects 8+/- methods 10+/- practices Scrum, XP, Crystal, Evo, pair-prog., TDD, refactoring 12 principles 4 values the agile manifesto 28
The believer s view Agile software development is cheaper, faster and better because: Only what s needed will be developed Misunderstandings (with late rework) is discovered early Communication is more efficient (internal and external) Better conditions for creativity Changes can not be controlled better to emphasize change response Teams self-organize 29
The skeptic s view One customer representative does not have sufficient knowledge of: Organizational knowledge Domain knowledge Coordinating contradicting needs A customer will not be available 24/7 Customers will not accept no plan no estimates Small releases only fit small problems and small projects Does not support good architecture Agile methods does not fit in traditional project management frameworks Hackers excuse 30
The scientist s view Trying to be objective Looking for evidence Comparing with alternatives All of these are hard to do! If something s hard to do it s not worth doing Homer Simpson, 2000 We choose to go to the moon in this decade and do the other things, not because they are easy, but because they are hard [ ] JFK, 1962 31
An example: Scrum An agile process aimed at controlling and managing software development A structure for use of existing techniques A team-oriented approach for developing software iteratively and incrementally with frequent changes Control chaos by prioritizing based on knowledge Improves communication and maximizes cooperation Helps to find and remove anything that does not add value to the development Scalable Establish comfort 32
Scrum: the origin The New New Product Development Game in Harvard Business Review, 1986.*: The relay race approach to product development may conflict with the goals of maximum speed and flexibility. Instead a holistic or rugby approach where a team tries to go the distance as a unit, passing the ball back and forth may better serve today s competitive requirements. Jeff Sutherland (1993), Ken Schwaber (1996) and Mike Beedle The book in 2001 *) Takeuchi, H. and Nonaka, I. The New New Product Development Game. Harward Buisiness Review, (1986). 33
In rugby football a scrummage or scrum is a of restarting the game, either after a minor infringement [ ], or when the ball has gon onto the ground after a successful tackle [ 34
Scrum: Process components Process Sprint Planning Meeting Sprint Daily Scrum Meeting Sprint Review Meeting Sprint Retrospective Meeting Artifacts Vision Product Backlog Sprint Backlog Burndown Chart Roles Product Owner Scrum Team ScrumMaster (Stakeholders) 35
An example: Scrum (From www.controlchaos.com) 36
Scrum: The Scrum master Represents management to the project Typically filled by a Project Manager or Team Leader Responsible for enacting Scrum values and practices Main job is to remove impediments (From Mountain Goat Software) 37
Scrum: The Scrum team Typically 5-10 people Cross-functional QA, Programmers, UI Designers, etc. Members should be full-time May be exceptions (e.g., System Admin, etc.) Teams are self-organizing Membership can change only between sprints (From Mountain Goat Software) 38
Scrum: Sprint Scrum projects make progress in a series of sprints Analogous to XP iterations Target duration is one month +/- a week or two But, a constant duration leads to a better rhythm Product is designed, coded, and tested during the sprint (From Mountain Goat Software) 39
Scrum: Requirements management High Priority Each iteration implement the highestpriority requirements Each new requirement is prioritized and added to the stack Requirements may be reprioritized at any time Requirements may be removed at any time Low Priority Requirements Copyright 2004 Scott W. Ambler 40
XP in 10 seconds (More on www.extremeprogramming.org) 41
Task 1: What do you like about agile software development? What do you dislike about agile software development? 42
The borders of a software project and its trade-offs Time (to market) Cost Result (scope and quality) 43 faster means less (quality of) result more result means more time faster means higher cost Less cost means later more result means more cost less cost means less result
Processes can not be copied Industry best-practice? From Pekka Abrahamssons EuroMicro 2003 Keynote 44
Context-dependent applicability Criticality (Loss due to impact of defects) Many Lives Single Life Essential Funds Personnel (% Level 1B) (% Level 2&3) 30 10 3 40 30 20 10 0 Discretionary Funds Comfort 15 20 25 30 35 90 50 70 50 30 10 Dynamism (% Requirements-change/month) 5 1 Agile Disciplined Other axes? Domain complexity (simple-complex) Urgency (calm-urgent) Technical complexity (simple-complex) Novelty (plain-experimental) 100 30 Size (# of personnel) 300 10 Culture (% thriving on chaos vs. order) Source: Allistair Cockburn, used in Boehm, B. and Turner, R. Balancing Agility and Dicipline - A Guide for the Perplexed. Addison-Wesley, 2004. 45
Agile homeground From: Cockburn, A. Selecting a Project's Methodology. IEEE Software, (2000). 46
Communication effectiveness QA No QA (Adopted from Allistair Cocburn and Scott Ambler) 47
Types of software engineering Consultancy Sales of head-power; individuals, teams or projects Single-case One customer or a group Very specific requirements Related to a specific domain Product development Large, diverse user mass No direct requirements from customers Constant refinement and new releases 48
What do we know? (1) Enormous interest in the industry 2 large international conferences (XP and Agile) Extensive coverage in professional magazines and many books A search for extreme programming on www.amazon.com gives 51 results agile gives 354 results 2 books criticizing XP Stephens, M. and Rosenberg, D. Extreme Programming Refactored: The Case Against XP. Apress, 2003 McBreen, P. Questioning Extreme Programming. Addison-Wesley, 2003 1 book as balanced and neutral Boehm, B. and Turner, R. Balancing Agility and Discipline - A Guide for the Perplexed. Addison-Wesley, 2004. Weak documentation on effects, costs and limitations 2946 articles addressing the topic found in a search 821 abstracts on agile software development 380 articles with empirical data 39 good research publications 49
What do we know? (2) 50
The hype curve 2006? 2000? 51
What do we know? (3) Preliminary results from study Mainly focus on XP Low scientific quality on most studies Few studies over time Few studies on mature teams Studies mostly focus on Pair-programming Test-driven development Customer engagement Conclusion: we have little knowledge on how agile methods affects development! However many promising experiences in the industry! 52
Quality criteria's Screening questions ( no on 1 or (2 and 3) meant exclude) 1. Is this a research paper? 2. Is there a clear statement of the aims of the research? 3. Is there an adequate description of the context in which the research was carried out? Detailed questions: 4. Was the research design appropriate to address the aims of the research? 5. Was the recruitment strategy appropriate to the aims of the research? 6. Was there a control group with which to compare treatments? 7. Was the data collected in a way that addressed the research issue? 8. Was the data analysis sufficiently rigorous? 9. Has the relationship between researcher and participants been adequately considered? 10. Is there a clear statement of findings? 11. Is the study of value for research or practice? (Applied on 821 papers) 53
Announcement! EuroSPI Keynote, Friday: Prof. Pekka Abrahamsson: The concrete business impact of agile solutions - 3 times faster and 50 times better! 54
BREAK (coffee and discussions) 55
Part 2: Reality Quality Requirements Money Management Customers Programmers Users Project manager Time 56
Part 2: Agenda Conditions for agility Research summary Research insights: pair programming Research insights: test-driven development Research insights: customer engagement Supporting tools Task 2: Questions and open issues The future Final advice Pointers to further learning 57
Conditions for agility (1) Customer on-site Is the customer willing to spend the time to be available? Can anyone represent an organization or other users? How should the representative interact with his/hers organization? Has the customer good enough understanding of the requirements? Has the customer good enough domain knowledge? Is the customer able to respond to increments? 58
Conditions for agility (2) Team co-location How to deal with geographically spread organizations? How to deal with team-members that work on multiple projects? Are the offices suited for teamwork? Skilled individuals and teams Will the team be able to self-organize? Does the team contain the skills needed? Is everybody in favor of working in a team? 59
Conditions for agility (3) Customer acceptance of an agile contract Will the customer trust you? What should you do if the project fails? How to make the customers budget? Will the most important features be covered? What about documentation? What about future development? What about ISO9000, CMM etc.? 60
Conditions for agility (4) Technical excellence (infrastructure) How to release each iteration? How to enable the customer to test and reply? 61
Research summary Just a few principles and practices are investigated thoroughly, e.g.: Pair programming Arisholm et al. 2006 Test-driven development Erdogmous et al. 2005 Active customer engagement Hanssen and Fægri 2006 Missing evidence for e.g.: Self-organizing teams Agile architecture Refactoring Creativity Trade-off Suggestions? 62
Pair programming evidence* *) E. Arisholm, H. E. Gallis, T. Dybå and D. Sjøberg. Evaluating Pair Programming among Professional Java Developers, Submitted to IEEE Transactions on Software Engineering, 2006. 63
Test-driven development: evidence* (Test coverage) Test-First programmers write more tests per unit of programming effort (Productivity) A higher number of programmer tests lead to proportionally higher levels of productivity (Quality) Test-First programmers did not achieve better quality on average (although they achieved more consistent quality results) *) Erdogmus, H. and Morisio, M. On the Effectiveness of the Test-First Approach to Programming. IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, 31 (2005), 3. 64
Customer engagement: evidence Prerequisites Only motivated (by benefit) customers will engage Customer engagement needs proactive management Lack of continuity has significant bad effects on the performance Selecting the right (expertise) stakeholders are essential Frequent iterations leaves little room for unrestrained activity An effective technical framework is essential Benefits Close customer cooperation has a highly motivating effect on the developers Customer s prioritizing of goals has increased developers confidence The process visibility can be beneficial for the rest of the organization Costs Extra overhead in actually running the process and the human resources required The technical infrastructure, being a prerequisite, is costly Short iterations with insufficient attention to management and process compliance increase fragility Reduced capability to capture the needs of other, non-appointed customers 65
Supporting tools Even an agile process may benefit from the right tools: Continuous integration Process monitoring and control Requirements management Estimation and prioritizing Testing Rapid deployment and feedback management Some examples follow: 66
Tools: Requirements management (1) Product backlog 67
Tools: Requirements management (2) Sprint backlog 68
Tools: Requirements management (3) 69
week 14 week 15 week 16 week 17 week 18 week 19 week 20 Tools: Progress monitoring Backlog items burndown Burn-down charts 35 30 25 20 15 10 5 0 70 remaining backlog items week 1 week 2 week 3 week 4 week 5 week 6 week 7 week 8 week 9 week 10 week 11 week 12 week 13 Time/iterations Estimated work (hours) burndown 3000 2500 2000 1500 1000 500 0 week 1 week 2 week 3 week 4 week 5 week 6 week 7 week 8 week 9 week 10 week 11 week 12 week 13 week 14 week 15 week 16 week 17 week 18 week 19 week 20
Tools: Test coverage 160 140 120 100 80 60 40 20 0 # tests defined accumulated # tests passed accumulated 71 week 1 week 2 week 3 week 4 week 5 week 6 week 7 week 8 week 9 week 10 week 11 week 12 week 13 week 14 week 15 week 16 week 17 week 18 week 19 week 20
Tools: Continous integration (Borland) (Open-source) (Open-source) Check out: http://www.martinfowler.com/ articles/continuousintegration.html (Open-source) (Microsoft) From Trond Johansen, FIRM as, Norway 72
Even more tools XP StoryStudio (a project-management portal - free) TargetProcess (agile project management and bug tracking software - licenced) VersionOne (agile project management licenced) Xradar (system analysis report tool for Java - free) Jira (bug tracking, issue tracking, & project management - licenced) Extremeplanner (agile project planning and tracking - licenced) Fitnesse (acceptance testing framework - licenced) Ant/Nant (build tool free) Maven (software project management and comprehension tool free) CruiseControl (build support free) FXCop (code analysis tool free?) StartTeam (source code control licenced) Wiki (information sharing free) Clover/Nclover (test coverage tool free?) JUnit/NUnit/NUnitASP (unit testing framework free) Cobertura 1.6: (Java test coverage tool free) EasyMock 1.2 RC2 (Mock Object for Java interfaces free) Exactor 1.1.4: (framework for automated acceptance tests free) Jakarta Cactus 1.7.1: (Unit-testing server-side Java free) JasperReports 1.0.2: (Java reporting tool free) Jameleon 3.0.3: (Java tool for automated acceptance testing free) log4j 1.2.12: (Java logging tool free) and many many more Do you remember the first agile value? Individuals and interactions over processes and tools 73
Task 2: Questions and open issues What are the most important questions still to be answered? Use your background (academic/industry) 74
Future Fundamentalism is (as always) not the best approach! Hopefully, the number of methods will converge into a few Tailored for domains, technologies etc. Easier to find the right one More validation by research The solution will be a balance between agility and discipline 75
Maturing ideas From Crossing the Chasm by Geoffrey A. Moore 76
Boehm and Turnes conclusions* 1. Neither agile nor plan-driven methods provide a silver bullet 2. Agile and plan-driven methods have home grounds where one clearly dominates the other 3. Future trends are toward applications developments that need both agility and discipline 4. Some balanced methods are emerging 5. It is better to build your method up than to tailor it down 6. Methods are important, but potential silver bullets are more likely to be found in areas dealing with people, values, communication and expectations management. Boehm, B. and Turner, R. Balancing Agility and Dicipline - A Guide for the Perplexed. Addison-Wesley, 2004. 77
Final advice* A lot of experts selling silver bullets however: 1. Read and understand yourself! Be interested but sceptical! 2. Check recent research results! (learn from other s experience) 3. Try and evaluate! (learn from your own experience) 4. Discuss with someone with practical experience 5. Don t be fanatic be creative 6. Involve everybody the SE process is everybody's concern *) Applicable for any hype (WebServices, MDD, Agile, SOA )! 78
Appraising published studies From: Dybå, T., Kitchenham, B. and Jørgensen, M., Evidence-Based Software Engineering for Practitioners, in IEEE Software. 2005. p. 58-65. 79
Pointers to further learning www.agilealliance.org Fresh info from the community www.agilemanifesto.org Simple documentation of the basic concepts www.extremeprogramming. org A good overview of XP basics www.controlchaos.com Scrum overview (some marketing) www.wikipedia.org Definitions Links Agile Software Development with SCRUM by Ken Schwaber and Mike Beedle Balancing Agility and Discipline: A Guide for the Perplexed by Barry Boehm and Richard Turner Takeuchi, H. and Nonaka, I. The New New Product Development Game. Harward Buisiness Review, (1986) Boehm, B., Get Ready for Agile Methods, with Care, in IEEE Computer. 2002. p. 64-69. Abrahamsson, P., Salo, O., Ronkainen, J. and Warsta, J. Agile software development methods - Review and analysis. VTT Publications 478, VTT Electronics, 2002. Cohen, D., Lindvall, M. and Costa, P. Agile Software Development A DACS State-of-the-Art Report. Fraunhofer Center for Experimental Software Engineering Maryland and The University of Maryland, 2003. 80