Understanding the dynamics of agile software development methods

Transcription

1 Understanding the dynamics of agile software development methods R. Lal Institute of Information & Mathematical Sciences Massey University at Albany, Auckland, New Zealand This study uses multiple case studies based on Fitzgerald s framework, known as the Framework for the IS Development Process, to investigate software development teams that use agile methods to provide an understanding of agile practices and impacts on IS development projects. The framework will be extended to enable a better understanding of agile software development methods and practices. A model for user-involvement and a model for adopting agile methods will be developed extending Fitzgerald s framework. 1 Introduction Software has become an essential technology for businesses. Since its inception for business data processing in early 1960s, numerous business software development projects have failed, costing substantial amount of money (Wallace & Kell, 2004; Keil & Robey, 2001; Mahaney & Lederer, 1999). These software project failures happen with companies regardless of their size. Table 1 provides data on project success and failure from 1994 to Table 1: Historical perspective on project success ( ). Year Succeeded Failed Challenged % 48% 47% % 31% 53% % 40% 33% % 28% 46% % 23% 49% Source: Standish Group Report 2000, US Government Accounting Office. Report FGMSD-80-4 The challenged projects listed in Table 1 are those projects that were completed but were over the budget and time estimates, with fewer features and functions. The statistics in Table 1 suggest that success rates went up except in 1998 and a significant decrease in failed projects. It is the belief that formal software development methodologies have an impact on project success (Sommerville, 2004; Whyte & Bytherway, 1996; Avison & Fitzgerald, 1995). The Standish Group 2000 report states that the use of formal methodology should increase the chance of project success by 16%.The available IS literature provides many perspectives for IS project failures despite the significant progress made in IS development methodologies and tools (Dhillon, 2000; Ewusi- Mensah, 1997; Abreu & Conrath, 1993). These perspectives can be grouped in to sociorganizational, technical, and economic factors (Ewusi-Mensah, 2003). It has been estimated by Jayaratna (1994) that there are over 1000 software development methodologies world-wide. Software development has been carried out using ad-hoc approaches, prescriptive methodologies (Table 2) or a combination of both (Avison & Fitzgerald, 2003; Taylor, 2001). Despite plethora of methods in 1980s and 1990s, several inconsistencies remained with software development such as; how to develop systems quickly while accommodating requests for changes late in development process, maintaining quality, and controlling cost (Baird, 2003). 75

2 Table 2: Methodologies used for information systems development since1960s. Period Era Methodology types 1960s and early Pre-Methodology era Ad-hoc approach 1970s Late 1970s-early 1980s Mid-1980-late 1990s late1990s present till Early Methodology Eraprescriptive methodology Methodology Era (proliferation)- software engineering, prescriptive methodologies Post methodology era SDLC- waterfall model Structured- STRADIS, Yourdon Systems Method, SSADM, Jackson Systems Development; Dataoriented- IE; Prototyping-RAD; Unified Process; Object-Oriented Analysis, Participative-ETHICS;, Strategic-ISP, Systems-ISAC,SSM, MULTIVIEW, Formal methods; Vienna Development Method Ad-doc, Agile methods; Scrum, Dynamic Systems Development Method, Crystal Methods, Feature- Driven Development, Lean Development, Extreme Programming, Adaptive Software Development (ASD), Agile modeling, Internet-speed development Source: Avison & Fitzgerald (2003), Avison & Fitzgerald (1995), Lewis (1994), Jayaratna (1994). The mid 1990s saw the emergence of a new set of informal analysis and design approaches known as lightweight methods later named agile methods (Highsmith, 2002). It appears that the agile methods are becoming popular and many software development teams are adapting this new way of developing software. Reasons have been suggested as to why these new methods are more appropriate for software development in a dynamic business environment such as the ability to: (a) move quickly and react to change, (b) accept and welcome change, (c) deviate from plan and treat it as new information, (d) optimize communication among various stakeholders, and (e) learn from each agile project (Koch, 2005). It appears that adoption of any of the agile method has been based on someone s account of how the method was used to design and develop the software in their organization (Abrahamsson et al. 2003). There are claims that the use of agile development method enables software to be created without the overheads of prescriptive methods but it appears that no major academic research has been undertaken to verify these claims and practices, or to provide a better understanding of the mechanics of agile methods. 1.1 Research Questions This research seeks to answer the following questions to help understand the dynamics of agile development methodologies: 1. What is the history of software project success and failure? Is there any evidence of evolution? What are the issues that have an impact on project success? 2. What determines the success of agile software development projects and how is success recognized and measured? 76

3 3. Why do software development teams want to adopt agile methods? What changes are made to adapt agile methods? 4. What are the ideal characteristics of customers? Do teams employ user-centred tools and methods? How are conflicts with customers recognized and resolved? 5. Is agile development suited for large software projects or virtual teams? 1.2 Background In February 2001 a group of seventeen software experts (Kent Beck, Mike Beedle, Arie van Bennekum, Alistair Cockburn, Ward Cunningham, Martin Fowler, James Grenning, Jim Highsmith, Andrew Hunt, Ron Jeffries, Jon Kern, Brian Marick, Bob Martin, Steve Mellor, Ken Schwaber, Jeff Sutherland, and Dave Thomas) got together in Snowbird ski resort in the Wasatch Mountains of Utah, to discuss the growing field of what used to be called lightweight methods (Highsmith, 2002). They decided to use the term agile to describe these new lightweight methods and the agile software development manifesto was written describing the values and principles of the agile methods (Fowler 2002). Agile software development is seen as an alternative to software engineering driven development. Software engineering is often seen as a rigorous process that requires substantial planning, modelling, and creation of various artefacts. Table 3 lists methods that are part of the agile family which are based on the belief that a better way of developing software is by actually creating the software itself rather than spending considerable amount of time determining what is to be developed, planning the various activities of the software development, designing, and modelling the features of the software before embarking on any software construction work. Table 3: Methods that are part of agile family. Method name Year Lean Development (LD) 1980s Dynamic Systems Development Method (DSDM) 1995 Scrum 1995 Crystal Methods 1998 Extreme Programming (XP) 1999 Internet-speed development (ISD) 1999 Adaptive Software Development (ASD) 2000 Feature-Driven Development (FDD) 2002 Agile modeling (AM) 2002 Source: Highsmith (2002), Abrahamsson et al. (2003) The Manifesto for Agile Software Development states the following: we are uncovering better ways of developing software by doing it and helping others to do it. Through this work we have come to value: Individuals and interactions over processes and tools. Working software over comprehensive documentation. Customer collaboration over contract negotiation. Responding to change over following a plan. That is, while there is value in the items on the right, we value the items on the left more (Highsmith, 2002, pg xvii). These new methods attempt to provide balance between the items on right with items on the left rather than replacing the items on the right. According to Martin (2003) it is a useful compromise between no process and too much process and providing just enough process to gain a reasonable payoff. Agile software development is based on customer satisfaction, early incremental delivery of software, small and highly motivated teams, informal methods, and minimal software processes (Pressman, 2005). Before the emergence of agile methodologies, 77

4 there was the belief in the software industry that successful software development would succeed only through careful project planning, formalization, quality assurance, the use of analysis and design methods supported by CASE tools, and controlled and rigorous software development processes (Sommerville, 2004). However, this plan-based software development tied the developers down so much in following the processes, where a considerable amount of time is spent in planning, eliciting systems requirements, designing and modelling the requirements, and writing extensive documentation rather than creating the actual system. Another major concern that was discovered was that the systems requirements had changed even before the new system is fully implemented. This type of methodology is more suited for large scale industrial and scientific software development and when applied to software development for small and medium-sized business, the cost is overwhelming (Sommerville, 2004). The minimal process that is used with an agile software development method is based on the belief that during systems development it can t be determined in advance which requirements will remain static and which ones will change due to changing business conditions. The requirements are prioritized, and the agile process recognizes the fact that the requirements and requirement priorities may well change during the development period. With agile methods, any function or feature of the software is developed immediately after minimal design to test it out, hence becoming a test-driven methodology. The agile process is based on the idea that analysis, design, development, and testing activities can t be predicted and planned in advance (Fowler, 2002). To adopt agile methodologies, the software development team will need to be equipped with people-skills or soft-skills. According to Highsmith (2002) agile methodologies place far more value on the interaction of talented individuals over the process and tools. The process and tools are key themes of prescriptive methodologies. Agile methods require team members to have appropriate communication skills for collaboration to happen in a team situation with customers and also have a friendly approach and talent to relate well with others (Cockburn & Highsmith, 2001). Pressman (2004) lists the following must have traits of agile team members; (a) competence, (b) common focus, (c) collaboration, (d) decision-making ability, (e) fuzzy problem-solving ability, and (f) mutual trust and respect. It is a significant move away from the technical skill as the only important skill for any team member using the prescriptive methodology. Pros and cons of agile methodologies are hotly debated and some critics of agile methodologies argue that agile concepts are simply the adoption of good practices from different prescriptive methodologies (Stephens & Rosenberg). Other pitfalls have been identified, such as; effective customer involvement is not possible, individuals have different personalities which can often be a barrier for forming an effective team, prioritizing requirements is difficult for large systems with many stakeholders, and simplicity requires extra work, which is difficult to carry out when working under pressure to meet deadlines (Sommerville, 2004). According to Pressman prescriptive methodologies can be classified broadly into four types; (a) the waterfall model, (b) incremental model-rad (c) evolutionary development prototyping, spiral model, concurrent development model and (d) specialized process models- component based software engineering. These models are process driven. Software process is a framework for tasks that are required to build highquality software. Process includes the approach to be taken for analysis, design and development of software (Pressman, 2005). Software engineering identifies the process layer (Figure1) as critical in enabling tools and methods to be used to develop quality software. According to Pressman, the process 78

5 layer defines the framework activities - communication, planning, modeling, construction, and deployment, which must be established for effective delivery of the software. For software development projects, it provides for the following: (a) control activities, (b) work products, (c) milestones, (d) quality standards, and (e) change management for requirements. SE (Software Engineering) as a layered approach (Pressman, 2005) Tools Methods Automated or semi-automated support for process and methods Process A quality focus Tasks: communication, requirement analysis, design, modeling, program construction, and support Figure 1. A considerable amount of time is spent determining requirements before any software is written. A feasibility study, requirement elicitation and analysis, requirement validation, requirement reviews, and requirements management, occur first. Various artifacts such as activity diagrams, class diagrams, and use cases are created to model systems requirements. System models -context models, behavioral methods, data models, and object models are developed based on requirements showing operational, functional, and behavioral characteristics of the system (Sommerville, 2004). Before any coding begins architectural design is created showing the structure of data and program components and also component-level design is created to determine if software will work as planned (Pressman, 2005). This shows that with software engineering a considerable amount of design and modeling work happen, which may take months, before any attempt is made to create the software (Sommerville, 2004). These detailed levels of activities, and the designs and models that are created are seen as unnecessary and time consuming by agilists (Kouch, 2005; Martin, 2003). 2 Literature review There is substantial literature available regarding agile methods, documenting conditions where these methods are more suited for software development than compared to prescriptive methods. Agile methods claim to offer an improvement in quality of software, requirements management, customer satisfaction, and team satisfaction (Ceschi et al., 2005; Patton, 2000). They suit software projects with uncertain requirements (Fowler, 2003), which suggests that the agile practice of iteration as essential for software to materialize into existence. The rapid and iterative aspect of agile methods also enables frequent release of working products for customers to see throughout the entire development process (Schatz & Abdelshafi, 2005; Johnson, 1999). Agile methods emphasize that any process used should be effective and efficient and needs to change as organization s needs change. Agile methods are based on the idea of barely sufficient process (Little, 2005; Grossman et al., 2004; Cockburn, 2001), which agilists believe enables successful software development. Researchers also point out that moving away from extensive process based design enables development teams to meet customer demand quickly (DePauw, 2002; Smith, 2002). Agile methods adopt the practice of simplicity i.e. avoid any unnecessary work or tasks that do not add value to the project (Koch, 2005). Communication and feedback enable developers to optimize 79

6 various stakeholder involvements in agile projects (Lindstrom & Jeffries, 2004). Agile methods advocates significant amount of interaction between the development team and the customer to create a positive working relationship. Hence, collaboration with customers is also regarded as one of the important agile software development practices (Thomas, 2005; Ceschi et al. 2005). This practice emphasizes that all stakeholders must work together as a team throughout the development process. 3 Theoretical basis for the research The study of agile methods will be carried out based on a framework developed by Fitzgerald (1999), known as the Framework for the IS Development process. It enables the investigation of the nature of development and the use and role of systems development methods in actual practice. The framework has the following components: (a) original formalized methodology vs. methodology-in-action, (b) development context: business opportunity and problem situation, (c) developer/methodology user and developer-embodied factors, (d) profile of development environment, and (e) roles of methodology: overt intellectual vs. covert political factors. Some practices have been identified through a literature review such as: change management, communication, coordination, software teams, organization culture, user involvement, user-centered and participatory design, and successful systems implementation. Based on these practices data will be collected for the five components comprising the framework. 4 Methodology The approach used for this study is interpretive (Walsham 1995). Case study is the preferred strategy for this research since the how and why questions are being investigated (Yin 1994). This study will involve multiple case studies (up to 4). It will involve the use of similar measures or procedures for all the case studies if replication is not possible (Yin 1993). The unit of analysis has been identified as the software development team. It is anticipated that interviews, documentation, and observation will provide vital information, which will be compiled and analyzed using Fitzgerald s framework to understand the dynamics of agile methods. The validity of the research will be achieved through triangulation of multiple sources of data using 3 different methods identified to collect data and through using representative samples to select cases (Leedy & Ormrod 2001). Patton (1990) provides various sampling strategies for case study research, which will be explored and used as bases to select appropriate cases for this research. The framework will be used as a general strategy to analyze data. The various theoretical propositions derived from the literature review and the identified framework enabled to establish research questions. The focus of the study will be to extend the Fitzgerald s framework by creating a model for user-involvement and a model for adopting agile methods. 5 Research Contribution The findings from this research should enable academics and software developers to better understand the nature of agile methods, the characteristics of agile projects, and what agile means in terms of software development. This research should make academics and software developers aware of: 1. The extended framework for the IS Development process. 2. Changes that need to be made to successfully adopt agile methods. 3. How agile methods have an impact on the costs and schedule of software projects. 80

7 4. Causes of agile project failure and success. References Abreu, A. F. & Conrath, D.W. (1993). The role of stakeholders' expectations in predicting information systems implementation outcomes. Proceedings of the conference on computer personnel research, Missouri, United States, Avison, D., & Fitzgerald, G. (1995). Information systems development: methodologies, techniques and tools (2 nd ed.). London: McGraw-Hill. Avison, D., & Fitzgerald, G. (2003). Where now for development? Communication of the ACM, 46(1), Baird, S. (2002). Teach yourself extreme programming in 24 hours. Indianapolis, Indiana: Sams Publishing. Ceschi, M., Sillitti, A., Succi, G. & Panfills, S.D. (2005). Project management in planbased and agile companies. IEEE Software, may/june Cockburn, A. & Highsmith, J. (2001). Agile software development: the people factor. IEEE Computer, 34(11) Cockburn, A. (2001). Agile software development.boston: Pearson Education, Inc. DePauw, T. (2002). Caterpillar digs into agile development. Article retrieved from Dhillon, G. (2000). Interpreting key issues in IS/IT benefits management. Proceedings of the 33 rd i International Conference on System Science, Hawaii, 1-9. Ewusi-Mensah, K. (1997). Critical issues in abandoned information systems development projects. Communications of the ACM, 40(9) Ewusi-Mensah, K. (2003).Software development failures. Cambridge: MIT Press. Extreme chaos. Report retrieved from Fitzgerald, B. (1999). An empirically-grounded framework for the information systems development process. Proceedings of the international conference on Information systems, Helsinki, Finland, Fowler, M. (2002). The new methodology. Article retrieved from /articles/newmethodology.html. Grossman, F., Bergin, J., Leip, D., Merritt, S., & Gotel, O. (2004). One XP experience: introducting agile (XP) software development into a culture that is willing but not ready. Proceedings of the 2004 conference of the Centre for Advanced Studies on Collaborative research, Markham, Ontario, Canada, Highsmith, J. (2002). Agile software development ecosystem. Boston: Addison-Wesley. Humphrey, W.S. (2000). Introduction to team software process. Massachusetts: Addison-Wesley Longman. Jayaratna, N. (1994). Understanding and evaluating methodologies NIMSAD: A systemic framework. McGraw-Hill, Maidenhead. Johnson, J. (1999). Turning chaos into success. Software Magazine Article retrieved from 81

8 Keil, M. & Robey, D. (2001). Blowing the whistle on troubled software projects. Communication of the ACM, 44(4), Koch, A.S.(2005). Agile software development: evaluating the methods for your organization. Boston: Artech house. Leedy, P.D. & Ormrod, J.E. (2001). Practical research planning and design(7 th ).Upper saddle River, New Jersey: Merrill Prentice Hall. Lindstrom, L. & Jeffries, R.(2004). Extreme programming and agile software development methodologies. Information Systems Management, summer Little, T.(2005). Context-adaptive agility: managing complexity and uncertainty. IEEE software, May/June Mahaney, R.C. & Lederer, A.L. (1999). Runaway information systems projects and escalating commitment. Proceedings of the 1999 ACM SIGCPR conference on computer personnel research, New Orleans, United States, Martin, R.C. (2003).Agile software development: principles, patterns, and practices. New Jersey: Prentice Hall. McKinney, D. & Denton, L.F.(2005). Affective assessment of team skills in agile CS1 labs; the good, the bad, and the ugly. Proceedings of the 36th SIGCSE technical symposium on Computer, Missouri, USA, Patton, J. (2000). Hitting the target: adding interaction design to agile software development. Paper presented at conference on Object Oriented Programming Systems Languages, Seattle, United States, 1 ff. Patton, M.Q. (1990). Qualitative evaluation and research methods. Newbury Park, California: Sage Publications. Pressman, S. (2005). Software engineering: a practitioner s approach(6 th ). New York: McGraw-Hill. Stephens, M. & Rosenberg, D. (2003). Extreme programming refactored. Berkley, CA: Apress. Schatz, B. & Abdelshafi, I. (2005). Primavera gets agile: a successful transition to agile development. IEEE Software, may/june Smith, W. (2002). Users warm up to agile programming. Article retrieved from Sommerville, I. (2004). Software engineering (7 th ). Boston: Addison-Wesley. Taylor, H. (1991). Information systems development practice in New Zealand. New Zealand Journal of Applied Computing and information Technology, 5(2), Thomas, D. (2005). Agile programming: design to accommodate change. IEEE software, May/June US Government Accounting Office Report FGMSD-80-4 (1980). Wallace, L. & Kell, M. (2004). Software project risks and their effect on outcomes. Communication of the ACM, 47(4), Walsham, G. (1995). Interpretive case studies in IS research: nature and method. European Journal of Information Systems, 4(2) Whyte, G. & Bytherway, A. (1996). Factors affecting information systems success. International Journal of Service Industry Management, 7(1)

9 Yin, R.K. (1993). Applications of case study research. London; Sage Publications. Yin, R.K. (1994). Case study research design and methods (2 nd ). London; Sage publications. 83