A Look at Software Engineering Risks in a Team Project Course

A Look at Software Engineering Risks in a Team Project Course Supannika Koolmanojwong and Barry Boehm Center for Systems and Software Engineering (CSSE) University of Southern California (USC) Los Angeles, CA, 90089-0781, USA {koolmano, boehm}@usc.edu Abstract Risk identification, management, and mitigation are essential to the success of any software development projects. At the University of Southern California (USC), CSCI577ab is a graduate level software engineering course sequence that teaches the best software engineering practices, and allows students to apply the learned knowledge in developing real-client projects. This paper analyzes the risks encountered by teams, identifies the relationships between effective risk analysis and the success of the project, and discusses how risk patterns determine the student s course of action. This paper also reports the top risks of software development in the software engineering class. 1. Introduction It is common to have risks in a software development project. Risk can come with opportunities for the risk takers to lead a project to a different course of action, or to successful development. At the same time, risks can also lead a project to failure and disaster. Hence, risk management is an essential topic in any software engineering course. There are many ways to introduce the concept of risk management to the class, such as using games [14], risk assessment tools [7], or educational case studies [10][13]. In our CSCI577-graduate level software engineering class, as part of the software development process, student teams learn how to identify, analyze, mitigate and manage the risks in their real-client team projects [5]. The data collected through weekly risk reports using a tool called Distributed Assessment of Risk Tool (DART), milestone reports, and additional surveys are used to identify the top ten risks in software team projects, and to show how the risks determine different project life cycle processes. A software engineering class is similar to a software engineering project in a sense that, at the end of each project and class, one should perform retrospective analysis and find out improvement opportunities. Risks that occur in each project should be compiled and analyzed so that the project managers will know the weak points and strong points of the project so that they will not repeat the same simple mistakes in the future projects. Similarly, the course instructor should learn from previous class projects and prepare lectures, homework exercises, or supplementary information for the next class. The rest of the paper is organized as follows. Section 2 elaborates on the nature of the software engineering class and, the Incremental Commitment Spiral model which is used as a main process model for the software engineering class. Section 3 reports the risks found in the software engineering class, comparing them to risks found in the industry. Section 4 elaborates on the relationship between risk management performance and project performance. Section 5 shows how risk patterns can determine a course of action and how they may drive different development life cycles for each project. 2. Related Works

2.1. The Software Engineering Team Project Class CSCI577ab [6] is a graduate level software engineering course at the University of Southern California (USC). The main objective of the course is to prepare students for software leadership careers through the 2050 s. Software Engineering I or CSCI577a focuses on software plans, processes, requirements, architectures, risk analysis, and feasibility analysis. Software Engineering II focuses on software product creation, integration, test, and maintenance with an emphasis on quality software production. Each year, students are self-organized into 15-20 teams of six on-campus and two off-campus students to develop projects or e-services projects for real-clients in effectively 24 weeks. The clients are primarily from various USC departments, neighborhood corporations, local government agencies, and community service organizations. Oncampus students act as operational concept engineers, requirements engineers, software architects, UML modelers, coders, life cycle planners, and feasibility analysts. Offcampus students act as Integrated Independent Verification and Validation (IIV&V) personnel. The course staff teach and use evolving best practices, techniques and tools such as requirement management, object-oriented analysis and design, risk management, quality management, peer reviews, configuration management, and valuebased software engineering. Risk management is a core activity in CSCI577ab. On a weekly basis, by using the Distributed Assessment of Risks tool (DART), all team members have to brainstorm to identify the risks. For each risk item, each team member identifies the probability of risk occurrence (0-10) and the size of loss (0-10) in order to identify risk exposure for each risk item. Risk exposure can be calculated by multiplying the probability of loss to the size of loss. Team members, together with the clients, discuss the resulting risk mitigation plans. The project manager takes the top ten risks and reports them in his/her weekly progress report. Additionally, for each milestone, the teams have to present their current risks and mitigation plans to the architecture review board as evidence of project feasibility. The authors, as the course staff, review and monitor the project risks. 2.2. The Incremental Commitment Spiral Model The ICSM [4][11] is a process model covering the full system development life cycle consisting of the Exploration phase, Valuation phase, Foundations phase, Development phase, and Operation phase. ICSM, as shown in Figure 1, has been shown to be effectively tailorable to a wide variety of system development situations. The four underlying principles of the ICSM include a) stakeholder value-based system definition and evolution, b) incremental commitment and accountability, c) concurrent system and software definition and development, and d) evidence and risk-based decision making. Evidence of project feasibility is the key ingredient to avoid the risks, and can be used to determine the future of the project. Our Software Engineering class uses the ICSM as its software development process model. Exploration, Valuation, and Foundations phases are in CSCI577a in the fall semester. To accommodate the possible changes during the winter break, once the spring semester begins, the teams execute a Rebaselined Foundations phase and proceed to the Development phase. Final project delivery marks the beginning of the Operation phase. Hence, our software engineering project activities end with the project transition, training, and the early phase of deployment. As shown at the bottom of Figure 1, at the end of each milestone, current project risks determine the possible project direction. More information can be found in Section 5.

Figure 1. Overview of the Incremental Commitment Spiral Model [4] 3. Top Risks in Software Engineering Class Projects Risk management is one of the fundamental activities in industrial software development. For each project, risks can be different based on their characteristics. Examples of traditional risks [1] are personnel shortfalls, requirements volatility, architecture complexity, and quality tradeoffs. For internet and intranet software development, the volatile technology requires rapid changes in methods and tools. Some of today s risks are quite different from the risks that we had 10 years ago due to the fast pace of technologies [12]. Table 1 summarizes the risk categories and example risk items found in our software engineering class. In general, risks found in our class are similar to risks in the industry. However, it is important to mention that some risks are found more often in the software engineering class, for example: No maintainer Since the class will end after the product is delivered, it is the client s responsibility to find a system maintainer or administrator. Since most clients are nonprofit small-scale organizations, it is a challenge for clients to appoint, hire, or learn to be a maintainer. Process maturity and quality assurance To support the concepts of process maturity and quality assurance, the class provides guidelines and tools for student teams to learn about such good practices as configuration management and quality management. But with a very steep learning curve of learning about the project and process at once, student teams might not be able to digest and follow all suggested process guidelines. Acquisition In various projects, the client has to acquire additional hardware, such as a mobile device or a scanner. However, most of the clients are non-profit organizations,

they also have problems of budget constraints. Hence, sometimes the clients were not able to acquire the required infrastructure as planned. Personnel capability Most on-campus students came directly from the undergraduate level with limited software development experience. Thus, they are learning software engineering and other computer science skills on the fly, and hence they have limited personnel capability. Risk Category Architecture complexity; quality tradeoffs Budget, schedule and resource constraints COTS and other independently Evolving systems Customer-developer-user team cohesion Personnel shortfalls Requirements mismatch Requirements volatility; rapid change User interface mismatch Process Maturity Lack of domain knowledge Acquisition and contracting process mismatches Others Table 1. Risk Categories and Examples Example of risk items Maximum optimization system design; generalizing the design modules for future evolutionary needs 24 week development schedule; often a zero monetary budget and limited computing power Unknown COTS infrastructure, unreliable COTS performance, COTS interoperability, future scalability Off-campus students work full time in different time zone, difficult to find a good meeting time slot Lack of technical and software engineering knowledge; personnel turnover, unknown maintainer Requirements-architecture mismatch New stakeholders emerge with different visions, hence different requirements. GUI may be too complex for non-technical users Possibility of inconsistent data due to team members not following the configuration management plan Learning curve about domains such as health care and business processes; Beyond Computer Science scope Often the clients are unable to provide special devices for testing as initially envisioned Migration complexity As the project progresses, student teams, together with the clients, have identified their risks in the weekly report. The risks collected from 86 teams from Fall 2005 Spring 2010 are analyzed to determine the possible categories of risks, frequency of risks and possible risk rankings. Figure 2 presents the percentage of frequency of risks in each category that occur during the project development. The blue columns represent risks that occur in the fall semester or Exploration, Valuation, and Foundations phases. The red columns represent risks that occur in the spring semester or Development and early Operation phase. The columns are sorted by average percentage of risk occurrence. As shown in Figure 2, the risk occurrence in the fall semester is uniformly higher than the risk occurrence in the spring semester. The risks that involve a learning curve such as process maturity, or lack of domain knowledge, are much lower in the spring semester.

Figure 2. Percentage of Risk Occurrence in each category Figure 3. Summary of Risk Ranking in Each Category On the other hand, for each risk, student teams have to calculate the risk exposure, which is the possibility of risk multiplied by the size of the loss. The Top Ten Risks List is determined by summing the risk exposure of each risk item. Figure 3 represents the summary of each risk ranking. The blue columns represent the risks in the fall semester, while the red columns represent the risk in the spring semester. While there are fewer risks and lower risk exposure in the spring semester, it is interesting to note that the risks about user interface mismatch remain the same in the spring semester. The result of the summary of risk ranking implies how difficult or significant each risk is, while the percentage of risk occurrence refers to how long the risk lasts. A comparison between Figure 2 and Figure 3 has shown that in general, the more difficult the risk is, the

longer the risk lasts. There are also some exceptional cases such as COTS and independently evolving systems risks, although Figure 2 shows that risk occurrence remains the same in the spring semester, but Figure 3 shows that the team put less weight to this risk possibly because this risk is beyond the team s controllability. During the product implementation and transition, the COTSs are working fine, however, since the team and the client have no control over its next release, so there is always a risk of future loss of interoperability or support from COTS vendor due to the changes from COTS. Table 2 reports the comparison between the top ten risks in industry in 1991[1] and a survey of the top ten risks in the industry in 2007 [2], as well as a summary of the top ten risk ranking based on 86 teams of the software engineering classes from 2005 2010. Since these teams are working in an academic setting where the group is only together for fifteen weeks in the first semester, there are many more challenges students face while identifying/mitigating risks in comparison to those who work in industry. Due to the time constraint, student team members face a high learning curve where they must understand the client s project requirements, as well as buy into the team and client dynamic. Misinterpreting and/or not fully understanding the client s requirements can result in the high risk of developing software that is not up to the client s standards/objectives, and may require having to revisit requirements later on in the development phase. The lack of buy-in from all personnel involved can result in the high risk of apathy where team members are not motivated to contribute. In addition, during two-semester project classes like CSCI577, there is a high team member turnover rate since some students do not continue onto the next semester. Losing a team member greatly impacts the progress of a project since responsibilities must be reassigned to existing members or, a new member may be brought in forcing the team has to re-calibrate. Table 2. Comparison of Top 10 risks Top 10 Software Risk Items (Boehm 1991) [1] 1. Personnel shortfalls 2. Unrealistic schedules and budgets 3. Requirement mismatch Top 10 Risks in Software industry (Boehm 2007) [2] 1. Architecture complexity, quality tradeoffs Top 10 Risks in Software engineering class (2010) 1. Architecture complexity, quality tradeoffs 2. Requirements volatility 2. Personnel shortfalls 3. Acquisition and contracting process mismatches 4. User Interface mismatch 4. Budget and schedule 5. Gold plating 5. Customer-developer-user 3. Budget and schedule constraints 4. COTS and other independently evolving systems 5. Customer-developer-user team cohesion 6. Requirement volatility 6. Requirements mismatch 6. Requirements volatility 7. Shortfalls in externally furnished components 7. Personnel shortfalls 7. User interface mismatch 8. Shortfalls in externally performed tasks 8. COTS 8. Process Quality Assurance 9. Real-time performance shortfalls 9. Technology maturity 9. Requirements mismatch 10. Straining computer 10. Migration complexity 10. Acquisition and

science capabilities contracting process mismatches Comparing to the industry settings, these challenges do not traditionally exist in the industry since many software engineering projects take years to complete. By increasing the development time of a project, the team has more time to gain a better understanding of the client s needs and to determine requirements that are agreed upon by all stakeholders before implementation begins. This lowers, and sometimes even mitigates, the risks of requirements mismatch/volatility. In addition, the more time a team has to work together on a project, the more time they have to mitigate group dynamic and personnel shortfall risks before team cohesion becomes crucial to the success of the project. Since there are also monetary motivations while working in the industry, project members have more of a vested interest in contributing to the project and seeing it succeed. A student is mostly motivated by grades, and often striving for high grades is not of value to some students and their performance in team projects is an additional project challenge. Figure 4. Relationship between Risk Management and Team Performance 4. Risk Management Process and Team Performance In 2008, a survey was conducted to assess how risks have impacted the teams project progress and how the Incremental Commitment Spiral Model s effectiveness in managing risks. Five questions were asked of eight teams in the spring semester. Figure 4 reports the results of the survey. The vertical axis represents the rating value and the horizontal axis represents the teams, which are ordered by their performance (grade). Dotted lines represent

their perspective about the risk management in the fall semester, and solid lines represent their perspective for the spring semester. Based on their self-evaluation, in general there is a relationship between the grade performances of the groups as well as how well they believed they worked as a group in mitigating risks. Looking at Figure 4 from a very high level, there is a negative slope. This means that the lower their risk assessment and management ratings (1 = unsuccessful and 10 = very successful), the lower the grade they received on the final project. Therefore, there is a correlation between group success as a team, its ability to identify and manage risks, and the overall project outcome. However, Group C shows the exceptional case. In the fall semester, Group C did not perform well and has a high personnel turnover while in the spring semester, Group C got new team members who rescued the team. The dotted lines for Group C shows that they realized that they have not managed the risks well in the fall semester, but they have improved in the spring semester, reflecting their team s performance. On the other hand, in spring semester, Group F knows they have high architectural complexity risks, but they are struggling in managing and mitigating the risks, as shown in the light blue line graph. 5. Risk-Driven Development Process Risk identification supports the team in determining a course of action as illustrated at the bottom of Figure 1, and Figures 5 and 6. Based on the ICSM key principle about risk-based decision making, a risk analysis result can be categorized into four distinct types: A) If a risk is acceptable, the team can move forward to the next phase. B) If a risk is negligible, the team can skip a phase, or spend little or no time in the next phase. This situation could happen when the team finds a perfect commercial-off-the-shelf (COTS) or web service that provides all required functionalities. The team could skip the Valuation phase and Foundations phase since the team does not have to spend time prototyping or defining the architecture. C) If the risk is too high but still addressable, the team might need to repeat the current phase until the risk is mitigated and the team is ready to move forward to the next phase. D) If the risk is too high and unaddressable, the team should consider halting the project in order to adjust the scope and priorities or, possibly discontinue the project. This could happen when the project is found to be unfeasible within the defined budget or schedule. On the other hand, in the commercial setting, if the competitor has already launched a product that has similar functionalities, the team should redirect the project in order to gain a competitive advantage. Figure 5 shows a typical risk pattern in a software engineering project with acceptable risks, satisfactory feasibility evidence, and committed success critical stakeholders. Figure 5. Risk Pattern Driving in Repeating Exploration Phase [8]

Based on an analysis of the risk patterns in 14 software engineering projects in Fall 2009 Spring 2010, 9 out of 14 teams had addressable risks with smooth project progress toward the Operation phase. As shown in Figure 5, 3 of the 14 teams had unclear project scopes. The teams are not ready to move forward to the Valuation phase, hence the teams briefly spent extra time in the Exploration phase to discuss and gather more information from the successful critical stakeholders, exploring additional alternatives and conducting additional prototyping. Figure 6. Risk Pattern Driving in Repeating Adjusting Scope and Priorities [8] One team proceeded from the Exploration to the Foundations phases with acceptable risks, as shown in Figure 6. But during the Development phase, the team found that there were applicable Net-Centric Services. Hence, the team went back to the Exploration phase to perform detailed COTS assessments, and bypassed the Valuation and Foundations phases since the selected web service provides most of the required functionalities. 6. Conclusions Team project courses have difficult front-loading problems in that students need to do a just-in-time learning about how to select and organize team members, how to interact with project clients to understand their needs and environment, how to assess candidate off-theshelf packages addressing the needs, how to negotiate feasible requirements, how to develop project plans, and how to define the system s architecture. Initially, we did not include risk assessment and management in this up-front material, but soon found that it was often success-critical for teams to be able to identify, assess, and manage their risks. This involved us in preparing lectures, homework exercises, tools, and tutorials for risk identification, assessment and management that significantly increased the projects success ratings.

As the software field has evolved from having mostly programming-intensive projects to non-developmental-item / open source / COTS / cloud services projects, we have found that the risk patterns, sources, frequencies, and potential impacts have changed significantly. The data, guidelines and evolved processes presented in this paper, should help project courses incorporate risk considerations in their students learning venues. 7. References [1] Boehm, B., "Software Risk Management: Principles and Practices," original version of IEEE Software, Volume 8, Issue 1, January 1991, pp. 32-41 [2] Boehm, B., Top 10 Software-Intensive Systems Risk Items, Presentation at USC Annual Research Review [3] Boehm, B. and Bhuta, J., "Balancing Opportunities and Risks in Component-Based Software Development," IEEE Software, November-December 2008, Volume 15, Issue 6, pp. 56-63 [4] Boehm, B. and Lane, J., Using the Incremental Commitment Model to Integrate Systems Acquisition, Systems Engineering, and Software Engineering, Cross Talk, October 2007, pp.4-9 [5] Boehm, B. and Port, D., "Educating Software Engineering Students to Manage Risk," ICSE, pp.0591, 23rd International Conference on Software Engineering (ICSE'01), 2001 [6] CSCI577ab Software Engineering Class Website, http://greenbay.usc.edu/csci577/spring2013/ [7] Groth, D.P.; Hottell, M.P.;, "How Students Perceive Risk: A Study of Senior Capstone Project Teams," Software Engineering Education & Training, 2007. CSEET '07. 20th Conference on, vol., no., pp.45-54, 3-5 July 2007 [8] Koolmanojwong, S. "The Incremental Commitment Spiral Model Process Patterns for Rapid-Fielding Projects," PhD Dissertation, Department of Computer Science, University of Southern California, December 2010 [9] Koolmanojwong, S. and Boehm, B., "The Incremental Commitment Model Process Patterns for Rapid- Fielding Projects," ICSP 2010, Paderborn, Germany [10] Mead, N.R., et.al, "Ensuring Cost Efficient and Secure Software through Student Case Studies in Risk and Requirements Prioritization," hicss, pp.1-9, 42nd Hawaii International Conference on System Sciences, 2009 [11] Pew, R. W., and Mavor, A. S., Human-System Integration in the System Development Process: A New Look. 2007, National Academy Press. [12] Reifer, D., Ten Deadly Risks in Internet and Intranet Software Development, IEEE Software, March- April 2002, Issue 6, pp. 12-14 [13] Swart, R.S, Erbacher, R.F., Educating Students to Create Trustworthy Systems, IEEE Security and Privacy, v.5 n.3, p.58-61, May 2007 [14] Taran, G. "Using Games in Software Engineering Education to Teach Risk Management," cseet, pp.211-220, 20th Conference on Software Engineering Education & Training (CSEET'07), 2007