Simulation in Software Engineering Training

Transcription

1 Simulation in Software Engineering Training Anke Drappa (On leave from University of Stuttgart, Computer Science Department) Bühlenstr. 57/3 D Holzgerlingen, Germany ABSTRACT Simulation is frequently used for training in many application areas like aviation and economics, but not in software engineering. We present the SESAM project which focuses on software engineering education using simulation. In the SESAM project a simulator was developed. Using this simulator, a student can take the role of a software project manager. The simulated software project can be finished within a couple of hours because it is simulated in quick-motion mode. In this paper, the background and goals of the SESAM project are presented. A new simulation model, the so called QA model, is introduced. The model behavior is demonstrated by investigating and comparing different strategies for software development. The results of experiments based on the QA model are reported. Finally, conclusions are drawn from the experiments and future work is outlined. Keywords Software Engineering Education, Simulation, Training, Experiment, Quality Assurance, Software Process Modeling 1 INTRODUCTION Developing software systems, in particular large ones, is mainly a non-technical, not as it is often believed a technical problem. This is due to the fact that planning and sound analysis are essential. Those who have participated in real projects agree that poor planning is the most important reason for disaster [18]. The second most important risk is insufficient understanding of the users needs [13]. Communication and management are very important in every project, and even more so in developing software which is intangible. Managing a software project means first of all collecting all the necessary information, keeping the staff well informed, and coordinating the individual needs and tasks with the overall goals. The key to successful education is motivation. If we mention a source for free software in our lecture, most students will listen carefully, and apply that knowledge immediately, because they are affected, and they know they are. If we talk Jochen Ludewig University of Stuttgart, Computer Science Department Breitwiesenstr D Stuttgart, Germany ludewig@informatik.uni-stuttgart.de about software project management, they will hardly listen, because they cannot imagine that there is a real problem. They have developed lots of rather small programs on their own, and they expect the same kind of difficulties when a team does (what they believe is) the same kind of work. Therefore, our message that software project management is vital to software development will not really be appreciated. The best (and probably the only) motivation for learning software project management is experience from projects that failed due to insufficient management. After students have failed as software project managers, they will start asking questions, and they will listen to the answers. Then, the ground is ploughed and ready for seeding. But no curriculum, not even the comparatively long curriculum we offer in German universities (about five years for the diploma degree, which is the first and only degree except for the Ph.D.), allows for a sufficient number of life size projects so that every student can be a project manager. If software project management were something which is done very well in industry, we could trust in on-the-job training which is usually considered the remedy for everything that we fail to teach at university. But that does not work in a field where good examples are extremely rare. If we do not teach them, most probably nobody will, and poor software project management will be around for very much longer. Making Use of Simulation for Training When future aircraft pilots learn their profession, there is a similar problem: They need to know how to behave in dangerous situations, but nobody would like to expose them to such risk just for the sake of training. Therefore, airlines provide powerful simulators that model the whole aircraft and its environment except for the pilot. Using these simulators all difficulties can be experienced without any risk (though at considerable cost). We apply the same approach for educating software project managers. The project and its environment is simulated, the pilot, i.e. the software project manager (or simply the player) is real. This concept was named SESAM (Software Engineering Simulation by Animated Models). Even a long distant flight will not last longer than some ten hours, while a software project will last for a couple of months or years. Therefore, we compress the time scale, simulating a project in a couple of hours.

2 Simulation can improve our software engineering courses, but it cannot replace traditional teaching, because it affects the player s motivation, but not his or her skill. When the pilot has crashed his simulated aircraft that does not mean he has learned how to avoid such a disaster in the future. Motivation prepares the ground, but knowledge is the seed. 2 SESAM SIMULATION SYSTEM The player controls the simulator using a purely textual interface, i.e. by reading and typing text. She can hire or fire employees and ask them to perform any of the tasks which are useful for software development (like start preparing a specification or review the design document, or test the code). Most of the messages she gets are statements by her employees, like I have finished the specification, or During testing, I have detected n bugs. The player is well advised to study such statements carefully and to react in some appropriate manner, because this is all the information she will get. The simulator tracks many internal variables like the degree to which the specification reflects the requirements, but they are for internal use only; the player cannot see them. She is just as badly informed as any real project manager. When the game is over, the player receives her score. She can analyze her performance using the analysis tool, which graphically displays the internal variables. Maybe she will recognize that the number of deficiencies in the specs was too high because she had not made sure that counter measures were taken. At this stage, she can get all the information that she did not have while playing. In short: SESAM can be described as a laconic adventure game, which takes lots of notes while the game is proceeding. Afterwards, it becomes a rich source of curves and statistics, generously presenting all the data. The SESAM Simulator A simple simulator, which models a well understood system, could possibly be coded as one single program. SESAM, however, is neither simple nor is the reality it models well understood. Therefore, all the information about software projects that is applied in the simulator is well separated from the simulator itself. This information is called the model (i.e. the model of software projects in general, see below). By separating the model from the simulator, we make sure that our system is flexible and can be used with various models. When the SESAM project was launched in 1990, nobody had an idea what the simulator would eventually look like. Therefore, we developed a series of prototypes in Smalltalk- 80 [17]. After the final prototype was what we wanted, we started again from scratch, implementing the new system (called SESAM-2) in Ada95. The whole system (i.e. the simulator plus a couple of auxiliary programs, see fig. 1) has about lines of code. Most simulators are either continuous or discrete (event driven). In software projects, both mechanisms are present, because many variables change gradually (e.g. the size of a document that is being written), while others change suddenly after some event (e.g. a newly hired person s activity). Accordingly, the SESAM simulator offers both. In this respect, it is more flexible and realistic than System Dynamics which was used for a software project simulator ten years ago [1] and again recently in a project which was also focused on software quality [8, 21]. Basically, the simulator is running in a loop. For each time step, it will first look for any events that may have happened. When all immediate effects have been reflected in the game state, the state variables are updated according to any changes which are currently effective. Currently, we are using a working day as our basic time unit, but we will also use hours in the near future. player interface analysis tool interpreter rules depository rules compiler rules model dictionary static model depository static model compiler static model model builder interface log file simulator game state initial state compiler initial state Figure 1: Architecture of the SESAM System SESAM Models As mentioned above, the generic simulator is supplemented with a model of software projects in general. This model consists of two parts: The static model describes the types and relations that are available for describing a project, like person, document, or is reading (a relation between person and document). The rules model defines the dynamic properties, like while an employee is specifying, the number of errors in the spec increases. This dynamic model is very complex, because it comprises all preconditions and consequences of all the changes that are available. The rules model is clearly the most complex part of the whole system. The current state of the simulated project is represented by the game state. The initial game state, the scenario, is provided with the model described above. While the game is proceeding, the game state changes as determined by the player s actions and under the control of the simulator as determined by the previous state and the rules model. The game state is an annotated graph, whose nodes represent people and documents, while the edges stand for relationships (like person P is reading specification S ).

3 Fig. 1 shows the flow of information from the models (at the bottom) to the simulator. The simulator actually uses internal representations of the models (level above the compilers) which can be accessed efficiently. The dictionary is generated as a by-product when the models are compiled. The interpreter translates messages from the simulator to the player, and vice versa. The log file can be used for analyzing the project after it is finished. The Player Interface The player interacts with the simulator using natural language, in fact in simplified English that the simulator can generate and understand; if anything is not clear, SESAM will ask questions. The abstract messages SESAM sends to the player are translated into natural language using a dictionary. The same dictionary is used for analyzing the input. Therefore, the language in which the player communicates with the simulator is determined by the dictionary, which can be easily replaced. When a new game is started, the first message informs the player about the scenario. The following example is a slightly adapted version from a very simple setting, that was actually used. Donegal Wool Centre, a new customer, has chosen our company for their next software project. Actually, it is only the first part of a software project because the customer wants to see the specs before ordering anything else. You will be the project manager, congratulation! In your group there are two employees, Patrick Murphy and Rita Smith. The estimated size and complexity of the final system is about 200 AFPs. Please make sure that your group has finished detailed specs within two weeks. The player s goal is to make this project a success. She can talk to her employees, ask questions, charge them with certain tasks, or withdraw tasks. A dialogue could look like the following (user inputs are italicised, prompts are boldfaced): Aug 10> ask Patrick about his experience Patrick Murphy ponders over the question, before he answers: "Well, I think my experience is average." Aug 10> let specify Who shall specify? Patrick Patrick Murphy starts working on the specification. Aug 10> proceed 14 days Aug 24>ask Patrick about his work Patrick Murphy reports: The specs are now 30 pages long, I think it will soon be finished Aug 24> proceed 21 days The customer has called, he has received a 63-pages specs on Sep 7. He will check it. Sep 14> When the simulation is over, the player will receive her score. In this particular case, the score will probably be rather low because the player failed to make sure that her employees would do useful work. A typical score (for a larger project) looks like this: The project was finished after 362 development days. The total effort for system development amounted to 40.5 person months. The equivalent cost is 581,640 DM. The document Specification comprises 172 pages. It contains 390 AFPs. Thereby 97.6% of the customer's requirements are covered. The Specification still contains 74 errors. This is an average of 0.43 errors per page. The System Design consists of Additionally, the attribute values of the delivered system are presented at the end of the simulation. All results are related to the initial customer requirements. The customer required an error rate below 10.0 errors per thousand lines of code. The actual delivered system contains 10.8 errors per thousand lines of code. The correctness of the system is acceptable, though it does not meet the requirements. The code contains 10.8 errors per thousand lines of code (KLOC). These are 0.8 errors per KLOC above the limit set by the customer. 3 A SIMULATION MODEL QA MODEL In order to intensify our efforts in teaching software quality assurance, a complex simulation model was developed. This model concentrates on the effects of quality assurance (QA) activities; therefore it will be referred to as the QA model. Since the available knowledge about software projects in general is still very limited, such a model cannot be developed without serious effort, including a fair amount of research. Below, the formation of the QA model is outlined. Every model building process begins by specifying the purpose and scope of the model [6]. Here, the overall purpose is to demonstrate the effects of software project management on software quality. Most deficiencies and failings become apparent only in the later phases of a software development project, therefore modeling the whole project is important. Since projects differ widely, the kind of software project that is reflected in the model has to be specified. The QA model concentrates on software that is developed under a contract e.g. by an external software house [14]. Besides being comprehensive with respect to the software development process, the model is also fine-grained. The commands a player can give resemble actions taken in real software projects, like hiring staff members, defining and assigning working packages, or controlling the project s progress. The player has to deal with each of the individual employees. This makes her job a complex and demanding task. Model Assumptions In a first model building step, real software projects have to be observed and analyzed. Thus, the most important causeand-effect relationships that should be reflected in the model can be identified. Our analysis of software projects is mainly based on the software engineering literature with a specific focus on empirical investigations. In this section, some important assumptions contained in the QA model are discussed.

4 Qualifications of the Developers It was shown that the qualifications of the developers taking part in a software project strongly influence the project outcome. Developers productivity varies greatly depending on the individual skills [22]. The experience concerning a development activity, e.g. specifying or programming, and the knowledge of the tools, methods, and notations used are only two important factors proven to affect the individual productivity [3, 7, 22]. What is more, team members new to the project or to the application area typically are less productive than adequately trained people. Effort and Duration There is a close correlation between the duration and the effort spent for development activities. The more developers work on a result simultaneously, the earlier that result is available. Unfortunately, there is a second relationship: The more developers work on a document, the more effort is expended to achieve this result since the effort spent for interpersonal communication is growing [3]. Boehm also has shown that there is an upper limit for the team size and a lower limit for the duration in every software project [3]. Furthermore, extreme time pressure in a project leads to a decreasing productivity [9]. Reviews Furthermore, it is interesting to observe the effects of quality assurance activities. Reviews allow errors to be found early in the software development process. However, the error detection effectiveness varies greatly depending on the reviewers as well as on the review documents. The reviewers should be qualified and prepared [24], and the documents used as a reference to examine the candidate document should be as complete and as correct as possible. Reviews of non-technical documents (e.g. requirements specification, user manual) were shown to be more effective if the customer is involved [20]. [23] demonstrates reviews without meetings being more efficient while not being less effective. Correction of Documents Documents that have been examined should then be improved as quickly as possible. The later an error is corrected in the software development process, the higher is the cost induced by that error. Error correction is often most efficiently done by the documents author(s). Similar cause-andeffect relationships can be identified for other quality assurance activities. Objects and Attributes The cause-and-effect relationships sketched above should be reflected in the simulation model in a quantitative way. In SESAM, we use a four step modeling approach (fig. 2). In a first modeling step, the real world is analyzed, and a mental model of the real world is built (collection of model assumptions). Then, the relevant real world objects and relationships are identified, and their properties are described by attributes. Finally, a metric is assigned to each attribute so that a certain object property can be evaluated quantitatively during simulation. Figure 2: Four-Step Modeling Approach in SESAM The most important objects in the QA model are developers and documents (fig. 3). A developer is mainly characterized by his/her experience concerning methods, notations, and various software development tasks. Developers and documents are connected by relationships; the three most important relations are works on, examines, and reworks (fig. 3). Among others, the relations are described by the attributes productivity and error rate. Thus, it is easy to model that developers fulfill their tasks with varying time and quality. Figure 3: Objects, Relationships, and Attributes in the QA Model 15 Loc/Day 30 Pages Real Projects Model of the Reality SESAM Model Metrics (Contract) (Model Assumptions) Name Costs Experience Model Building works on examines reworks Productivity Error Rate Developer Document Name Content Errors A document is characterized by its content. To evaluate the content of a document quantitatively is a critical problem during model building. In the QA model, we use adjusted function points as an abstract measure that can be applied to most types of documents created during software development. The measure allows assessment of the project s progress during simulation and it also serves as a basis for all productivity values. A document also contains errors. Errors are defined according to [12] and are further distinguished by the activity in which they were introduced. Model Behavior The dynamic behavior of the QA model is defined in more than 400 rules. The rules allow changes to the simulation state, i.e. objects and relationships are added or deleted and attribute values are changed. The rules that have to be applied are determined by the commands the player enters during simulation. Thus, varying management approaches can be tried out for a certain simulated software project. The progress and the results of the project are characterized by the intermediate and final attribute values (like costs, duration, effort, number of errors, etc.).

5 In order to provide a realistic quantitative model behavior the parameterization of the QA model is based on empirical data. The most important data source was found in [14]; the data is sufficiently detailed, based on function points, and covers several thousand software projects. Model quantification was supplemented by data from other sources where necessary [23, 24, 25]. More information on the parameterization of the QA model can be found in [10]. 4 DEMONSTRATION OF THE QA MODEL Below, the quantitative model behavior of the QA model is demonstrated. First, it is shown how selected model assumptions are reflected in the model. Then, the results of complete simulated software projects are analyzed and compared. Selected Model Assumptions As discussed before, there is a close relationship between effort for and duration of a development activity. This relationship is mainly determined by the number of developers assigned to that task. In the following example, we compare effort and duration of the detailed design task when performed by two or four developers, respectively. All developers are assumed to be equally qualified. Two developers need 28 working days for the detailed design, while four developers have already finished the task after 20 working days, i.e. on y 14th, 1999 (fig. 4). However, the effort for the detailed design increases significantly to man hours (Mh) (vs Mh for two developers). Managers cannot speed up a software project without increasing the cost for that project. Completeness (fig. 4) and quality of the detailed design, however, is not affected by the number of developers assigned to a task. AFP/Effort in Mh Jun 17 Jun 22 Jun Mh Mh Figure 4: Detailed Design Effort of two vs. four Developers A second interesting effect that can often be observed is the exponential rise of cost when errors are detected and corrected late in the software project. Table 1 shows the effort for review and correction of several software documents (columns 2 and 3). The review of the system design, for example, takes man hours. The correction of the errors found in the review takes an additional man hours Date Effort (4) AFP (4) Effort (2) AFP (2) Assume now that a project manager observes the lack of quality very late so that reviews of the early documents are done only when the code is already written. In that case, errors not only have to be removed from the reviewed document but also from all successing documents including the code. For example, errors found in the system design review must then be corrected in the system design, in the detailed design document, and in the code. The late correction of errors causes a significant additional effort as shown in table 1 (right column). Correction after Review of Effort (Mh) for Review Correction of Reviewed Document Correction of all Documents Specification System Design Detailed Design User Manual Sum Table 1: Increasing Effort for Late Correction of Documents The correction of errors in the specification only takes man hours if done immediately. If not, improving the specification and all following documents consumes man hours. This is an increase of approximately 4000 %! Simulation demonstrates that it is terribly inefficient to skip early reviews. Process Comparisons Project managers are often in doubt as to which software development process is most suitable for a given project task. Only simulation can answer this question. In this section, a software project is simulated by using four different development strategies, while all other parameters remain constant. Thus, the influence of the management approach taken on the overall project outcome can be analyzed. Assume that a software house has signed a contract to develop a new software system for the administration of client data. A preliminary function point analysis resulted in 400 AFP. Table 2 shows four different development processes that are used to simulate the client administration project. Process A is very systematic: All required documents are developed, all reviews (except code reviews) and four different tests are conducted (table 2, second column). In process B all reviews are skipped (third column). In process C no tests are performed, except the acceptance test which is done by the customer (forth column). An unsystematic, iterative code-&-fix approach is assumed in process D (fifth column). The rightmost column in table 2 shows the number of developers (with average qualification) assigned to the specified tasks, if performed in the project.

6 Activities A B C D #Dev. Analyze x x x x 1 Specify x x x 2 Rev. of Specification & Corr. x x 2 Write User Manual x x x 2 Rev. of User Manual & Corr. x x 2 Design System x x x 3 Rev. of System Design & Corr. x x 3 Specify Modules x x x 4 Rev. of Detailed Design & Corr. x x 4 Write Code x x x 2x 4 Module Test & Corr. x x 2x 4 Integration x x x x 2 Integration Test & Corr. x x 2 System Test & Corr. x x x 2 In the second simulation run, process B is applied. The manager of that project wants to save resources and does not care for reviews. The results of the project are shown in table 3. Due to the inferior quality assurance, the code quality is lower. The code contains errors per thousand lines of code, the defect removal efficiency significantly decreases to 66.59%. Additionally, the code completeness is lower than in project A. However, most surprising is that process B caused more effort (6.4 % or Mh) than process A even though the time for reviews in the early project phases was saved. To analyze this effect more closely, the effort needed for the different development tasks is compared for project A and B. Fig. 5 illustrates that process B helps to save effort in the early project phases. However, after coding the effort spent for each task significantly increases compared to process A. In the late project phases, much effort is devoted to error correction. As shown before, correction is more expensive the earlier the errors are introduced. Since the reviews were not performed in project B, many errors could not be found until testing began. Acceptance Test & Corr. x x x x 2 Table 2: Definition of Development Processes The development of the client administration system needs man hours (36.61 person months 1 ) when process A is applied (table 3). 47% of the effort is spent before, 20% in, and 33% after coding. Boehm recommends an effort distribution of 65:15:20 % [4]. The delivered system comprises 18,898 Lines of Code (Loc) in a third generation programming language (Ada), its defect density is errors per thousand lines of code (KLoc). This result fully agrees with other empirical investigations, e.g. [11,15]. The defect removal efficiency in the code amounts to %, i.e % (or 197) of all errors remain in the code when the system is delivered (table 3). Additionally, the team developed more than a thousand pages of high quality internal and external documentation (specification, system design, and detailed design; user manual). Use of Effort (Mh) Completeness Defect Density Def. Rem. Efficiency ProcessA % % ProcessB % % ProcessC % % ProcessD % % Effort (Mh) Specification User Manual System Design Detailed Design Coding Module Test Integration & I. Test System Test Acceptance Test Effort (A) Effort (B) Activities Figure 5: Effort for Development Tasks in Projects A and B If a compromise between effort and quality must be found, leaving out tests is in any case more efficient than cancelling reviews. This is demonstrated by process C. When applying process C in simulation, the resulting code quality is similar to that of process B (table 3). However, the effort to achieve these results is significantly lower than for process B (7.7 person months). Process D finally sketches a typical code-&-fix approach with several iterations of coding and testing activities (table 2). The resulting code quality is completely insufficient (table 3) and is not accepted by the customer. Table 3: Simulation Results for Processes A, B, C, and D man hours per person month according to [14].

7 Effort (Mh) Specification User Manual System Design Detailed Design Coding Module Test Integration & I. Test System Test Acceptance Test Figure 6: Effort for Tasks of Process D Correction The performed tests are unsystematic and ineffective since documents from the early phases that could serve as a reference are missing. A predominant part of the effort spent in the system and acceptance test phases is in fact devoted to error correction (fig. 6). Model Validation The QA model was carefully verified and validated. Whereas model verification deals with building the model right, model validation deals with building the right model [2]. Simulation experiments replace experiments with the real system that often as in software project management cannot be performed. Therefore, validation should make sure that users of the simulation system come up with the same decisions as they would with the real system [16]. Typically, simulation models contain random variables. Validation is then based on a large number of simulation runs to make sure that the real system is approximated sufficiently well by the simulation model. In the QA model, random effects only play a minor role. Thus, validation is done by studying varying processes and projects. The simulation results are compared to real data wherever possible. Currently, model validation suffers from a lack of empirical data which is comprehensive, detailed, and supported from a broad statistical background. Besides others, we used the COCOMO II cost estimation model to check the simulation results [3, 5]. The COCOMO model is based on empirical data. An excerpt of the comparison is shown in the remainder of this section. We use again the client administration scenario, and vary the project size. Three project tasks comprising 200, 400, and 1000 AFP, respectively, are defined, all other parameters remain unchanged. Then, the projects are simulated using process A (table 2) which is also called the reference process. Results of the projects are shown in table 4. The COCOMO II estimation model provides sixteen estimation parameters so that the project conditions defined for the client administration project can be mapped easily to Test Development Activities COCOMO. All COCOMO parameters were rated nominal, except RELY, CPLX, TOOL, and SITE, which were rated high. The five project parameters provided in the COCOMO II model were set to high as well. As shown in table 4, the effort estimated by the COCOMOII system differs only slightly from the effort computed by the QA model (9% for the 200 AFP project, less than 3% for the other two projects). Consequently, the QA model shows realistic behavior. Model Attribute 200 AFP Project 400 AFP Project Table 4: Simulation vs. Estimation Results 1000 AFP Project QA Model Effort (PM) Duration (Months) Team Size COCOMO Effort (PM) Duration (Months) Team Size There are larger differences with respect to the duration, but the estimated team size is higher than the team size chosen in our simulation runs. Therefore, the results are consistent. Further investigations and checks against another estimation tool led to a similar congruity. 5 EXPERIMENTS WITH THE QA MODEL The primary purpose of SESAM is to improve the education of software project managers. The theoretical lessons are to be complemented by simulation exercises which allow students to gather realistic management experience. To check whether the simulation based on the QA model helps to improve project management education, two experiments were conducted. Experimental design as well as the results of these studies are presented in this section. We distinguish two types of investigations and use the term experiment as a generic term for both. The controlled experiment is a repeatable observation under controlled conditions, which aims at validating a hypothesis. The case study is a weaker but cheaper validation form, typically without a control group. Experimental Design The underlying question was whether or not players learn when using the SESAM system. For the experiment, a more precise hypothesis was formulated: A learning effect can be stated if a player possesses more software engineering knowledge or a player has improved his/her project management abilities [9].

8 To validate the hypothesis stated above, a case study and a controlled experiment were conducted. The case study helped to widen the data basis and to check the experimental design. For both experiments a pretest-posttest design was chosen. The knowledge state is recorded before the simulation (pretest), then the simulation is done (stimulus). After the simulation, the state of knowledge is recorded again (posttest). The case study can demonstrate possible changes in the knowledge of the players. The controlled experiment additionally assures that these changes are caused by the simulation only (i.e. not by other influences). Case Study Controlled Experiment Experiment Group Only Experiment Group Control Group 9 Participants 10 Participants 9 Participants Pretest: 5 Games: Introduction 200 AFP (duration) 200 AFP (quality) 300 AFP (duration) 300 AFP (quality) Posttest: Pretest: 2 Games: 200 AFP (duration) 300 AFP (duration) Posttest: Table 5: Experimental Design Pretest: Posttest: The experimental design of the case study is shown in table 5. All nine participants were third-year students majoring in computer science with a special focus on software engineering. To pass the pretest, the students had to fill in a questionnaire (comprising 45 questions) within one hour. Additionally, they had to prepare a project plan for a given project. Then, the students had to simulate five projects. The projects were of different complexity and had different goals. The first simulation run was used to introduce the students to the SESAM system. In two projects, the students goal was to minimize project duration. The objective of the other two was to develop a high quality system. Finally, the students had to pass the posttest, which was in fact a repetition of the pretest. The design of the controlled experiment is similar to that of the case study (table 5). For that experiment, we used a total of 19 second-year computer science students. The experiment group passed pretest, stimulus, and posttest as discussed before. Since the effort for the simulation was underestimated in the case study, the students simulated only two projects (instead of five). The control group only passed pre- and posttest. Results To assess the performance of a student, the results of the pretest and the posttest are compared. A relative improvement in the consecutive simulation runs plays a minor role in the assessment. While the evaluation of the questionnaire is simple (counting the number of correct answers), the evaluation of the project plans and of the simulation runs is more complicated. We translated each project plan into a simulation run and defined a complex evaluation function to assess the results of the simulation runs. This function takes into account the results of the simulated project (e.g. effort, duration, or system quality) with respect to the given project goals. Case Study The results of the case study are shown in table 6. An entry Y (Yes) denotes an improvement in the posttest. The numbers in brackets show the relative changes for the number of correct answers (column Comp.) and the quality of the project plan, respectively. Except for persons 6 and 9 there are only minor changes in performance. The simulation runs are only considered when there is no unambiguous result from the first two comparisons (person3). Participants Comp. Comp. Simulation Runs Table 6: Results of the Case Study Improvement? Person 1 Y (+2) Y (+0,6%) (Y) Y Person 2 N (-1) N (-6,8%) (N) N Person 3 Y (+3) N (-6,9%) N N Person 4 Y (+4) Y (+4,9%) (Y) Y Person 5 N (-1) N (-3,4%) (N) N Person 6 Y (+4) Y (+20,6%) (Y) Y Person 7 N (+0) Y (+9,1%) (Y) Y Person 8 N (-3) N (-8,3%) (N) N Person 9 Y (+1) Y (+20,6%) (Y) Y In total, five students improved their performance while the other four students did not. Consequently, a learning effect caused by the SESAM system could not be stated. Controlled Experiment Table 7 shows the outcome of the controlled experiment which was finished by 18 out of 19 initial participants. Columns 1 to 5 refer to the results of the experiment group, columns 6 to 10 show the results of the control group. In the experiment group, seven students improved their performance while two did not (column5). However, since we can observe the same improvement in the control group, a learning effect caused by the SESAM system could not be shown either (column 10). A more appropriate hypothesis may be that a three step process (playing, teaching, playing) is more successful than just teaching. Experiments for testing that hypothesis are currently being conducted. These experiments are based on those reported above, and follow the same approach. The effort for such experiments is considerable.

9 EG a QC PC SR Imp? CG QC PC SR Imp? P. 1 Y Y (Y) Y C. 1 Y Y (Y) Y P. 2 N N (N) N C. 2 Y Y (Y) Y P. 3 Y N Y Y C. 3 Y Y (Y) Y P. 4 Y Y (Y) Y C. 4 N N (N) N P. 5 Y Y (N) Y C. 5 N Y Y Y P. 6 Y N Y Y C. 6 Y N Y Y P. 7 Y Y (Y) Y C. 7 Y N N N P. 8 Y Y (Y) Y C. 8 N Y Y Y P. 9 Y N N N C. 9 Y Y (Y) Y a. EG=Experiment Group, CG=Control Group, QC= Comparison, PC= Comparison, SR=Simulation Runs, Imp=Improvement Table 7: Results of the Controlled Experiment By-Products In the experiments, we noticed a couple of surprising phenomena. These observations may be used not only for improving SESAM but also for teaching Software Engineering more successfully. Students do not like planning Most participants did not plan the simulated projects. This typically caused a chaotic course of the project. While the players organize the early phases well, they lapse back into code-&-fix in the later phases. In the case study, only one student prepared plans. This student achieved significantly better results than his colleagues. Students commit the same mistakes again and again In the experiments it became apparent that many errors and failings from the pretest were repeated in the posttest, e.g. the process model used or the effort estimated for different project tasks. Thus, there was no learning effect that helped to avoid mistakes in the next software project. Students do not reflect on the details of their scores In order to become a better project manager, the player should know what was wrong in his or her project. Therefore, a detailed analysis of the results is necessary. But we found that the players did not take the effort to analyze their results. One reason could be that the output is not yet very well presented. We have to improve that component in order to make a sound analysis of the results much easier. However, this kind of final project evaluation was insufficient. The students did not analyze the course of the project and the results achieved. When the final score was already fairly high, they did not change their approach in the next simulation run. 6 CONCLUSIONS AND FUTURE RESEARCH To date, the SESAM project has produced three results (not to mention many by-products). A universal tool for simulation was specified, designed, and implemented. This tool is particularly useful for simulating software projects. A large, complex set of rules (a model) describing the dynamics of a software project was developed. These rules are based on data about projects available in text books and papers; some additional data was collected in academic and industrial projects. The complete system (simulator and model) provides a powerful environment for project manager training. In a couple of well organized experiments, the effects of using SESAM for project manager training was investigated. Our results do not suggest that people who use SESAM without any complementary training improve their abilities. As stated in chapter 5, more experiments are being conducted. They will show which benefits can be expected from using SESAM, and how SESAM should be integrated into software engineering courses. Neither the simulator nor the model are perfect. The simulator is continuously being improved. Performance is a critical aspect; models which are much more complex than the QA model would paralyze the simulator. Therefore, it may be necessary to improve the data structures and algorithms used in SESAM. The QA model serves as a foundation for other models which focus on various aspects of the software development process. Metrics is another area where more research is necessary. All models used in SESAM are based on metrics. But most metrics in software engineering are not very well defined, and their relationships are obscure. We also need far more data from real software projects. In short: Software engineering is a strange field because its message is extremely simple, but still too demanding for most people. They pretend they understand, but their behavior shows that they do not. SESAM is a simulation system which can demonstrate the same simple message any book can tell, but with SESAM the user has a chance to apply his or her insight, and achieve better results. In our current and future work, we try to find an appropriate mixture of lessons, playing, reflecting on the results, and playing again. Our first experiments have shown that just playing is not enough, because the students are not able to understand the reasons why they have failed. Their behaviour while playing is very similar to their behaviour when they actually develop software: They do not really trust in the lessons they (should) have learned, so they are controlled by their emotions. The SESAM system including the QA model and full documentation is available from our web pages: ACKNOWLEDGEMENTS Three anonymous referees have contributed to the final version of this paper. Many thanks to them and to Paul Hulbert in Bristol, England, who has polished our English.

10 REFERENCES 1. Abdel-Hamid, T.K., Madnick, S.E. Lessons Learned from Modeling the Dynamics of Software Development. Comm. ACM 32, 12 (Dec. 1989), Balci, O. Validation, Verification, and Testing Techniques throughout the Life Cycle of a Simulation Study. Annals of Operations Research 33, (1994), Boehm, B.W. Software Engineering Economics. Prentice-Hall, Englewood Cliffs, Boehm, B.W. Industrial Software Metrics Top 10 List. IEEE Software 4, 5 (1987), Boehm, B., Clark, B., Horowitz, E., Westland, C., Madachy, R., Selby, R. Cost Models for Future Software Life Cycle Processes: COCOMO 2.0. Annals of Software Engineering 1, (1995). 6. Bossel, H. Modeling and Simulation. Vieweg, Braunschweig, Bryan, G.E. Not All Programmers Are Created Equal. In: Thayer, R.H. (Ed.) Software Engineering Project Management (2nd Ed.). IEEE Computer Society Press, Los Alamitos, USA, 1997, Collofello, J.S., et al. Using Software Process Simulation to Assist Software Development Organizations in Making Good Enough Quality Decisions. Summer Computer Simulation Conference, 1998 (SCSC98). 9. Drappa, A. Quantitative Modellierung von Softwareprojekten. Doctoral Dissertation, Shaker-Verlag, Aachen, Drappa, A., Ludewig, J. Quantitative Modeling for the Interactive Simulation of Software Projects. Journal of Systems and Software 46, No. 2-3 (April 1999), Fenton, N.; Pfleeger, S.L. Software Metrics A Rigorous & Practical Approach (2nd Ed.). International Thomson Computer Press, London, IEEE Std IEEE Standard Classification for Software Anomalies. Institute of Electrical and Electronics Engineers, Jackson, M., Zave, P. Deriving Specifications from Requirements: An Example. Proceedings of the 17th ICSE, (1995), pp Jones, C.: Applied Software Measurement Assuring Productivity and Quality (2. Ed.). McGraw-Hill, New York, Kouchakdjian, A., Basili, V.R. Evaluation of the Cleanroom Methodology in the SEL. Proceedings of the Software Engineering Workshop, NASA Goddard, Greenbelt, USA, Law, A.M., Kelton, W.D.: Simulation Modeling and Analysis. McGraw-Hill, New York, Melchisedech, R., Deininger, M., Drappa, A., Hoff, H., Krauß, S., Li, J., Ludewig, J., Mandl-Striegnitz, P. SESAM - A Software Engineering Education Tool Based on Graph Grammars. Bulletin of the European Association for Theoretical Computer Science (EATCS) 58, 1996, Metzger, P.W., Boddie, J. Managing a Programming Project (3rd Ed.). Prentice Hall, Englewood Cliffs, New Jersey, Porter, A.A.; Johnson, P.M. Assessing Software Review Meetings Results of a Comparative Analysis of Two Experimental Studies. IEEE Transactions on Software Engineering 23, 3 (1997), Pressman, R.S. Software Enginering A Practitioner s Approach (4th Ed.). McGraw-Hill, New York, Rus, I., Collofello, J., Lakey, P.: Software Process Simulation for Reliability Management. Journal of Systems and Software 46, No. 2-3 (April 1999), Sackman, H. et al. Exploratory Experimental Studies Comparing Online and Offline Programming Performance. Comm. ACM 11, 1 (Jan. 1968), Votta, L.G. Does every Inspection need a Meeting? ACM SIGSOFT Software Engineering Notes 18, 5 (Dec. 1993), Weller, E.F. Lessons from Three Years of Inspection Data. IEEE Software 10, 5 (1993), Wheeler, D.A., Brykczynski, B., Meeson, R.N. Software Peer Reviews. In: Thayer, R.H. (Ed.) Software Engineering Project Management (2nd Ed.). IEEE Computer Society Press, Los Alamitos, USA, 1997,