Software Development Using Daily Build Concept at Ericsson Radio Systems AB

Transcription

1 Software Development Using Daily Build Concept at Ericsson Radio Systems AB Tomas Myrberg Reg. no. LiTH-IDA-Ex Examiner Kristian Sandahl

2

3 Abstract Abstract Daily build is a software development model that is based on a very simple principle: build and test the entire system very frequently, preferably every day. This process should be handled automatically in order not to increase the workload on the designers. A number of benefits will then arise, such as minimization of integration problems, fast feedback on project progress, detection of errors early and the avoidance of placing testing too late in the project. A requirement to be able to use the daily build concept is to also utilize the incremental development lifecycle model. Department Wideband Radio Networks at Ericsson Radio Systems AB strives to introduce the daily build concept, and this master s thesis was aimed at developing a tool to handle the automation of the process. Furthermore, a study of two other departments at the company, which already have initiated their own daily build processes, was performed in order to find out how their solutions was implemented. The experience from these two department shows that daily build contributed to higher software quality and better project control, even though planning becomes more complex when incremental development is used. The increased control and quality benefits coincides with the goals Wideband Radio Networks management has stipulated for daily build.

4 Abstract

5 Table of contents Table of contents 1 Introduction Ericsson Radio Systems AB The master s thesis Purpose of the report Structure of the report Reading recommendations Method The report Interviews Literature Critique concerning the method Implementation of daily build tool Lifecycle model Semi-prototyping Critique concerning the method Wideband Radio Networks Organisation Product development Why daily build? Daily build Problems with standard methods Daily build concept Basic concept Broken builds Daily build under pressure Frequency of builds Benefits of daily build Drawbacks of daily build Impact on development process Automatic testing pitfalls Scope and purpose of automation Saving time? Change Value of automation First automation project The daily build tool Implementation Choice of language Choice of design model Functionality Build Tests and sms messages HTML report pages...17 i

6 Table of contents Javadoc User interface Setup file Tool invocation Build and test report Daily build at Wideband Radio Networks Daily build usage Organization and scope Processes Build frequency Automatic tests The daily build tool Daily build at OSS Application Design & Services Daily build usage Benefits of daily build The daily build tool Daily build at BSC Development Daily build usage Benefits of daily build The daily build tool The old tool The new tool Comparison of the departments daily build usage Organisation and lifecycle model Team organization Incremental development Build frequency Processes and instructions Daily build tools Maintainability Performance Automatic tests Reports Recommendations Feature teams Documentation Automatic testing Start-up project Terminology References...45 Index...47 ii

7 Introduction 1 Introduction This report constitutes a master s thesis at the Master of Science program Computer Science and Engineering at Linköping University. 1.1 Ericsson Radio Systems AB The master s thesis has been performed at Ericsson Radio Systems AB in Linköping (Center for Radio Network Control). The task of the company, as described by their own presentation on the web, is: The unit holds the responsibility for the design and development of central parts of software used in all the current cellular systems as well as future wideband systems. The unit is a competence center within the areas of Operation and Maintenance and Security Management for cellular systems. The Center for Radio Network Control is also responsible for development and marketing of telecom simulators, a Supply Unit for Software Products and a department for Research and Innovation, which has a close cooperation with the Linköping University. More precisely, the work has been performed at Wideband Radio Networks, also known as department LVA/U, one of several departments that employs about 70 of the 850 employees. Wideband Radio Networks is further described in chapter 3, Wideband Radio Networks, on page The master s thesis Most of the time spent on the master s thesis has concerned implementing a daily build tool to be used by the department. The concept of daily build is explained in chapter 4, Daily build, and the tool itself is briefly described in chapter 6, Daily build at Wideband Radio Networks. Furthermore, the thesis work has resulted in this report. The programming part will however not be described in detail here. 1.3 Purpose of the report The aim of this report is to: Describe the daily build concept. Describe the daily build work at departments LVA/K (BSC Development) and LVA/L (OSS Application Design & Services). Describe and to some extent evaluate the daily build plans at Wideband Radio Networks and at the same time describe the functionality and intended usage of the implemented daily build tool. Compare the daily build work of departments LVA/K, LVA/L and LVA/U. 1

8 Introduction 1.4 Structure of the report This is the structure of the thesis: 1. Introduction This chapter explains the purpose and scope of the master s thesis and briefly describes the structure of this report. 2. Method The method chapter explains the methods used during the master s thesis. 3. Wideband Radio Networks Chapter 3 describes the department Wideband Radio Networks. It also accounts for the reasons stated by department management to initiate the daily build work. 4. Daily build This chapter explains the concept of daily build and what benefits one might expect when using it. 5. The daily build tool A major part of the master s thesis dealt with implementing a tool that automates the daily build process. This chapter describes the result. 6. Daily build at Wideband Radio Networks This is where you find a description of how daily build will be performed at Wideband Radio Networks. 7. Daily build at OSS Application Design & Services OSS Application Design & Services has been using daily build for some time, and this chapter tells how. The tools and methods used are explained and the benefits experienced by the users are accounted for. 8. Daily build at BSC Development Like OSS Application Design & Services, BSC Development has already started a daily build project. This chapter explains how, as in chapter Comparison of the departments daily build usage By comparing the daily build work and tools from the three departments, one can extract some conclusions of how Wideband Radio Networks should manage their daily build work. 10. Recommendations This chapter contains some recommendations to Wideband Radio Networks concerning the usage of daily build. 11. Terminology Definition and explanation of some words and concepts used in this thesis. 12. References An index of references used to gather the information in this thesis. Index A list of words to look up. 1.5 Reading recommendations Those who know nothing about Ericsson Radio Systems AB and the current daily build activities should read the entire thesis to grasp the subject. If you only are unfamiliar with the topic daily build and would like to learn some more, read chapter 4, Daily build to chapter 10, Recommendations, but focus on chapter 4. In order to find out more about the implemented daily build tool itself, reading [7] is recommended. The reader is advised to use chapter 11, Terminology, on page 41 when unfamiliar expressions are found. 2

9 Method 2 Method The method used for retrieving the information in this thesis is presented in this chapter, as well as how the requirements and usability demands of the daily build tool were gathered. 2.1 The report The primary source of information to this report has essentially been interviews and study of literature Interviews To be able to evaluate the daily build process at Wideband Radio Networks, some interviews have been made. The persons interviewed are involved with the daily build work at other departments of Ericsson Radio Systems AB, namely BCS Development and OSS Application Design & Services 1. The questions asked concerned the usage of daily build and the benefits they have experienced from it, the tools they use, their functionality and usability and finally the follow-up work performed. Interviews at Wideband Radio Networks have also taken place to enquire the departments plans for daily build Literature Books partly dealing with daily build and some articles concerning the subject have been studied. Test automation is commonly discussed in various papers, so papers on automation have also been read. Furthermore, documentation from BCS Development and OSS Application Design & Services dealing with their solutions was also studied, even though this documentation have been sparse Critique concerning the method Holding more interviews to gain a fuller understanding of other perspectives of the daily build process and tools of the other departments would have been interesting. Comparing this with the work at other companies would have been even more interesting, but this was not prioritized and was never performed. 2.2 Implementation of daily build tool Most of the time allotted for this master s thesis has been spent on the implementation of the daily build tool. This section briefly describes how Lifecycle model The Code-and-Fix model as stated in for example [2] was selected. The main reason for this choice is the time saved in the activities that are not outstanding in a project involving only one person, such as vast planning, documentation and complex quality assurance measures. The main focus in this model is the code. The extreme approach where no documentation is produced was however avoided. Both a detailed project plan and a requirement specification was written before the coding was initiated. 1 Two persons at BCS Development and one person at OSS Application Design & Services have been interviewed, see [10] and [11]. One main interview took place at each department, followed by several follow-up occasions. 2 The ones interviewed were department manager Stefan Werna and section manager Lena Eklind. 3

10 Method Semi-prototyping The functionality of the tool and mainly the contents of the documentation it automatically generates has been developed in cooperation with future users and others who have taken interest in the product. They have especially in the beginning of the implementation work given feedback on how they would like to use the tool when a prototype has been presented. This was very valuable in the beginning of the project since it was not quite clear what the orderers wanted the daily build tool to be able to perform. Calling it prototyping could however be an exaggeration. No strict prototyping method has been used. Feedback would then have been received by evaluating questionnaires and documentation from interviews with the users. The response has rather been given to me on a voluntary basis when someone felt they had something to say about the product. Their evaluations have therefore not been controlled or supervised by me at any time Critique concerning the method The Code-and-Fix model may save some time at first glance, but since some important actions are not present, there is always a risk that a project using it can utterly fail. If a more secure method had been chosen, the master s thesis would most likely have consumed more time, but risk management and quality assurance would have been improved. Using a stricter form of prototyping could have given feedback earlier. Evaluations on voluntary and non-controlled basis tend to be postponed and incomplete. Some may have neglected to report any opinions they might have regarding the prototype, and others have even ignored the whole evaluation. Thereby there is a risk of dissatisfaction when the tool is used at a later time. Letting a team use the tool in the final phase to try it out would have been tremendously advantageous. But since their available time is rather limited, this has been scheduled to after the completion of this thesis. 4

11 Wideband Radio Networks 3 Wideband Radio Networks This chapter describes department LVA/U, Wideband Radio Networks, and their reasons to initiate a daily build project. Details concerning the daily build plans are found in chapter 6, Daily build at Wideband Radio Networks, on page Organisation Department Wideband Radio Networks is divided into four organizational units. The first one is the management team, and the other three are called sections. They are the RNC Element Management, the RNC Traffic and the Security section. The daily build tool will be used by the RNC Element Management section, which is further divided into separate teams consisting of about half a dozen designers each. 3.2 Product development The department is developing parts of the infrastructure for the third generation mobile telephone system WCDMA for Ericsson. The design responsibility is focused on the node Radio Network Controller (RNC). The RNC Element Management concept is briefly presented below. In order for the network operator to be able to configure the RNC and supervise it, a web based operation and maintenance system, the RNC Element Manager, is produced. It is designed so that the operator using a thin client, for example a PC or UNIX station with a web browser, through an IP intranetwork can load the RNC s Element Manager. A RNC Element Manager consist of a number of applications needed to manage the RNC. An example of an application is the RBS Manager which is used to set up the links with the base stations that the cellular phones communicates with. The RNC Element Manager applications are all implemented in Java. The RNC Traffic parts are however implemented in C++. Furthermore, the incremental development lifecycle model is used by all sections. 3.3 Why daily build? Department manager Stefan Werna states three main reason to introduce daily build, which all are described in section 4.3, Benefits of daily build, on page 9. Increased control Department management gains better control of current project status. They can see how far you have reached, not only the amount of hours spent on the project, which may differ from actual progress. Detection of errors sooner Frequent builds and tests reveal errors in the code sooner than if daily build was not used. This saves a lot of time. Earlier integration tests Testing the interfaces between different parts of the code early saves huge amounts of time when flaws are detected soon after their introduction compared to if they were found late in the project. 5

12 Wideband Radio Networks 6

13 Daily build 4 Daily build Daily build is hardly a new process. It has been used by various companies for more than a decade, but the name daily build was not commonly used until Cusomano and Selby published their book, see [1]. The process is still not widely spread, and this chapter will explain the basic concepts for those who are unfamiliar with it. 4.1 Problems with standard methods Those who are not using the daily build process often suffer from disadvantages stated in this section, as described in [1] for example. The software is often developed by separate teams, with limited or no contact. But sooner or later the subsystems need to be integrated, and this is when you discover flaws in subsystem interfaces. Depending on the gravity of the errors, actions required may vary from small code alterations to large time-consuming design modifications. Testing is often performed quite late in the project, in most cases when the coding is nearly completed. This testing phase is usually rather extensive in terms of time since the stability of the product has not been tested before. If errors found while testing are severe, vast amounts of time may be wasted while fixing it. Completing the debugging before the intended release date can then be a problem. Since some time has passed between the actual coding of the incorrect code and the discovery of the fault another problem arises. Perhaps the programmer does not remember what that part of the code was all about, and then have to waste some time trying to recall its purpose. Or even worse: the programmer may even have left the company, and no one else is fully aware of how that code works. If you are unlucky, the source of the error is not easily detected. It might be anywhere in the code, which may be quite extensive at the end of the project. Finally, project managers find it hard to estimate the current stability of the code and measuring the progress. If a certain amount of the time available for the current project has been spent, how are the managers to know if the same amount of the work has been done or if they are behind schedule? 4.2 Daily build concept As the name reveals, daily build concerns everyday activities. Even though frequent build or even weekly build might be more appropriate in some cases, the term daily will consistently be used in this thesis Basic concept The basic ideas are very simple and are described below. Build every day The entire system shall be compiled and linked every day. This requires a sufficient code growth and that the designers actually check in their alterations often into the configuration management tool used. Run tests every day Smoke tests or regression tests shall be executed every day when the build is successful. Whether the tests shall include the current increment s code or only test completed increments when incremental development is used, see section 4.5 on page 11, is optional. 7

14 Daily build Actually, the test part is not required to call your process daily build, but as stated in [2], without it the daily build can be considered meaningless. Perhaps the name daily build and test is more appropriate when tests are involved, but the and test part is often implicit. Keeping the tests up to date is of course necessary in order for them to be meaningful. If they do not evolve at the same rate as the code, they most likely will not find errors introduced after the last change in the test. Automating this process, both build and test, is advantageous. [3] claims that automation is even necessary to make the daily build work without requiring too much managing efforts. The automated build and test is then often run at night so it includes the latest changes made during the day. The result is presented in some way the next morning. Some manual effort can however not be avoided. Furthermore, all tests are not suitable for automation, see section 4.6 on page Broken builds If either the build or tests fail, you have a so called broken build. This is of course to be avoided. The goal of the daily build process is to keep the number of broken builds minimal. Breaking the build is strictly forbidden. It is utterly important that each build and test succeeds, and that the persons involved understand this. The one responsible for breaking it should be punished in some way according to [1] and [2]. Suggested punishments, that actually are used by some companies, are paying a fine, wearing a silly hat or having to be in charge of the daily build process. There is however a risk that this kind of punishments only ridicule the daily build process, and makes it into a game instead. Perhaps breaking the build becomes considered fun. Furthermore, this kind of punishments most likely is not compatible with Swedish traditions, in this case especially Ericsson s customs. [3] recommends that the selection of those responsible for the daily build supervision is performed carefully to increase the seriousness of breaking the build. If they are recognized as good designers and are at least informal leaders with some authority, they can influence the attitude of the other designers. This also requires that working with daily build is a high status activity. Additionally, if high management is informed each time a build is broken, the daily build process is taken more seriously. But if the build or test fails, the error must be found and corrected before the next build. Otherwise you loose some or all of the advantages with the process, see section 4.3. It is also hard to maintain a no-broken-builds attitude among the developers if the builds are broken too often, which contradicts the daily build goal stated above. In [1], [2] and [3] it is described that in extreme cases, the one responsible for the broken build will immediately be summoned to work, where he must correct the bug in order for the build to continue. Remember that the build and test is most often run at night, which makes this rule quite unpopular, but effective Daily build under pressure In critical periods of the project, when the workload is high, one might tend to view the daily build process as something that one does not have time for. That is exactly the opposite of how you are supposed to think according to [2]. When the programmers are stressed, they often make more errors and test their own code less, which makes it even more important to keep the daily build and tests running. Conclusion: discipline is required. Never halt the daily build process. 8

15 Daily build Frequency of builds To fully take advantage of all the benefits from the process, as described in section 4.3, a build and test should be executed each day. However, if the code growth is not large enough to justify this, a frequent build approach may be taken, as many companies do, see [1]. Builds and tests are then performed a few times each week. Some satisfies with once a week, and others even as seldom as once a month, even though this hardly justifies the name daily build. But as [2] points out, if a broken build appear, the impact on the benefits is larger the less frequent you perform the build and tests. If several weeks go by until you produce a non-broken build, practically all advantages are lost. 4.3 Benefits of daily build There are a number of benefits gained from daily build stated in [1], [2] and [3]: Minimization of integration problems Since interfaces between subsystems made by different programmers or teams are checked very frequently, integration problems are minimized and a big bang integration at the end of the project is avoided. Within an ordinary development model, the waterfall model for example, integration tend to take place late in the project, which may cause detection of bugs that are not that easily corrected. Redesign may even have to be made, or in extreme cases the whole project can fail. Fast feedback on project progress Code growth and error detection are reported on a daily basis. At any time you know whether the system is executable or not. The managers can easily tell if the time spent on the project corresponds to the amount of code written. Avoids placing test phase last Usually the test phase is placed rather late in the project, often too late. Daily build ensures that tests are executed from day one, or whenever tests are constructed. Improves programmer motivation One can literally watch the code grow every day, and seeing that it works and that your part is useful increases the programmer motivation. In [3] it is also stated that the stress on the designers is reduced. This is due to that high workload is avoided at the end of a project, when lots of problems ordinarily appears. The pressure is instead distributed more evenly across the entire project, as illustrated in figure 1. Designer s stress level End of project Not using daily build Using daily build Time Figure 1. The stress level is more evenly distributed when daily build is used. The figure shows hypothetical data. Errors are found quickly and are easily located When the interval between builds and tests is short, errors are found very soon after their introduction, and can therefore rapidly be corrected. This requires good tests, of course. Perhaps the error even makes the code uncompilable, which perhaps would not have been detected until the integration test phase late in the project. 9

16 Daily build A large amount of time can be saved when errors are detected early since the cost of correcting it increases dramatically the longer it takes to find it. This is shown in figure 2 which also illustrates that errors are found earlier when daily build is used. Number of errors found Using daily build Not using daily build Cost of correcting errors Module / unit test Integration test System test Client s usage Time / Test phase Figure 2. Errors are found earlier and with a lesser cost when daily build is used. This figure shows hypothetical data and is based upon the experience of Stefan Werna, the former manager of OSS Application Design & Services and now manager of Wideband Radio Networks. The latter part of the figure, system test and onwards, is however somewhat speculative, since the department had not reached those phases yet when Stefan Werna left. If the system passes both build and tests on day and either one fails on day + δ, where δ is the build interval, then the bug must have been introduced somewhere between these two days. If δ is small the code probably has not grown that much and the possible incorrect code sections can easily be isolated. If daily build is not used, locating the error can be a difficult task since it can be located anywhere in a rather large part of the code. There is always a deliverable product Each time a build succeeds you have a deliverable product, at least in theory. If someone needs an executable version of the product made so far, it is there. Brings testers and programmers tighter together By making testers and programmers work together one can bridge the gap between them. Increases accuracy of planning Since bugs are detected and fixed continuously and the system is kept stable at all times, the risk of breaking the time plan due to severe bugs decreases 3. Decreases leadtime [3] claims leadtime is reduced since the pulse of the project gets higher and system test can proceed in parallel with the development. Finally the amount of documentation can be decreased since the communication often only resides within the feature team and documenting it may become unnecessary. Communication with the test team, if any, also increases as stated above, which means some bugs they have found can be managed without complex error report handling. But this can also lead to a drawback as stated in section But note that the planning itself is not any easier just because daily build is used. Incremental development, which is required (see page 11), may on the contrary make the planning harder and more time-consuming, especially if this method is new to the persons involved. 10

17 Daily build 4.4 Drawbacks of daily build No development model lacks flaws, and neither does daily build. The problem that arises is a tendency towards premature releases, according to [2]. Even though no errors are found by the daily build tools at a specific time, the product may not be ready for an external delivery. This might not be understood by people outside the development group, and they therefore demands a release. A release requires that time have to be spent on preparations, including writing documentation, hiding features in the code that are still under development and cloaking debugging devices used. There is also a risk that quick fixes makes the code work for this release, but in the long run, those parts will have to be replaced. Thus that code will have to be written twice, which is not very efficient. 4 In [3] it is recognized that the decreased leadtime resulting from increased communication advantage also may lead to a drawback. The documentation can degenerate and become far too insufficient. This is particularly a risk if the documentation already was sparse before daily build was applied. 4.5 Impact on development process Using daily build impacts the development process. Incremental development Using the traditional waterfall model, no buildable or fully testable code is completed until very late in the project. This makes daily build impossible. In [6] and somewhat in [1], [2] and [3] incremental development is suggested instead, which heavily affects design. Feature teams and customer feature design [3] and [6] also recommends feature teams instead of teams based on system modules. A feature team is responsible for parts of all modules involving a certain customer feature, unlike system module teams where teams handle one or several modules each. That would make coordination and planning between those who are module responsible unbearable when a daily build approach is used. Feature teams implementing a customer feature also allows early prototyping. Testing The traditional test phases, such as module tests, integration tests and system tests, may disappear. Instead a feature team can include any tests in the daily build when it feels appropriate. It is however unlikely that all phases will be covered by the tests included in the daily build process. The later tests, including systems test and so on, will most likely be performed outside the scope of daily build. 4.6 Automatic testing pitfalls This section will briefly point out some important problems and misunderstandings dealing with test automation, even though it is outside the scope of this thesis. A more complete discussion can be found in [4] and [5]. Regression tests are not fully affected by the reasoning in this section. 4 The premature release flaw is however not outstanding at Wideband Radio Systems. Release dates are specified in advance and are not changed. 11

18 Daily build Scope and purpose of automation First of all, I would like to stress that automatic testing is not the final solution when one tries to discover bugs in the code. It can never replace manual testing and inspections, but is only a complement. Furthermore it is important to realise why one has automatic testing at all. It is not to be used because it makes testing more exciting or because it is considered to be a hot method. The only purpose is to find errors. If no errors are found, perhaps your automated test suites only verifies some functionality instead of searching for bugs Saving time? Perhaps the intended goal is to reduce the time spent on testing. That will never be achieved. Creating and maintaining a good test suite intended for automation take more time than running a manual test once. So for each automatic test suite you create, you will have less time to spend on running manual tests. And those are the ones that will find the most errors, even as many as 80% says [4]. So when deciding if to automate a test or not, you have to estimate what manual tests you will loose, and how many errors will go undetected. If those are many, and severe ones, you probably should not automate that test. The estimation is however not easy to do, but nevertheless important. Time is also not saved concerning documentation of test suites. This will still have to be thoroughly performed. A process and a test plan concerning test automation are also to be maintained Change Then consider the next issue: if a test passes the first time you run it, what is the chance it will fail the next time? The answer is negligible if the code has not changed in some way 5. Then you have to ask yourself Is it meaningful to repeat this test time after time?. Well, if the code does change, then another problem arises. Sooner or later the test suite will become obsolete since it is not adapted to the current code status. Then you either have to discard it or update it. The latter is not necessarily easily done, but might consume as much time as writing an entirely new suite. No matter what alternative you choose, if the suite has not found enough errors by now to justify the effort automating it, then you should have left it a manual test after all. So when determining if it should be automated at all, consider how many code changes it will survive. This is particularly to be contemplated if the code involves a GUI since those tend to change a lot during the development Value of automation So what are the benefits of automatic testing compared to the manual counterpart? Well, [4] claims that such an comparison is meaningless. They are two different processes, not two ways of executing the same one. The time spent on running test suites is often used as a measurement, but since consuming time is not the purpose of testing, the number of bugs found is a better way of measuring the value of the tests. So when deciding whether to automate a test or not, one should ask oneself Will I find more bugs if I automate this test?. [5] claims that An automated test s value is mostly unrelated to the specific purpose for which it was written. It s the accidental things that count: the untargeted bugs that it finds.. This is deduced from the fact that the error often is not introduced in the specific code you are testing. The error has appeared due to a change in code you are using in the tested code, and which was assumed to be correct. These kind of errors are more eas- 5 The exception is timing and load tests. 12

19 Daily build ily found by an automatic test since a manual test would not be rerun very often, if rerun at all. But expecting that the test will detect bugs exclusively in the designated code is consequently not necessarily correct First automation project When starting a test automation process, determining what to automate is not easy. Since no exact figures can be deduced when reasoning about automation and its problems, such as If I automate this test, I will loose exactly five manual tests which would have found seventeen errors, testing the automation process by starting a small scale experiment is recommended. First make sure there is a test plan. It ought to tell what to automate, how to automate it and how and by whom the tests will be created and maintained. Expected costs and benefits are also to be pinpointed here. Then start the automation in a very small scale. You can not automate all the tests for your system at once. Gather statistics from each test execution. Also make sure your test suites are not made weaker in order for them to be automatable. Now evaluate the test plan and the outcome of the tests automated so far. Consider feedback from those involved. This will hopefully give valuable experience of what to automate, how it shall be done and what benefits and cost it produces. The number of test suites used can now gradually increase. They must constantly be updated so they test the current features of the code, not just the old ones. 13

20 Daily build 14

21 The daily build tool 5 The daily build tool In order to handle the automation of the daily build process, Wideband Radio Networks needs a tool that handles the compilation and testing of their system with minimal manual involvement, and which also reports the result in a meaningful way. This chapter describes the tool that serves this purpose, and that was implemented as a major part of the master s thesis. A more thorough description of the daily build tool is presented in the user s guide, see [7]. 5.1 Implementation This section accounts for the choice of implementation language and design model Choice of language The tool is implemented in the script language Perl, version 5. A system programming language, such as C++ for example, was not chosen due to the following reasons, which are discussed in [8]: The daily build tool uses existing programs 6 to perform its tasks; There is no need to implement complex algorithms; String manipulation is heavily used when report pages are produced, see section 5.2.4, HTML report pages, on page 17; Execution speed is not a primary concern. The last point raises an important issue. A scripting language is undoubtedly slower than a system programming language, which commonly is the result of using an interpreter instead of a compiler and the fact that the language is designed for easy usage not for efficient mapping to the underlying hardware. Even though speed is an important factor for the daily build tool, the intention is that it can spend an entire night performing its task. The amount of data handled can therefore grow rather large before the execution time becomes a problem. Furthermore, the time you loose in execution speed you gain in development speed. It is stated in [8] that any programmer can produce that same number of lines of code in a certain time, no matter what language is used. Since scripting language code is very compact in one line you can perform tasks that require several dozen of lines in a system programming language development time is reduced. The limited amount of time to spend on this project calls for a scripting language. Perl was not selected after an extensive study of other scripting languages, but since it is commonly used at Wideband Radio Networks, which might benefit future maintenance. Perhaps another language should have been chosen since it is more efficient in some areas. But finding the best language could have meant implementing the tool, or parts of it, in each of the potential implementation languages and then comparing the result. This time-consuming exercise was however not prioritized. And since the most timeconsuming part of the execution in fact is not the execution of the script itself, but the compilation of the code, only a negligible amount of execution speed can be gained by changing implementation language. 6 These are the Cello build support tool (the compiler), the hardware simulator SimCello, the test component executor ttcn_execute, the ClearCase command line interface cleartool, the HTML documentation generator Javadoc and the UNIX tool sendmail as well as some UNIX system calls. 15

22 The daily build tool Choice of design model The object oriented features in Perl version 5 has not been used since there is no call for it in this case. Functional programming and a structure with several separate scripts and modules have been used. 5.2 Functionality The daily build tool automatically executes a number of steps when initiated: Build the system; Test suite execution; and SMS message sending; HTML report file and Javadoc documentation generation. No manual actions have to be taken once started. A setup file and arguments given at invocation control the tool s behaviour, see section 5.3.1, Setup file, and section 5.3.2, Tool invocation, on page 18. Every action performed is logged in log files Build The build is performed by Cello s build support tool, which is based upon clearmake. Compilation is controlled from specification files named build.spec where packages to be compiled are declared. The locations of the specification files can be found in the directories stated after the BUILDSPEC tags in the setup file. The version of each file to select is determined by ClearCase labelling. The actual result of the build is interpreted from the messages given by the compiler, which are directed to a log file. Only Java is supported by the daily build script and the jar files that are generated by a successful build are made available to the test component executor Tests All test suites are in the Tree and Tabular Combined Notation (TTCN) format, a choice made by the test team. The test component executor executes test suites based upon their respective configuration file. It communicates with the compiled code via test ports where signals are sent and received. If unexpected signals are returned by the code, an error has occurred. Errors as well as correct results are documented in log files. This is all illustrated in figure 3. Test suite Configuration file Test component executor Log files Test ports System under test Figure 3. Overview of the test case executor environment. Test suites and configuration files are selected in the daily build setup file, see section on page 18. The log files are used to determine the results of the tests. 16

23 The daily build tool The daily build tool interprets the log files after executing the test suites, and thereby determines the outcome. There is no support for automatic GUI testing and SMS messages Each time the daily build script is run, an is sent to each recipient specified in the setup file. It contains a summary of build and test results, as in figure 4. Choosing a newsgroup as one of the recipients makes the report widely available. Unless the SMS-MODE tag in the setup file is set to off, SMS messages are also sent to telephone numbers specified. There is also a failure setting which makes these messages only being sent if either build or tests fail. However, the contents is a limited version of the text in figure 4, since SMS messages only can contain a very small amount of characters. Result of daily build and test for 1 Dec Result of build: Success! Result of designer tests: Success! Result of test team tests: Failure! There are no build errors. All designer test suites passed. 3 out of 4 test team test suites didn t pass. The report is located at Figure 4. An example of the contents of an sent after completing build and tests HTML report pages The result of the build and test is presented in a number of HTML files that are automatically generated each time the daily build script is run. The intention is to make the data presented in a compact way where only the information needed is shown. Old reports are not deleted but are stored and easily accessed as well. Some simple build and test statistics are also maintained. There is a presentation of the HTML pages in section Build and test report on page Javadoc If the JAVADOC-MODE tag in the setup file is not set to off, Javadoc documentation will be generated for the packages that have been selected for build. This documentation is integrated with the rest of the report. 17

24 The daily build tool 5.3 User interface This section describes the user interface to the daily build tool and the HTML report files that are generated each build Setup file The behaviour of the daily build tool is controlled by a setup file, which is a simple text file. An example is given in figure 5. Among other things, you state the following information in this file: What to compile; What test suites to execute. Designers and the test team have separate test suite identifiers; Where to save HTML report files (see section 5.3.3); Whom to contact to report the result. For further information, check the user s guide [7]. # This is where HTML files are saved. HTML: /home/demo/public_html/ # The url to HTML directory. URL: # The location of build.spec directories. BUILDSPEC: /vobs/rnc/roam/em/rbshandling/service/lm/ BUILDSPEC: /vobs/rnc/roam/em/startrestart/service/lm/ # addresses to those who shall receive . TheOneResponsible@era.ericsson.se DailyBuildNewsgroup@era.ericsson.se # Telephone numbers to those who shall receive # SMS messages. SMS: # If off is stated, no Javadoc documentation # is generated. JAVADOC-MODE: on # The ClearCase label to compare built files to. COMPARE: ROAM_FISH_0.1 # If off is stated, test suites will not be executed. TEST-MODE: on # Test team test suite and configuration file # location pairs. T-MPFILE: /home/demo/mp/roam_rbs_handling_ft.mp T-CFGFILE: /home/demo/cfg/roam_rbs_handling_ft.cfg # Designer mp and cfg file location pairs. D-MPFILE: /home/demo/mp/start_restart.mp D-CFGFILE: /home/demo/cfg/start_restart.cfg D-MPFILE: /home/demo/mp/cell_config.mp D-CFGFILE: /home/demo/cfg/cell_config.cfg Figure 5. An example of a setup file. Some tags are not included. To make the script work, the tags HTML, URL and BUILDSPEC need to be stated in the setup file, but the rest can be ignored Tool invocation The daily build tool is invoked from a UNIX shell by calling > perl dailybuild.pl or since the system supports the #! syntax, which starts the first row in the script file and tells where to find the interpreter, > dailybuild.pl is enough. If default values are not wanted some arguments may be given, see [7]. An example is > dailybuild.pl -clean -d /tmp/db/ -cello /home/demo/cello/ -view daily_build_view 18

25 The daily build tool Build and test report The report consists of several separate files that shall be viewed using a web browser. Their structure is illustrated in figure 6 and their contents are described below. Old reports Jar files Main page Test main page Build main page Build details and file information Build.spec files Setup file Test suite information Figure 6. Structure of HTML report pages. Arrows represents main links, but all links are not displayed. 19

26 The daily build tool The main page presents general information and from this page all other sections can be reached. Among other things, the following information is presented here, which is shown in figure 7: 1. General result of the build; 2. General results of the tests; 3. Link to setup file, see figure 11; 4. Link to old reports, see figure 8; 5. Link to Javadoc documentation; 6. Statistics gathered from all builds and tests run so far. This is also where you would find a link to an enumeration of the jar files generated at compilation time, but this is not the case in figure 7 since jar build obviously failed. An example of that page is however shown in figure 9 on page } 6 Figure 7. Example of report main page. Compilation has succeeded, but the jar build failed for some reason which is accounted for in the log file. Errors have been found by the designer s test suites, but the test team s suites have been executed without any errors. Javadoc documentation has been generated. Old reports are all stored so that one can track any occurring events. From the main page you reach links to every report generated so far, as in figure 8. These lead to respective report main page. Old Javadoc documentation and old jar file enumerations are however not saved, but are discarded when a new report is generated. Figure 8. Directory listing of old reports directory. If the build and test script is executed several times on the same day, subdirectories in the date-labelled directory contain the separate reports. Otherwise a click on the date-labelled link leads directly to the main report page for that date. 20

27 The daily build tool Jar files generated from a successful build are accounted for in the report page as shown in figure 9. By listing the generated jar files, one can conclude what correct files were used for testing, as described below. Figure 9. Each jar file generated during the build is enumerated on a jar information page. The information dealing with the build is separated into several pages. This is shown in figure 10 where an example of build and file information is presented. Included features in the main build page are: 1. Statistics for this build; 2. Compilation time; 3. Link to build.spec files that tells what was selected for build, see figure 11; 4. Links to detailed description of all packages built; a b Figure 10. Build report pages. a) shows the main build page. b) shows a package description page. Each package have their own page where files are studied in detail. The information found there consists of: 5. ClearCase version of the compiled file; 6. Error warnings if there are any errors in a class, with link to error description; 7. The number of lines and comment lines in the file; 8. The change in number of lines compared to other ClearCase version 7 ; 9. Creation date and creator, with mailto-link; 10. Compilation error descriptions, if any (not shown in figure 10); 11. Links to Javadoc documentation. 7 Selected in the setup file by the COMPARE tag. This is probably the latest release label. 21

28 The daily build tool Figure 11. The build.spec files used are stored on the report page (image to the left) as well as the setup file used (to the right). This enables complete tracking of the latest and also old build and test reports. The test information is also split over several different pages. Just like in the build pages, there is a main page with general information. Each test suite then has a link to a separate test suite report page. This is illustrated in figure 12. Each test suite is executed even if the build fails. The result of this is that the jar files generated from the previous successful build is used instead of the ones that were not produced. The main test page will in this case contain a warning that states that some tests might be irrelevant. The jar file page shown in figure 9 tells what tests are not affected by compilation failure. The test feature can be entirely turned off in the setup file. In this case, there will of course not be any descriptions of test suites in the report. 22