}w!"#$%&'()+,-./012345<ya

}w!"#$%&'()+,-./012345<ya M A S A R Y K O VA U N I V E R Z I TA FA K U LTA I N F O R M AT I K Y Effectively Integrating Unit Testing and Code Coverage into Continuous Integration D I P L O M A T H E S I S Tomáš Svrček Brno, spring 2015

Declaration Hereby I declare, that this paper is my original authorial work, which I have worked out by my own. All sources, references and literature used or excerpted during elaboration of this work are properly cited and listed in complete reference to the due source. Advisor: Ing. RNDr. Barbora Bühnová, Ph.D. ii

Acknowledgement I would like to thank my supervisor, Barbora Bühnová, for all valuable advice during the writing of this master thesis. I would like to thank Ronny Kolb as well for the patience, guidance and feedback during the whole process of implementation and writing part. I cannot forget to thank all my colleagues in ACT team for the great cooperation and willingness to help. Without these people the work would not be completed. iii

Abstract These days, automated testing is a fundamental part of the software development process. It helps the developers to avoid the basic mistakes immediately and therefore it saves the development time. Thanks to the rapid errors discovering, the quality of the developed product is increasing. This master thesis investigates how to effectively integrate Unit Testing and Code Coverage to the Continuous Integration process. The idea is analyzed in the State of the Art chapter and the current situation is inspected in the State of the Practice in Honeywell chapter. Based on the results of the analysis, the plugin for the Atlassian Bamboo server is designed and implemented. Evaluation of the developed product is done and its improvements are proposed for the future work. iv

Keywords Atlassian, Bamboo, Code Coverage, Continuous Integration, Unit Testing, xunit v

Contents 1 Introduction............................... 3 1.1 Context............................... 3 1.2 Goals................................ 4 1.3 Thesis Structure.......................... 4 2 State of the Art............................. 6 2.1 Software Development Process................. 6 2.1.1 Rational Unified Process................. 7 2.1.2 Other Approaches.................... 8 2.2 Capability Maturity Model Integration............. 9 2.3 Continuous Integration...................... 10 2.3.1 Continuous Integration Servers.............. 11 2.3.2 Summary......................... 12 2.4 Automated Software Testing................... 12 2.4.1 Reasons and Strategies for Test Automation..... 14 2.4.2 Unit Testing........................ 15 2.4.3 xunit Framework Testing................ 16 2.5 Code Coverage.......................... 17 2.6 Unit Testing in Continuous Integration............. 19 2.7 Summary.............................. 19 3 State of the Practice in Honeywell.................. 20 3.1 ACS Software Development Process.............. 20 3.2 Software Development Process in ACT............. 22 3.3 Continuous Integration in Honeywell............. 22 3.4 Bamboo............................... 23 3.5 Unit Testing............................ 24 3.6 Summary.............................. 25 4 Feasibility Study............................ 26 4.1 Overview.............................. 26 4.2 Bamboo Architecture....................... 26 4.2.1 Project Structure..................... 27 4.2.2 Agents and Server.................... 27 4.2.3 Capabilities and Requirements............. 29 1

4.3 Bamboo Extensibility....................... 30 4.4 xunit Frameworks......................... 31 4.5 Code Coverage Tools....................... 33 4.6 Tools Integration......................... 34 4.7 Summary.............................. 35 5 Implementation............................. 36 5.1 Architecture Design........................ 36 5.2 Bamboo Plugin Composition.................. 37 5.3 Used Technologies........................ 37 5.3.1 Java............................. 37 5.3.2 JavaScript......................... 37 5.3.3 XML, XSD and XSLT................... 38 5.3.4 Active Objects....................... 38 5.3.5 FreeMarker Template Language............ 38 5.4 Data Flow............................. 39 5.5 Analysis of Potential Problems revealed during the Implementation.............................. 41 5.6 Presentation Layer......................... 41 5.7 Summary.............................. 43 6 Evaluation................................ 45 6.1 First Round............................ 46 6.1.1 Evaluation Method.................... 46 6.1.2 Scenarios.......................... 46 6.1.3 Questionnaire Results.................. 47 6.1.4 Proposed Improvements................. 48 6.2 Second Round........................... 49 6.2.1 Evaluation Method.................... 49 6.3 Summary.............................. 50 7 Conclusion................................. 51 7.1 Future Work............................. 51 7.2 Obtained Benefits......................... 52 7.3 Summary.............................. 53 Appendices................................. 57 A Terms and Abbreviations....................... 58 B Package Diagram............................ 59 C Installation Instructions........................ 60 2

Chapter 1 Introduction 1.1 Context Honeywell is an international company with headquarters in the USA. It employs over 132,000 people all over the world. The company develops and manufactures a wide range of products from systems for residential homes and commercial buildings such as thermostats and security devices, through chemicals and materials up to turbochargers and equipment for airplanes. The company is divided into four strategic business groups: Aerospace Automation and Control Solutions (ACS) Performance Materials and Technologies (PMT) Transportation Systems (TS) This master thesis has been designed as improvement of the current state for the ACS Center of Technologies (ACT) team. ACT was founded to drive software excellence such as development of Best Practices, training, mentoring and improvement of the software development tools used in Automation and Control Solutions (ACS). For the purposes of software improvement, Honeywell ACS follows ACS Software Development Process (ASDP) which is compliant with Capability Maturity Model Integration (CMMI) as described in Section 3.1. 3

1. I N T R O D U C T I O N 1.2 Goals Bamboo from Atlassian is a continuous integration, deployment, and release management solution that is used by Honeywell ACS. The latest version of the Bamboo Continuous Integration (CI) server offers only a few plugins for xunit Testing and does not provide any Code Coverage functionality by default. Honeywell needs better support of mentioned tools and therefore this master thesis has been designed. The objective is to adjust Bamboo functionality for Honeywell users by implementing the plugin for this tool. The integration xunit framework family and Code Coverage analysis into Continuous Integration (CI) process is chosen to fulfill the assignment. The goal of this master thesis is to improve part of the development process and save the development cycle time. The result should be saving developers time during their activities. Based on the information from the State of the Art and also from the State of the Practice in Honeywell studies the extension for this Continuous Integration server is developed. 1.3 Thesis Structure This master thesis is further divided into the following six chapters. Chapter 2 State of the Art provides an overview of Unit Testing automation and Code Coverage as important parts of software development process. It also describes Continuous Integration process, tools and their usability. Chapter 3 State of the Practice in Honeywell describes how Unit Testing, Code Coverage and Continuous Integration are used in Honeywell ACS (Appendix A). Moreover, it discusses possibilities for improvement. Chapter 4 Feasibility Study discusses the possibility of the Unit Testing tools integration into the Bamboo CI (Appendix A) server. It describes Bamboo architecture, its extensibility and possible integration with testing and Code Coverage tools. Chapter 5 Implementation describes the implementation process. It contains a section about the architecture, code sharing and used technologies. The end of this chapter outlines the potential problems revealed during the implementation. 4

1. I N T R O D U C T I O N Chapter 6 Evaluation describes the methodology of thesis evaluation and summarizes gathered results. Measurements and findings are also provided. Chapter 7 Conclusion summarizes the results of this master thesis and its fulfillment. It also contains suggestions for the future improvements of the extension. 5

Chapter 2 State of the Art This chapter describes Continuous Integration and its importance in the software development process. It describes Unit Testing, Code Coverage and other parts of the Continuous Integration process. The chapter also mentions Automated Software Testing. 2.1 Software Development Process Software Development Process (SDP) also known as a Software Development Life-Cycle (SDLC) [1] is division of software development work into distinct phases. In other words it is a set of steps that a developed software program goes through. First mentioned in 1960, SDLC was the first framework for building information systems. This framework is meant to develop software in a structured and methodical way from the first birth of an idea to the deployment and support. There are more approaches to define the SDP (Appendix A). Figure 2.1 describes one of the possible approaches [2]. Figure 2.1: Phases of Software Development Process Plan represents planning of the whole project and obtaining users requirements. 6

2. S TAT E O F T H E A R T Analysis of the problem and design suggestions are the second step of the SDP. The output is usually expressed in the graphical notation such as the Unified Modeling Language (UML) [3]. Development is programming of the application. Tests for verification and validation should be executed to reveal the defects and therefore to increase the product quality. Deployment of the application to the customer s environment is done after the test results. Maintenance is the last phase of the SDP. The customer can find errors during the use of the system. The customer can also suggest improvements for the system. These activities are in all software development processes. Depending on the methodology of the software development, small differences in terms may arise. 2.1.1 Rational Unified Process The SDP tries to introduce the basic steps of the development to ensure the quality of the produced software. The company can use one of the certified processes to ensure the quality. Good example is Rational Unified Process (RUP) from Rational Software Corporation, a division of IBM. It is the software engineering process which gives a disciplined approach to distribute tasks, roles and responsibilities within a company. Its goal is to produce high-quality software. It is made to standardize and divide software development into four distinct phases [4]. Every phase is focusing mainly on one part of the process. Project overview and basic analysis are established in the Inception phase. The business context, expected revenue and also financial situation of the company are introduced. Potential risks during the construction phase, potential threats and time management are considered. The Elaboration phase provides the basic use-case model which should be fulfilled from at least 80% at the end of the project. The architecture is defined and prototypes are the outcomes of this part. This information helps to identify technical risks. The principle is to go forward to the vision. The vision has to be set wisely and must not change the main idea of the process. The current plan and architecture have to be set to fulfill the vision. 7

2. S TAT E O F T H E A R T Figure 2.2: RUP diagram [5] The main components and other features are developed during the construction phase. This part may consist of several iterations for better outcome. The end of this part is the first release of the developed product. The Transition phase does the switch ( transit ) from the development phase to the production. Training the users and beta testing of the application is provided. The comparison of the requirements from the inception phase is set during this phase. After the evaluation there is the final decision if the product has succeeded or not. The cycle can begin next iteration regardless of the result. 2.1.2 Other Approaches There are several models for such processes, each describing approaches to a variety of tasks. In the waterfall process, every phase is done only once. The next phase is started only after the successful end of the previous one. The customer is supposed to define the requirements before implementation of the product. In case of new ideas the customer is usually not able to change or improve functionality due to the fixed requirements. As the customer is mostly not a part of the implementation phase, she is not able to see the product until after the implementation phase. 8

2. S TAT E O F T H E A R T Commonly used models are iteration and spiral. Both approaches consist of the four phases as mentioned before. These four parts are iterating more times during the development process until customer s needs are fulfilled. Spiral development is a combination of waterfall approach and rapid prototyping methodology. It begins with concept of customer s requirements. The first prototype is designed based on the requirements. The requirements are improved by customer and the new prototype is designed again. The process is repeated until the customer is satisfied. After several rounds of prototypes the development begins. Nowadays a lot of companies use the agile [6] model of software development. It is based on a new different way of managing IT projects. Basic principles are adaptive planning and flexible response to change. Due to this way of thinking, projects are usually split into smaller pieces. The management has quick status of work done in shorter iterations. More information about agile in Honeywell is provided in Section 3.1. There are several possible approaches to do the software development. Every methodology has its pros and cons and depending on project capabilities the project manager has to choose the one that fits the project. 2.2 Capability Maturity Model Integration Capability Maturity Model Integration (CMMI) is a process improvement framework which helps to define and improve the development of the products. It involves the best practices covering the whole product life-cycle from conception through delivery to maintenance. CMMI (Appendix A) for development provides the concept of core components which can be extended by additional material to provide application-specific models which have common content. The maturity level of an organization (Figure 2.3) consists of related specific and generic practices that improve the organization s overall performance. Maturity levels are used to characterize organizational improvement relative to a set of process areas [7]. At Initial maturity level, the processes are usually ad hoc and chaotic. The organization environment is not stable to support processes. Success depends on the individual people in the projects instead of the following rules of the processes. 9

2. S TAT E O F T H E A R T Figure 2.3: CMMI Maturity Levels At Managed maturity level, the projects must execute the processes in accordance to policy. Projects contain plans and are also monitored, controlled, reviewed and evaluated for the future use. In other words management is able to control the milestones of the projects. This level helps to retain the plan during the times of stress. At Defined maturity level, the processes are well characterized and understood. Established organization s standard processes are improved over the time. The performance of the defined processes is managed. These standards are used to establish the consistency across the company. At Quantitatively Managed maturity level, quantitative objectives are based on the customer s needs, end users and organization. Processes and projects are based on measures and statistic terms. The performance of projects and selected sub-processes is controlled using statistical and other quantitative techniques. At Optimizing maturity level, the main goal is focused on continuous improvement process. The effects of deployed process improvements are measured using statistical and other quantitative techniques. This level is focused on improvement across all the projects in the organization. 2.3 Continuous Integration Continuous Integration (CI) is a software development practice to simplify the developers commit of source code into the repository followed by other steps such as static code analysis, Unit Testing, Code Coverage analysis, functional tests and others (Figure 2.4) to the one automatized unit. The 10

2. S TAT E O F T H E A R T basic idea is to add the new piece of code to the build process as soon as possible. After every commit to the repository the CI (Appendix A) process should be executed. The developers subsequently know possible errors and can fix them. CI process can take more time depending on its content. The vision is to automate as many procedures as possible. It should be as easy as clicking a button. Figure 2.4: Continuous Integration Cycle [8] 2.3.1 Continuous Integration Servers CI process provides automation of work which is achieved by CI servers. There are several solutions on the market and the following part discusses and compares three of them [9]. 11

2. S TAT E O F T H E A R T Jenkins CI server was forked from Hudson in 2011 and is developed as an open source project. Hudson focuses on improving stability and performance for the enterprise organizations. The main advantage of the Jenkins is the number of plugins and big community support. Development of new plugins is easy way how to extend this server. The negative side is its stability. Plugins have big differences in the implementation which leads to unexpected behavior of the system. Bamboo CI server is developed by Atlassian company. The biggest difference compared to Jenkins is its price. Bamboo is not an open-source software and provides reliable support especially on the portal named Atlassian Answers [10]. Extensibility of this tool is more complicated than Jenkins. The biggest advantage is the simplicity of integration with other Atlassian products. Bamboo provides also great integration with GIT revision control system. Moreover, this solution has remarkably better User Interface than Jenkins. TeamCity is the product of JetBrains and it is a commercial tool. Team- City offers integration with version control systems (VCS) though does not contain sufficient support. On the other hand Bamboo provides great branch and merge management for the version systems. TeamCity also offers sufficient integration with defect tracking tools. The use with Amazon Elastic Compute Cloud (EC2) is included as it is in Bamboo. TeamCity provides great integration with other JetBrains products. 2.3.2 Summary Each mentioned CI solution is capable to be used for the development. Bamboo is good for bigger teams because of integration and more complex jobs. Jenkins wins in the price aspect and ease of use. TeamCity provides the easiest workflow and project definitions. 2.4 Automated Software Testing Software testing is an essential and expensive activity in the Software Development Life Cycle (SDLC). Automated Software Testing (AST) is the application and implementation of software technology throughout the entire software testing life-cycle (STL) with the goal of improving STL efficiency and effectiveness [11]. Test automation provides the following major benefits. 12

2. S TAT E O F T H E A R T Team members do not have to spend time on inspections and reviews and can spend time on higher-priority testing tasks. Reduction in missed findings related to human errors. Allowance for measurement of testing coverage, i.e. identification of areas that are not covered by a test (almost impossible to find manually). Nowadays the developers are able to convert most of the types of manual tests into the automated tests. Automation of many recurring actions offers another potential source of saving company s money. Of course, it is necessary to consider whether all tests need to be automated. The used metric is called Return of Investment (ROI). If automated software testing is implemented correctly, in many cases it helps to save the time, lower investments (costs), and also helps to raise the quality of the development software products. AST is the integral part of the CI process and contains the following four levels. Unit Testing is the first part of testing and is described in more detail in Section 2.4.2. Since the Unit Testing checks the smallest parts of code such as classes, Integration Testing tests cooperation of bigger parts of the system called modules. The user can add individual applications to the modules integration and also is able to add client-server applications on a network. This type controls interactions between different parts of the system such as an operating system, file system and also hardware interfaces. Many modules are developed in the same time and they need to be integrated together. It falls into the black-box and also white-box testing. System Testing tests the scope and verifies all requirements from the customer. It also includes tests based on risks and/or requirement specifications, business processes, use-cases or other high-level descriptions of system behavior. It tests the interactions with the different parts of the system such as file system and hardware or interfaces between systems. The basic practice is to test interactions with the different operating systems and system resources. This type falls into the black-box testing. Acceptance Testing is the last part of testing process. The system is delivered to the customer for the Acceptance Testing. The customer verifies if the system meets all the requirements and if he/she accepts the final product. It falls into the black-box testing category. 13

2. S TAT E O F T H E A R T AST (Appendix A) encompasses many types of testing phases, including cross-platform compatibility, and is process-independent. Advantage of AST against the manual code testing is the possibility to avoid repetitions of scenarios many times by a tester. AST is a part of software development and does not replace the need for manual testers analytical skills, test strategy know-how, and understanding of testing techniques. Manual software testing and AST cannot be separated. Basically, they complement each other. Automation is the natural solution where a lot of personnel resources are saved and more errors are revealed. 2.4.1 Reasons and Strategies for Test Automation The main advantages of Test Automation are the following [11] Reduces the time and cost of software testing. Improves software quality. Enhances manual testing efforts via increased testing coverage. Does what manual testing can not accomplish, such as concurrent testing, performance testing, etc. There are many strategies how to approach test automation. Many companies use the white-box, black-box or optionally gray-box testing as a powerful method. White-box testing is one of the three well-known methods of testing software. Developers can see the source code of an application during this approach therefore they are able to predict outputs of their tests [12]. They can see if the program diverges from its intended goal. It is a well-known process for all manual testers. The big problem is that the testers do not test it sufficiently, as they are influenced by knowledge of the source code. Also the developer is only a human and can forget to test some situation. The most important knowledge is which line of the code is being executed and if line s result is correct or not. The advantage of this technique is the simplicity of the automation. White-box testing can be applied at the unit, integration and system testing levels of the program. This technique can include Code Coverage which includes branch coverage, statement coverage, decision coverage and modified condition/decision coverage criteria (more explained in Section 2.5). 14

2. S TAT E O F T H E A R T Black-box testing [13] is unlike the white-box testing based on the specification and requirements of the applications supposed functionality. Test cases are made in the same manner. The tester usually knows what result the program returns. On the other hand during black-box testing the tester does not have to know how the program is executed. Black-box testing can be applied to all levels of the software testing such as unit, integration, system and acceptance testing. Figure 2.5: Black-box Gray-box testing [13] is a combination of white-box testing and blackbox testing. The testers do not have the access to the source code. They have access to the documentation and they are familiar with the used algorithms. The testers usually have the specification and requirements of the application. Their task is to create feasible test cases from this data. Gray-box testing maintains boundary for testing between the tester and the developer. Findings from the statistics show that combination of black-box and white-box testing is the best solution. The problem of the software testing in general is money. In fact, 30-40% of the total costs are spent for the application testing. The other strategy how to test the product is horizontal vs. vertical testing [14]. Horizontal automation testing represents point to point flow where each test case is independent of the others and is following strict scenario. Vertical automation testing is executing test cases sensibly to save the execution time. The tester uses the state of the process and can execute more test cases at once. 2.4.2 Unit Testing Unit Testing is the first step of software testing. This part tests individual hardware or software units or groups of related units. In objected-oriented programming, a unit is a software component that cannot be subdivided into other components [15]. The developers write Unit tests concurrently with 15

2. S TAT E O F T H E A R T the source code to avoid the first wave of QA returns. Approximately 65% of all the bugs can be caught by Unit Testing. The tests should be done by the developers and falls into the white-box testing category (Section 2.4.1). It also helps to reduce the cost of bug fixes when the bugs are found in the earlier stage during the testing. As the name Unit Testing suggests, this part ensures that the blocks of the software work correctly independently from each other. Unit Testing might include Static Code Analysis which provides early defect detection. In general, it is used to ensure that code meets the architecture and behaves as intended. Benefits of Unit Testing are isolation of each part of the program and insurance of the individual parts as correct. Automation of the work is probably the most used benefit and so there is no need for group of testers who have to test the application manually. 2.4.3 xunit Framework Testing xunit is a common name for a set of Unit Testing frameworks such as JUnit, CppUnit, PHPUnit, etc. The unified name was chosen for the structure similarities of all frameworks. The generic name xunit [16] is originally created from SUnit from 1998 from Kent Beck and was known mainly in Smalltalk developers community. JUnit had arose on the flight from Zürich to Atlanta when Kent Beck traveled with Erich Gamma and introduced him his idea about xunit frameworks. The red-green progress bar with passed and failed results was the first time used in JUnit. This idea subsequently came back to the SUnit framework as an improvement. Then the idea had appeared in many others xunit frameworks. First from new wave was CppUnit where Michael Feathers found way how to use a core from JUnit. He developed a CppUnit which is the C++ port of the famous JUnit framework for Unit Testing. The developers worked with other languages made their ports of JUnit in a short time. In the meantime the JUnit became the standard of xunit family frameworks and was the essential idea in expansion of Extreme Programming (XP) and Test Driven Development (TDD). Unfortunately, JUnit is still not a part of JDK and therefore it is a third-party project. All tools of the xunit family frameworks include following four phases to clearly identify the intent of the test [17]: Set up test fixture/pre-test state (more in Section 4.4). Interaction with the system. Verification of the outcome. 16

2. S TAT E O F T H E A R T Tear down the test fixture. While the developers are able to run the same test many times in a row with xunit, the xunit framework is not particularly well-suited for automating performance tests [17]. 2.5 Code Coverage Code Coverage analysis is a technique that measures the percentage of system s code that is being tested by particular test suite while the automated tests are running. Code Coverage analysis is a powerful tool for finding more of the software bugs enabling a release of a software product of better quality. More technically, Code Coverage analysis reveals the areas in a program that are not covered by any test cases. Code Coverage tools are divided into two main categories. The first type adds the statements into the code and calls for recompilation of the code. The second type is the group of tools which instrument the code before running or during the execution. In this process a specialized tool is used to instrument the binaries to add the tracing calls and run a full set of automated tests against the instrumented product. During the process the data is stored and subsequently the report is generated. Good Code Coverage tools give to user the percentage of the code that is tested and also allow seeing exactly which lines of code were executed during the process. This part of the Continuous Integration process is expanding because software developers would like to know how many lines of the source code are covered by Unit tests. Usually, the result of this analysis is the ratio between the lines of code which are covered by Unit tests and all lines of the code. Code Coverage tools also provide other measures. There may also be information about function call coverage and branch coverage in the report. Code Coverage tests the code directly, therefore it is a white-box testing. The Code Coverage analysis compares the results of percentages against the given set of criteria. The main types of criteria are following [18]: Statement Coverage (also known as Line Coverage, Segment Coverage or Block Coverage) Function Coverage Function Call Coverage Condition Coverage (also known as Decision Coverage or Predicate Coverage) 17

Branch Coverage (also known as Path Coverage) 2. S TAT E O F T H E A R T Modified Condition Coverage (also known as Modified Decision Coverage) Statement Coverage reports the number of lines in the source code that have been executed. A variant of Statement Coverage is Block Coverage. This measurement is based on code blocks instead of statements. Basically, block is a set of statements. From a test quality point of view, there is a oneto-one relationship between Statement Coverage and Block Coverage. 100% success indicates that every line of code in an application has been tested. Function Coverage reports whether a function has been called or not. It does not say anything about execution inside the function or method. Neither it does not count how many function calls are executed. 100% covered indicates that every function/method in an application has been called by a test suite. Comparing to the Function Coverage Function Call Coverage measures the count of the available function calls. Every function and all its parts (i.e. if-statement) have to execute all branches in the code section at least once to achieve 100% Function Call Coverage. Condition Coverage inspects if each Boolean expression has been evaluated to both values: True and False. 100% success indicates that for each conditional statement, both outcomes have been tested. Branch Coverage is an advanced type of code coverage. It requires that all code blocks and all execution paths have been tested. This type adds more advanced requirements on top of statement or Block Coverage. For the if-statement, Branch Coverage is fulfilled when the code is executed at least two times. First time it is evaluated to the false and second time to the true. Only then all the code blocks and all the execution paths have been tested. Modified Condition Coverage is considered as advanced type of the Code Coverage analysis. In general, it is a combination of all types of Code Coverage. It requires that all code blocks and all execution paths have been tested. It adds the requirement that all the sub-expressions in the complex branch expressions (with more statements in if-condition) are considered as well. Basically, this means that the code section must be executed many times, until all the sub-expressions have been tested independently of the others. For example the block of code 18

2. S TAT E O F T H E A R T if ( (A B) && C) { result=0; a=evaluatea(b); } result++; should be evaluated 8 times because in if-condition there are three variables A, B and C. In many cases Branch Coverage and Statement Coverage have the same effect. For example, if the code only contained if-then-else statements, the metrics are equivalent. Testing all paths through code implicitly tests Statement Coverage and Function Coverage. Statement Coverage implicitly tests Function Coverage. 2.6 Unit Testing in Continuous Integration Automated Unit Testing and Code Coverage analysis are in most cases provided by the Continuous Integration solutions. They exist in the form of plugins for the CI tools. Jenkins (Section 2.3.1) provides many useful plugins for Unit Testing utilities for a various programming languages. It also offers code coverage tools for better control of Unit tests. The examples of Code Coverage tools are Cobertura[19] and Clover[20] which exist in Jenkins by default. The work and settings with these plugins are simple and the workflows are direct. Overall results look transparent and therefore they bring the inspiration for the new Bamboo extension. Jenkins also provides xunit Plugin by default which accepts formats as XML, TXT, CSV, etc. as input file and generates the results. Bamboo offers poor support for xunit frameworks in comparison to Jenkins. This may be due to smaller user base of this tool. Moreover, Bamboo does not provide any Code Coverage tools by default. 2.7 Summary This chapter provides introduction to the various tools, methods and related terms. It compares available solutions on the market and discusses their advantages and disadvantages. The Unit Testing and Code Coverage implemented as parts of Continuous Integration have a high importance in the development of high-quality software. They offer metrics and also reduce time of the development process. 19

Chapter 3 State of the Practice in Honeywell Honeywell ACS developed ACS Software Development Process (ASDP) to gain CMMI Level 5. The best practices for each of the software development disciplines have to be implemented to achieve this standard. This chapter describes ASDP and outlines current situation in Honeywell ACS teams regarding the use of Unit Testing, Code Coverage in Continuous Integration. 3.1 ACS Software Development Process The first version of ACS Software Development Process (ASDP) was released in January 2010 and current version is 3.6 published in September 2013. ASDP (Appendix A) aims at being CMMI Level 5 compliant software development (Section 2.2). ASDP defines an iterative and incremental lifecycle. ASDP is based on best practices from existing Honeywell ACS Procedures and Industry Standards as Rational Unified Process (RUP) and others. The content of the ASDP is organized into the two sets of the Disciplines [21]. Each discipline has its responsible person. The first type are the life-cycle disciplines called Core Disciplines. Requirements Architecture Design Implementation Test Deployment The second type is called Support Disciplines which are performed throughout the whole project life-cycle, from the inception phase to release. They help to manage and organize the whole process and to resolve unexpected accidents. Project Management Change Management 20

Quality Assurance Organizational Management 3. S TAT E O F T H E P R A C T I C E I N H O N E Y W E L L ASDP defines various roles, one for each activity. The process provides 43 roles by disciplines and 27 project roles (i.e. DB Designer, Project Manager, Security Architect, Tester, UI Designer, etc.). They perform a task and are responsible for a work product. Each discipline has to produce a comprehensible outcome. There are several categories of work products within the ASDP process. The two most important are Deliverables and Mandatory Work Products. Deliverables are the expected outcomes from the concrete activities. Typical outputs are the product (including source code), its parts and project and product documentation. Deliverables are those critical work products generated in a project that need to be verified. Predefined templates are used to unify the outputs of the reports. Mandatory Work Products (outputs) are all types of the reports. ASDP supports the reports by templates as it was mentioned before. Reports have to exist for evidence during process compliance audits. Finally there are work products that are optional in ASDP. These work products are related to the best practices. For example, a Static Code Analysis Report or Code Coverage Analysis Report would only be generated if a team chose to perform that practice. Measures and Indicators are needed to measure the success of the process. Typical examples of measurement are number of defects, effort, project duration, defect leakage or defect density. Honeywell s ASDP provides different points of view on the measurement. Measures aligned with the business goals are delivery on time and reducing cycle time. As ASDP aims at being CMMI level 5 compliant, its basis is a continuous improvement of itself, the projects and the company. It evolves the methodology from the past and recommends the best estimation in general. For this purpose ASDP contains the knowledge how to estimate expected delivery date, required resources and project complexity. Honeywell supports project management to improve processes in the organization. Measurement is important for comparing more projects in the company and therefore for the future improvement. There are four main reasons for measuring software and system processes, products, and resources. After the measurement it is able to better characterize the process and therefore establish the baseline for future comparisons. 21

3. S TAT E O F T H E P R A C T I C E I N H O N E Y W E L L The evaluation helps to recognize when the project is going out of the track and therefore brings it back under the control. The prediction is powerful ability for better planing. Gathered values from observation of modeled relationships can be used to predict other relationships between the processes. The prediction and the estimates based on historical data are used to analyze the risks. This helps to establish achievable goals for cost, schedule and quality. Future improvement is the global goal of all measures. ASDP uses measurement to identify inefficiencies, root causes and other places for improving product quality and process performance. As it was mentioned, the ASDP also contains Templates, Guidance and Tools suggested from disciplines and supported by Honeywell. UNITY Project Integration of used tools and Atlassian plugins sticks these two together under the one project called UNITY. The main idea is to remove data duplication and incompatibility across the Honeywell ACS and therefore there will be less human errors during the development process as well. 3.2 Software Development Process in ACT One of the most important decisions is the strategy to approach the problem. There are several known proven solutions such as agile software development or waterfall model (described in Section 2.1.2). ACT tries to use agile software development process with the SCRUM methodology. ACT usually practices strategy of 2 to 4-week sprints to achieve partial deliverables (explained in Section 3.1). All development approaches fall into the general iterative PDCA (Appendix A) process shown in Figure 3.1. There is fourphases similarity with the RUP process from Section 2.1.1. Recommended approach is agile software development with ASDP support. The strategy is set by Project Manager depending on the individual project. There is still used iterative waterfall development process a lot across the ACS teams. 3.3 Continuous Integration in Honeywell Continuous Integration is a current trend of software development and Honeywell aims towards this process as well. The main goal is to simplify the whole development process as much as possible. 22

3. S TAT E O F T H E P R A C T I C E I N H O N E Y W E L L Figure 3.1: Continual Improvement Honeywell ACS uses parts of CI and there is still a space for improvement. ACT (Appendix A) brings the integration of the particular tools to the CI process. Current situation of ACS shows that not all teams in Honeywell ACS are on the same level of using Continuous Integration. Nowadays, ACT team uses only Static Code Analysis and Unit Tests to speed-up bug findings and increase the quality. Current state of ACS Honeywell CI runs only the Static Code Analysis tool Klocwork during the nightly builds. Not all teams are able to build, test and deploy applications. A few of them are only in the previous stages of CI and obviously the ACS effort is to unify the work process and using CI effectively as much as possible. Future plans include more types of Unit Tests, Code Coverage analysis, Dynamic Code analysis, Automated Functional Tests and application Deployment to the Continuous Integration process. 3.4 Bamboo ASDP recommends various tools for most of the disciplines. Atlassian Bamboo is proposed as useful tool for the implementation within Continuous Integration. This tool is not used only for running builds and test. It connects issues, commits and test results and is able to deploy applications. It serves the results across the whole team from developers and testers to the project management. 23

3. S TAT E O F T H E P R A C T I C E I N H O N E Y W E L L Bamboo is one of the few tools for Continuous Integration. In general, most of the CI tools offer the same functionality. Honeywell uses Bamboo mainly for its smooth integration with the Atlassian Software Suite. Other benefits are comfort User Interface and easy extensibility via plugins with Atlassian SDK. Another useful feature is automatic branch management. When this function is enabled Bamboo creates the copy of current plan for every new branch in the project and everything is done automatically. 3.5 Unit Testing The big expansion of software testing during the past decades improves the quality of the developed software. ASDP leads to improve quality by several different approaches. Contrary to Integration Testing, which focuses on the possible interactions between two or more parts, Unit Testing only considers the separate parts in isolation. Unit Testing is a practice to examine the individual testing of source code units to determine if they behave as intended. At this time most of the Unit Testing frameworks used in ACS belong to the xunit test frameworks. The following list shows the most popular tools across the ACS. JUnit and TestNG for Java language. NUnit and Visual Studio plugins for.net languages (C#, VB). CppUnit for C++ language. Using these xunit frameworks gives way to unification the results from various Unit Testing tools. Integration of tools under UNITY project is partially conditioned by the use of xunit frameworks. Code Coverage analysis is closely related to the Unit Testing. There are only few attempts of the Code Coverage analysis tools such a dotcover for.net tested. Nowadays there is no Code Coverage analysis performed across the Honeywell ACS. Teams working with.net platform would like to use Visual Studio plugins. Real addition of the analysis to the CI process is plan for the future. 24

3. S TAT E O F T H E P R A C T I C E I N H O N E Y W E L L 3.6 Summary This chapter describes current state of processes and tools used within Honeywell ACS. ASDP, mentioned in Section 2.2, is a CMMI Level 5 compliant software development process. For this purpose Honeywell ACS uses Atlassian products, Bamboo especially for Continuous Integration process. As it is analyzed in this chapter, ACT team uses Unit Testing to increase quality of developed software but the Code Coverage is still missing. 25

Chapter 4 Feasibility Study The goal of this master thesis is to develop a plugin for Bamboo Continuous Integration server. For this purpose the initial analysis before the implementation is performed. The extension gathers the data from various Unit Testing frameworks and Code Coverage Analysis tools. Atlassian uses its own SDK to develop the plugins. This chapter provides an overview of the Bamboo server and its extensibility. Next parts contain basic information about xunit Frameworks, Code Coverage tools and their integration into Bamboo. 4.1 Overview ACT team has a lot of experience with the Atlassian products extensions. The essential requirement declares to simply run one or more tools in the build. This is done by design uniform input XML file for all user inputs (example in Listing 4.4). It is the only place for information from the user. All tools are usable via command line, so the user can easily write commands into input XML file. The basic assumption for the tools integration is similar structure of the XML output files produced by xunit testing frameworks. Code Coverage tools produce the similar output files as well. 4.2 Bamboo Architecture Analysis and understanding of Bamboo architecture is necessary for better integration. This section describes project structure, build possibilities and capabilities and requirements. 26

4. F E A S I B I L I T Y S T U D Y Figure 4.1: Bamboo Plan Architecture [22] 4.2.1 Project Structure Bamboo CI server contains projects which contain plans which are structured as depicted in Figure 4.1. Tasks are executed sequentially within a Job. Jobs are executed in parallel within a Stage. Stages are executed sequentially within a Plan. The project may contain more plans. Other CI servers can use different terms for the project structure parts. 4.2.2 Agents and Server Agent is a resource for Bamboo server to perform builds. The server is installed on the central machine. Bamboo is able to run jobs on different agents (machines) for parallelism. There are two types of agents. Local agents run on the same machine as Bamboo server and remote agents obviously on separated machines. Remote agents communicate with the server via the network. Bamboo uses more agents to increase the computing power and save the time. The user is not able to manage administration without logging in. The user should check if at least one agent is capable to run the job in case there are any job requirements. Administration section Agents provides view of all local and remote agents and their statuses. Agent can be either: idle (available to execute builds); building (currently executing a build); 27

cancelling (currently cancelling a Job build); disabled (not available to execute builds); 4. F E A S I B I L I T Y S T U D Y disabled-building (currently executing a build, but disabled so cannot execute further builds); disabled-cancelling (currently cancelling a build, and disabled so cannot execute further builds). Bamboo provides interesting function called Elastic Bamboo (Figure 4.2) which grants user access to computing resources from the Amazon Elastic Compute Cloud (EC2) to run builds. Amazon EC2 (Appendix A) is preferred for the companies and it changes the economics of computing by allowing a customer to pay only for capacity that he/she actually needs. The customer can rent the virtual machines from Amazon for building his/her builds for monthly fee. Using this feature is a big advantage for the companies as for every job requirement there is at least one agent capable to build it. Figure 4.2: Elastic Bamboo architecture [23] 28

4. F E A S I B I L I T Y S T U D Y 4.2.3 Capabilities and Requirements Every job on Bamboo has a possibility to set the requirements. Requirements basically define the set of the agents where the job can be performed. The job inherits the requirements of its tasks by default. Administration section of Bamboo contains a Server capabilities tab where there are displayed Matching Agents and Matching Images numbers that is constantly updated. Each agent provides executable capabilities which are external programs that Bamboo uses during the build process. Current version of Bamboo in ACS supports following executable capabilities: Ant (to build projects by Ant) Custom command (e.g. make ) devenv.com (to build projects from Visual Studio) Grails (to build Grails projects) Maven (to build Maven projects) msbuild.exe (to build.net/c# projects) NAnt (to build.net projects) PHPUnit (to build PHP projects) Script Agent can be either chosen by the administrator or Bamboo assigns the job to the random capable agent by itself. Capability is a feature of an agent and defines the path to an executable program. Capability can be shared by all local agents or all remote agents. Bamboo has its own predefined capabilities such as Maven, Mercurial or JDK. Job requirements offer only agents defined capabilities. Bamboo offers an option to add specific custom capability if there is missing one. Agent specific custom capability is defined by key and value. Key represents capability name in Bamboo and value contains the path to the specific capability. The user may setup Bamboo job requirements via input XML file (Figure 4.4). 29

4. F E A S I B I L I T Y S T U D Y 4.3 Bamboo Extensibility Bamboo CI server is extensible by plugins. ACT team has a lot of experience with extending Atlassian products. Atlassian SDK is designed to create extensions for Atlassian applications such as Bamboo, Confluence, JIRA and Stash. Bamboo plugin is compound from plugin modules. The essential part is a plugin descriptor atlassian-plugin.xml which contains the definition of all modules of the extension. There are four categories of module types. User Interface plugin modules Build Life-cycle plugin modules Notification plugin modules System plugin modules Special separate category represents Bamboo Event Listeners. Nowadays, there are 43 plugin modules available but for the purpose of this master thesis only the following few plugin modules are used [24]. Task plugin module (from Build Life-cycle category) represents a basic unit of work during the Build (Task in Figure 4.1). Every Job can have multiple Tasks which run in predefined order on the Agent. However, if only one Task fails, the following tasks will not be executed except for the Final Tasks. Tasks are designed to be easily extensible. The only need is to implement single Java interface to activate the Task. Moreover, there are optional extension points allowing user to add a configuration screen and other powerful features to the Task. With Web Item plugin module (from User Interface category) developer is able to add a link to some defined context on certain place on the web page. The link should have an ID and can be modified by CSS classes. XWork plugin module (from User Interface category) allows the user to react to the action. The module allows users to create their own XWork package and accessible actions. Developer usually needs to add a Web Item module to link XWork action with the system. Example of this plugin module is in Listing 4.1. 30

4. F E A S I B I L I T Y S T U D Y <xwork key= xworkresult name= CoveragexWork > <package name= resultxwork extends= buildview > <action name= xworkaction class= plugin.result > <result name= buildkey type= ftl >results.ftl</result> <result name= error type= ftl >err.ftl</result> <result name= fourohfour type= ftl >404.ftl</result> <result name= none type= ftl >none.ftl</result> </action> </package> </xwork> Listing 4.1: XWork plugin definition example The developer often needs to use external downloadable web resources during the plugin development. Web Resource plugin module (from User Interface category) allows developers to define them. CSS and JavaScript are usually also included web resources for the web design into the specific contexts of the application. With Component plugin module (from System category) the developer is able to share Java components between modules and optionally with other plugins running at the same time. The typical use is for singleton class used internally in the plugin. For the purpose of this thesis the pooling is used to save the time of the user. The biggest savings are during the view of builds history because the data is cached. 4.4 xunit Frameworks The frameworks from the xunit family (Section 2.4.3) generate XML file with a similar structure. The file always contains test suites, test cases and their results. ACT comes in with a requirement to develop a Bamboo plugin supporting these frameworks which simply and logically displays the data. The results from all used xunit tools are displayed in one tab under the build results. These frameworks are used especially in bigger companies where developers work on more projects written in the different programming languages. Using the frameworks provides the unification of the output XML file. All frameworks from the xunit family have the same following parts in common. 31

4. F E A S I B I L I T Y S T U D Y Figure 4.3: Core Parts of xunit framework Test Case is usually one test. The conventions prescribe to write one test method with only one assertion. This practice produces a lot of redundant tests or unwanted lines of code. On the other hand, many lines of code are better than hidden failures. It is hard to find which assertion had failed in multi-assertion methods without this practice. Test Suite is a set of Test Cases. It is possible to create multilevel Test Suites (means Test Suite of Test Suites) during the test development. Test Runner runs Test Suites, Test Cases, or multilevel Test Suites as all of these objects implement the same interface. Test Runner is reliable for the reports and results especially for the information how many tests had run or how many tests had failed the assertions. It also reports the number of tests that raised errors or exceptions. Another capability of Test Runner is simultaneous run of all the tests, automated by different people. Test Fixture (also known as a Test Context) is a set of preconditions for the running Test Cases. The developer is able to set the state of test environment for running tests. After the test, developer should return the program to the previous state without dependency on the other tests. Test Fixture is used for better identification of the errors (independent of the environment). 32

4. F E A S I B I L I T Y S T U D Y <report>... <method name= getalluserroles (): List > <coverage type= method, % value= 0% (0/1) /> <coverage type= block, % value= 22% (2/9) /> <coverage type= line, % value= 50% (1/2) /> </method>... </report> Listing 4.3: Emma output example 4.5 Code Coverage Tools Nowadays there are many tools for Code Coverage Analysis on the market. The most wide-spread solutions are free open-source tools Cobertura[19] and Emma[25] however they are not maintained anymore. Each Code Coverage tool generates XML file with the similar structure like it is in xunit frameworks case. XML file contains file coverage, package coverage, class coverage, method coverage and line coverage. Most of the results are expressed in percentage. The outputs of the tools in the Listings 4.2 and 4.3 contain basically the same data. The generic XML file is designed to gather information from various tools. The XSLT file is used for the transformation of the XML output files. The examples of the given tools outputs are following. <coverage>... <method name= getcar line rate= 1.0 branch rate= 1.0 > <lines> <line number= 35 hits= 2 branch= false /> <line number= 36 hits= 2 branch= false /> </lines> </method>... </coverage> Listing 4.2: Cobertura output example According to the task specification of the thesis it is preferable to use Emma which is ancestor of JaCoCo[26]. All of mentioned tools support command line execution, ready to use and even for free (more details in Table 4.1). Table 4.2 also includes non-free Atlassian s Clover tool, which is simply integrated with other Atlassian products. The advantage is various report types and also statement coverage metric. The assignment of this thesis is to find free and useful tools. Other paid tools are the part of suggestions for the future work. 33

4. F E A S I B I L I T Y S T U D Y Tool Commercial use Licence Cobertura free GNU General Public Licence Emma free Common Public Licence v1.0 JaCoCo free Eclipse Public Licence (EPL) NCover non-free End-User Licence Agreement (EULA) PHPUnit non-free Creative Commons Attribution 3.0 Atlassian Clover non-free proprietary CodeCover free Eclipse Public Licence (EPL) Pitest non-free Creative Commons Licence Table 4.1: Code Coverage tools licences Cobertura Emma/JaCoCo Atlassian Clover Coverage metrics Method no yes yes Statement no no yes Line yes yes yes MC/DC no no no Report types HTML yes yes yes PDF no yes yes XML yes yes yes JSON no no yes CSV no yes no Table 4.2: Code Coverage tools comparison 4.6 Tools Integration ACT team decided to design a solution with the XML input file shown in Listing 4.4. The user defines used tools, all necessary directories and requirements of the task if there are any. There is an option to define directories of test files on the local machine or on the remote machine. The input XML file is parsed and commands are executed. Used tools generate intermediate data placed into one result file which is parsed again and final test results are displayed in Bamboo. <input xsi:nonamespaceschemalocation= inputvalidator.xsd > <name>coberturatest</name> <execution> <type>command</type> <commands> <command> <basedir relative= true >JAVA\</basedir> <query>mvn cobertura:cobertura</query> </command> </commands> </execution> 34

<results> <result> <name>cobertura</name> <location>local</location> <format>xml</format> <filepath>result.xml</filepath> <parsing type= XSL >convert.xsl</parsing> </result> <result> <name>junit</name> <location>local</location> <format>xml</format> <filepath>result.xml</filepath> <parsing type= XSL >junit.xsl</parsing> </result> </results> <requirements> <requirement>executable.ant.ant1.4</requirement> <requirement>executable.msbuild.32</requirement> </requirements> </input> Listing 4.4: XML input file example 4. F E A S I B I L I T Y S T U D Y 4.7 Summary This chapter provides the basic information about Bamboo architecture, its extensibility, and tools integration. Essential idea is similar XML output file of the used tools. The goal of this master thesis is to develop simple extensible Bamboo plugin for the Honeywell ACS. It is easy to extend Bamboo CI server thanks to Atlassian SDK and its implemented modules. Analysis of current state of ACS environment implies that it is possible to fulfill customer needs. The development of Bamboo plugin for xunit testing frameworks and Code Coverage ensures time saving and better results summary. 35

Chapter 5 Implementation This chapter describes implementation and technologies used during the implementation process. The data flow in the plugin, potential problems and user interface issues are also discussed in following sections of this chapter. 5.1 Architecture Design Before the actual implementation of the integration, a number of architectural design decisions had to be made. Honeywell ACT team aims at integrating Continuous Integration process into the development work. The plugins for Bamboo are developed for this purpose. Implementation of this master thesis is closely interconnected with the thesis of Samuel Peťovský. Samuel writes about effective integration of static code analysis into the Continuous Integration process. Further analysis identifies related connections between these two theses. Due to this fact, ACT decided to implement the logic of the application into the common library. We named it Core. Another issue was the development of the Core as a Bamboo plugin or as a standard Maven JAR library. The Bamboo plugin is accessible via Bamboo component system and therefore there is bigger overhead. The resolution was implementation as the Maven library for simplicity of integration. The Core contains functions for plugin configuration, commands execution and result conversion and parsing. The Core contains parsers for XML files and classes for conversions to other formats such as JSON and plain text. Package diagram of the plugin and Core relations are in Appendix B. The database is accessed by the ORM framework called Active Objects [27]. Every access to the database must be in the transactions for the data persistence. Plugin also contains available REST API for the external communication of Core with other systems. 36

5. I M P L E M E N TAT I O N Unit tests and Code Coverage tools are either launched from command line or they are included in the build systems as Maven or Ant for Java..NET platform uses MSBuild for this purpose. Command line option is chosen because Bamboo provides interface supporting script execution (precisely execution method) and therefore Unit Test tools and Code Coverage tools may be executed. 5.2 Bamboo Plugin Composition Atlassian extension is the JAR file which contains XML plugin descriptor, source code and front-end files. The plugin descriptor atlassian-plugin.xml encloses labels and linkage to all necessary files. It includes plugin modules (mentioned in Section 4.3), external front-end files (.ftl,.js,.css) and core component injected from the Bamboo server. Attlasian API offers simple plugin establishment for the Bamboo plugin by only one command atlascreate-bamboo-plugin. During the plugin creation the artifact ID, group ID, version and package for the source code must be set. 5.3 Used Technologies The following section describes used technologies during the implementation process. 5.3.1 Java Java is a programming language and computing platform originally developed by James Gosling at Sun Microsystems and released in 1995. Java is fast, secure, reliable and especially cross-platform language which is almost in every device. Nowadays, there are more 9 million Java developers worldwide and the community is still growing. Some applications and websites do not work properly until Java is installed. This language is mostly used in client-server web applications. Atlassian also uses Java programming language in their products and provides a good support. 5.3.2 JavaScript JavaScript (JS) is a dynamic programming language used for special actions on the client side of the application. JavaScript[28] is widely used as it allows web pages to display dynamic content and transfers part of computational power to the client side. Another big advantage is using the JS as cheap tech- 37

5. I M P L E M E N TAT I O N nology because there is no license needed. JavaScript together with HTML and CSS are among the most important parts in the web development. Another reason to use JS is using external graphic libraries for visualization of results into charts. Developers in ACT Honeywell use HighCharts [29], JS library to model uniform charts. 5.3.3 XML, XSD and XSLT XML is well-known abbreviation for Extensible Markup Language. This metalanguage is used to define documents with a standard format. It is supported by various programming languages and can also serve as a small database. All the tools used for this project support this format without any problem. XSD (XML Schema Definition) is a recommendation of the World Wide Web Consortium (W3C) which specifies how to formally describe the elements in an XML document. This thesis uses the technology as the validation of the users input. It is easy and fast validation option which helps to avoid most of the incorrect inputs already in the beginning. XSLT means Extensible Stylesheet Language Transformations and is able to transform XML documents to other forms or formats. Due this fact, this transformation feature is used to create a generic XML file from various tools outputs. 5.3.4 Active Objects Active Objects is the Object Relational Mapping (ORM) layer in Atlassian s products which was designed to simplify the work with the database. Active Objects use dynamic interface proxies as entities to avoid writing beans and to work with database objects more comfortably. 5.3.5 FreeMarker Template Language FreeMarker Template Language (FTL)[30] is a free template engine, licensed under the Apache License. FreeMarker is a tool which generates the text output based on the templates. The developer prepares the data to display and FTL engine just fills the variables from Java program into the FTL templates. FTL is designed primarily for servlet-based applications following the Model View Controller (MVC) pattern. Due to this fact the designers and the programmers are often split to the separate groups of developers and 38

5. I M P L E M E N TAT I O N they will not touch the logic nor the code of the other group. If the designer and programmer is the same person, this pattern helps him to maintain the product. FTL is not an application for end-users but something that programmers can embed and use in their programs. 5.4 Data Flow Figure 5.1 shows the data flow of extension and user s communication with the used tools through the Bamboo. Following text describes each of the four steps in detail. Figure 5.1: Data Flow Figure 5.2: Task configuration 39

5. I M P L E M E N TAT I O N Input from the user is the first part of the data flow diagram in Figure 5.1. The user s input file has gone through several design changes. The result solution is available in Figure 5.2. Final proposal allows user to configure the plugin with three options: Use of predefined configuration. Upload of the XML file. Editing of current configuration. Predefined configuration reads the XML files from the particular directory and offers them to the user in drop-down menu. Example of the input XML file was shown in Figure 4.4. The validation of the XML input file is performed by the XML Schema Definition (XSD) file to avoid basic failures and therefore failed builds. There is a more user-friendly possibility to configure the plugin via Graphical User Interface (GUI). The object design and entities are created after the parsing of the input file. Every assignment to the Active Objects must be in the transaction for the data persistence. Active objects (Section 5.3.4) represent the Object- Relational Mapping (ORM) database while the entities are created. Trigger step (Unit Tests or Code Coverage analysis) is running when the build is started. Command execution usually produces the data from the used tool. Bamboo provides command execution by Windows PowerShell or by the command line. Designed solution uses the Windows PowerShell because it allows pipelining of the commands. The plugin supports two types of execution. Local machine is able to run commands from command line and remote machines via REST web-services. Results have to be transformed in third phase. Multiple tools generate multiple output files. Various formats of the output files are generalized by XSL Transformation. All JSON outputs are transformed to the XML files at first and after that all files are transformed to the generic form. Visualization takes the last step of the process. This part collects the data from the back-end and displays them on the user s screen with the help of FreeMarker Template Language (FTL) [30]. More details about visualization are provided in Section 5.6. 40

5. I M P L E M E N TAT I O N 5.5 Analysis of Potential Problems revealed during the Implementation Problems might occur when the used tool does not produce output of the analysis in the form of the XML file. The proposed solution does not support such tools. Another problem might be using a tool not supporting command line operations. In general, most of the tools from xunit frameworks supports command line by default therefore the threat is limited. In case the Unit Testing tools produce XML file output with structure different from the one from the xunit frameworks family, the transformation into generic the XML file may not be possible at the moment. The performance problem had been discovered during the testing part of the development. Parsing of a long XML document takes too much time. The investigation shows that the document with 6500 unit tests takes 7 minutes to get parsed. Solution was found quickly and the removal of the child after the reading in every loop solved the problem. Moreover it takes only two lines of the code. Such simple improvement of the code reduces parsing time from 7 minutes to 20 seconds. Node node = testsuites.item(i); node.getparentnode().removechild(node); Listing 5.1: Cutting the list 5.6 Presentation Layer The idea is to keep the same look-and-feel as the original Bamboo application for the consistency. Mock-ups for the purpose of this master thesis are created in Balsamiq Mockups tool. Mock-up prototype of an application is in Figure 5.3 and a few comments occurred from the users. None of them were critical. Mockup prototype displays all the data on one tab. Input data for xunit frameworks is from the test directory of the project and Code Coverage analysis takes the data from the source code directory. Due to this fact ACT decided to split the results view into two separate tabs called Unit Tests and Code Coverage and therefore it does not fit with the proposed mock-up in Figure 5.3. Unfortunately this occasionally happens in the business because unexpected findings are detected during further analysis or implementation. 41

5. I M P L E M E N TAT I O N Figure 5.3: Build Results Mock-up The screenshots of the real application are in Figures 5.4 and 5.5. The upper part contains the most important information, numeric results and charts and lower part contains the table with the package structure and its results distribution. The results can be filtered by test status and alternatively by used tools. The ACT department suggests the idea to put the results history to the plan information and therefore more space on the results screen will be available. Visual Flow Human Computer Interaction (HCI) teaches to use visual flow as a userfriendly property of the applications. Grid is a basic principle for screen composition and visual consistency (repetition of the signs used). Computer literacy is used in many applications because the users are trained to use the learned operations such as close the window by red cross in Windows operating systems. This behavior is caused by repeating the actions and objects used. These patterns are also used in extensions of Atlassian products in Honeywell ACS for the filtering as a clickable hand cursor. 42

5. I M P L E M E N TAT I O N Figure 5.4: Unit tests results The most important data on the screen should be seen first. The structure of the screen helps to navigate the user s view. The plugin follows the same rules. The upper part in Figure 5.4 contains the results of the analysis. The build history dashboard in Figure 5.5 is basically identical to the column chart mock-up proposal. As it is showed in Figure5.3 original design was quite different. The main difference is the view split into two separate tabs. 5.7 Summary This chapter introduces the architecture of the solution and summarizes the highlights during the implementation. In the end the presentation layer is mentioned as very important part of the development. The customer communicates through it with the application and therefore it should be userfriendly environment. After all, it is often the user interface what sells the applications in the business environment. 43

5. I M P L E M E N TAT I O N Figure 5.5: Build history 44

Chapter 6 Evaluation The chapter evaluates the usability of the plugin developed as a of this master thesis. The evaluation is conducted by Honeywell ACS. The users have to evaluate effects, advantages and disadvantages of the developed product. The proposal includes two rounds of evaluation. The first one takes place in the ACT team in Brno, the second one is prepared for other users across Honeywell ACS. The goal of the evaluation is getting the feedback from the real users. The plugin can be upgraded for the second round of evaluation based on the results from the first round. Figure 6.1: ASDP evaluation process There is the ASDP evaluation process in Figure 6.1 with four distinct phases used in evaluations. Both suggested rounds of evaluation satisfy presented four steps. 45

6. E VA L U AT I O N 6.1 First Round The first round of the evaluation collects data from ACT team (the first phase in Figure 6.1) in Brno because it maintains Atlassian extensions and uses Atlassian products on daily basis and therefore it is a suitable target group. The questionnaire (the second phase in Figure 6.1) contains seven questions. The first part is about using the tools and also about the scenarios. The next questions are about the visualization and suitability of the user interface. The last question focuses on the opinion about the efficiency of the work performed. Five evaluators participated in the first round. 6.1.1 Evaluation Method This plugin is supposed to decrease the work time with the Unit Tests results and Code Coverage analysis results and increase the code quality due to expected higher use of Unit Testing and Code Coverage with the use of created plugin. The major benefits of developed plugin are tested in this evaluation and are the meaningful goals of this thesis. For this purpose the evaluators obtain the tasks and scenarios to fulfill. They can summarize the findings and comments in the additional questionnaire. The next subsections describe the detailed information about the tasks, scenarios and questionnaires. 6.1.2 Scenarios The scenarios are designed to simulate the real use-cases during the software development process. Navigation of the user through each implemented functionality is the intended goal. The outcome is the comparison of the results obtained with the extension and without it. There are four scenarios for the evaluators prepared. The user goes through the whole process of adding and configuring Unit Tests Automation task from the predefined XML configuration in the first scenario. Predefined in this case means that validated XML file is in the particular directory. The second scenario is focused to get detailed results of the certain failed test. It navigates the user through the visualization features of the plugin. The users test the intuitiveness of a development product and they can use the filters to save the time to achieve the goal. 46

6. E VA L U AT I O N The user is navigated to the build history screen and evaluates the visualization in the third scenario. The task is to read certain data from the screen. The main point in this case is the time for getting the results. The last scenario is very similar to the third one. It is also reading result data from the trend history, this time from the Code Coverage analysis. 6.1.3 Questionnaire Results The questionnaire is the last part of the evaluation of the features. It gathers the findings and insights (the third phase in Figure 6.1) during the work with the Bamboo extension. The evaluators can classify the features as the benefits or as drawbacks. The subjects obtain the questionnaire with the seven questions prepared. Question 1: Do you use more Unit Test tools (JUnit, NUnit, CppUnit, etc.) in one project? This question analyzes the current use of more tools by developers. None of the evaluators uses more xunit frameworks in one project. Question 2: Are all scenarios clear and easy to accomplish? Did you have any problem to find anything? If so, what was that? The question works as the analysis of user s visual flow. None of the evaluators had any problem to find the objects. They responded that the scenarios were intuitive. Question 3: Did you get the information you were looking for? The respondents claim to have obtained all information they were looking for. Question 4: Do you miss anything on the results view screen? From the previous question they do not miss anything. Question 5: Do you have any proposals for the improvements of the plugin? There are several interesting ideas for the improvements. Only a few of them are applicable in the near future. All others are the proposals for the future work (details in Section 7.1). The respondents note a few cosmetic issues which can be fixed easily. Nobody was able to figure out what a strange icon next to the Tests title was (Figure 5.4). It is the typical example of a non-intuitive icon. 47

6. E VA L U AT I O N One respondent would like to go one more level deeper in a Code Coverage hierarchy. The evaluator misses the coverage of each method on the screen. Next opinion was that it seems to be logically incorrect to put linkage from actual build results screen to the build history. The argumentation was that the information belongs to the plan, not to the actual build. Information about the final result of Unit Tests (passed or failed) could be visible in the plan details. Great suggestion with the trend issue is to view comparison from the last build on the result screen. WEB API for the third parties software and optional linkage with other tools would be a great improvement for the future work. Question 6: Would you use the plugin for your project in the future? All respondents agreed and marked the plugin as helpful and time saving. Question 7: Do you think that the plugin saves the view-time of results? And so save the time of the development process? This question regards the usability and efficiency of the developed extension. All responses are supported by statements that nowadays this kind of statistics are not even available in Bamboo. Another observation is having result directly in the Bamboo and therefore the user does not have to look for the results on the file-system. 6.1.4 Proposed Improvements The question number 5 from the questionnaire contains comments of two categories: Visualization and Functionality (the fourth phase in Figure 6.1). Visualization There are a few User Interface (UI) issues from the users feedback. The users identify the following notes as UI inconveniences. They suggest the idea that important and relevant information should be more visible on the screen. The suggestion is to place the method name on the top of the test-case details. Interesting design proposal was put forward on the build history screen. The user would appreciate the change from column chart of Code Coverage build history to the line chart for better visualization of the 48

6. E VA L U AT I O N progress from the last build. When the users got stuck during the process they were not able to continue the action and so they would appreciate more helpful information in the tooltips or help buttons. They propose to change the position of unimportant test-case information to different place. Functionality There are a few functionality recommendations in the results. Most of them are mentioned before in the answer to the fifth question. The findings do not discover any fatal issues with the plugin. There is a little performance issue during the large test suite but the problem was solved shortly after the detection. The evaluators described the task configuration as complicated. The most of the visualization issues were fixed before the second round of evaluation and respondents like the solution. Functionality issues must be consulted with the responsible authority, approved and suggested as future improvements for the plugin. 6.2 Second Round The second round of evaluation will begin after the fixes of all issues found. The Bamboo extension is supposed to be used across the whole Honeywell ACS therefore the evaluation will be performed with several different teams. The second round of the evaluation will be finished with deployment to the Quality Assurance environment. This round is not part of this thesis. The results are supposed to present the benefits of this plugin for the Honeywell ACS developers. 6.2.1 Evaluation Method There are improvements for the second round of evaluation learned from the first round. The instructions for the evaluators should be written in a more detail and with more steps. Other issue of this round is the set of evaluators. The first round was performed in the ACT team and the plan for the second round is to expand the base of the evaluators. After the second round, the tests in the Quality Assurance environment will be performed. The experience of the evaluators has to be considered during the scenarios formulation. Nowadays there is not so much data prepared, therefore the 49

6. E VA L U AT I O N results will not be objective. More test data for Bamboo is required. The second round of the evaluation will follow a different plan. The scenarios dealing with the task configuration have a too complicated settings and so the scenarios will be changed. 6.3 Summary The evaluation provides the important data about the functionality and the User Interface. It reveals to which extent the environment is user-friendly and detects the imperfections during the work with the implemented features. The evaluators appreciate the results statistics and time saving during their development activities. The linkage with the other Atlassian products is a big advantage for the users. The issues are mainly in the visualization and are described in this chapter. All the evaluation results and suggestions are definitely considered as activities for the future work (in more detail in Section 7.1). 50

Chapter 7 Conclusion The summary of all work done and its benefits for Honeywell ACS team are included in this chapter. Moreover conclusion contains the comments about the deliverables and summary of goal achievement. Unit Testing is closely related with the Code Coverage analysis and they are introduced as an important part of software development process. They help to increase the code quality and also considerably save the development time. This master thesis discusses the possibilities of effective integration of Unit Testing and Code Coverage tools into the Continuous Integration process for the Honeywell ACS. All terms and methods are introduced in the State of the Art chapter and in the State of the Practice in Honeywell the possibilities are reduced because of Honeywell ACS environment. The Feasibility Study chapter analyzes the possibilities of integration, available practices and solution. Implementation describes the plugin architecture, interesting parts and problems during the activity. Obtained results to given scenarios and proposed features are summarized in the Evaluation chapter. 7.1 Future Work Further improvement of the application is standard way of software development. As many others this extension has also a potential for improvements. The most of the suggestions are from the users which evaluate the plugin in the Evaluation chapter. There are various findings from the users. The most suggested are the functional features. Simple upgrades and visualization improvements were performed and the rest of the proposals is assigned to the following part. 51

7. C O N C L U S I O N Visual Enhancements There is a variety of visual details for change. Users suggest to change test information composition. The duration of each test is not so important therefore place it to another part of the screen. Details such as Show more button (pagination) to the all tests view should be provided. Another requirement from the users is adding help explanations. User who has never used Bamboo before would prefer better explanations in the tooltips. Another feature for further implementation is the result of all the unit tests shown already on the plan summary page. This can make the Unit Testing results more noticeable. Functional Extensions From functional point of view more important are suggestions to extend intended behavior. Users suggest to provide more information about the tested class and expand the filters to be more variable (to use more combinations of filtering). Another feature for further implementation could be wallboard-view and results could be reachable also from the outside of the Bamboo. For this purpose WEB API for the third parties should be implemented. All of the functional extension intent to save the user s time. 7.2 Obtained Benefits The main intention of the thesis is using more Unit Testing frameworks simultaneously for one build, if necessary. The new Bamboo plugin helps the developers primarily with the bigger projects. Afterwards the developers are able to view Code Coverage of Unit Tests on the second tab. The whole process saves the developers time because nowadays they needs to look for the particular information on the file-system (depending on tool used). Saving the time implies the increase of the software quality because the developers can spend more time on their activities instead of the overhead of finding information. The next important aspect for Honeywell is Continuous Improvement of Continuous Integration across the ACS teams. Bamboo extension contributes to the automation of the software development process and therefore using Continuous Integration process. 52

7. C O N C L U S I O N 7.3 Summary The thesis shows the possibility of integrating Unit Testing and Code Coverage into the Continuous Integration process. It should save the time and bring the developers more comfort into their activities and therefore raise the quality of the software produced. Using the recommended tools, which provide command line operations and generate XML outputs, will produce the results more effectively after this integration and therefore Honeywell ACS is closer to achieve fully implemented Continuous Integration process. 53

Bibliography [1] RADACK, Shirley. The System Development Life Cycle (SDLC). [online]. [cit. 2015-03-30]. URL: http://csrc.nist.gov/publications/ nistbul/april2009_system-development-life-cycle.pdf [2] DALBEY, John. Software Process Models. [online]. [cit. 2015-04-15]. URL: http://users.csc.calpoly.edu/ jdalbey/308/ Lectures/SoftwareProcessModels.html [3] OBJECT MANAGEMENT GROUP (OMG). Unified Modeling Language [online]. [cit. 2015-03-30]. URL: http://www.uml.org/ [4] AMBLER, Scott. Agile Modeling and the RUP. AMBYSOFT INC. Agile Modeling [online]. 2014 [cit. 2015-03-15]. URL: http://www. agilemodeling.com/essays/agilemodelingrup.htm [5] Viaprofessionalism. RUP picture [online]. [cit. 2015-03-05]. URL: http://viaprofessionalism.blogspot.cz/2012/09/ lecture-26-software-development.html [6] AgileManifesto [online]. 2001 [cit. 2015-03-06]. URL: http:// agilemanifesto.org/principles.html [7] CHRISSIS, Mary Beth, Mike KONRAD a Sandy SHRUM. CMMI for development: guidelines for process integration and product improvement. 3rd ed. Upper Saddle River, NJ: Addison-Wesley, c2011, xxiv, 26 p. ISBN 9780321711502. [8] Techrains: information solutions. TECHRAINS. [online]. [cit. 2015-03- 04]. URL: http://www.techrains.com/ci.html [9] PULSIPHER, Darren. Yoly Software Engineering [online]. 2014 [cit. 2015-03-17]. URL: http://yolyeng.blogspot.com/2014/02/ comparisons-of-ci-build-systems.html [10] Atlassian answers [online]. [cit. 2015-03-05]. URL: https://answers. atlassian.com/ 54

7. C O N C L U S I O N [11] DUSTIN, Elfriede, Thom GARRETT a Bernie GAUF. Implementing automated software testing: how to save time and lower costs while raising quality. Upper Saddle River, NJ: Addison-Wesley, c2009, xxv, p.23. ISBN 9780321580511. [12] KHAN, Mohd. Ehmer. Different Approaches to White Box Testing Technique for Finding Errors. [online]. 2011, no. 3 [cit. 2015-03-16]. URL: http://www.sersc.org/journals/ijseia/vol5_ no3_2011/1.pdf [13] SAMURIN, Alex. Explore the World of Gray Box Testing?. [online]. 2009 [cit. 2015-03-17]. URL: http://extremesoftwaretesting. com/articles/worldofgrayboxtesting.html [14] JAIN, Naresh. Test Automation Strategies For Agile. Slideshare [online]. 2015 [cit. 2015-03-17]. URL: http://www.slideshare.net/ nashjain/test-automation-strategies-for-agile [15] WILLIAMS, Laurie. White-Box Testing. [online]. 2006 [cit. 2015-03-18]. URL: http://www.chaudhary.org/whitebox.pdf [16] FOWLER, Martin. Xunit. [online]. 2006 [cit. 2015-03-17]. URL: http: //www.martinfowler.com/bliki/xunit.html [17] MESZAROS, Gerard. XUnit test patterns: refactoring test code. Upper Saddle River: Addison-Wesley, c2007, lx, 883 p. Addison-wesley signature series. ISBN 9780131495050. [18] Types of code coverage analysis. ATOLLIC. [online]. [cit. 2015-03-04]. URL: http://www.atollic.com/index.php/trueanalyzer/ types-of-code-coverage-analysis [19] Cobertura tool for code coverage [online]. [cit. 2015-03-04]. URL: http: //cobertura.github.io/cobertura/ [20] Java Code Coverage Atlassian Clover. Atlassian [online]. 2015 [cit. 2015-03-17]. URL: https://www.atlassian.com/software/ clover/overview [21] ACT. ACS Software Development Process: Version 3.6. [online]. 2013 [cit. 2015-03-18]. URL: http://acsprocesses.honeywell. com/common/sw/#acs.common.processes.sw.asdp.process/ customcategories/introduction_dd072101.html (internal) 55

7. C O N C L U S I O N [22] Understanding the Bamboo CI Server. [online]. [cit. 2015-03-04]. URL: https://confluence.atlassian.com/display/bamboo/ Understanding+the+Bamboo+CI+Server [23] Atlassian. [online]. [cit. 2015-03-05]. URL: https://confluence. atlassian.com/download/attachments/289277118/ Elastic%20Bamboo%20Conceptual%20Overview.png? version=1&modificationdate=1305190168642&api=v2 [24] Bamboo Developer Documentation. [online]. [cit. 2015-03-04]. URL: https://developer.atlassian.com/display/bamboodev/ Bamboo+Developer+Documentation [25] EMMA tool for code coverage [online]. [cit. 2015-03-04]. URL: http: //emma.sourceforge.net/ [26] JaCoCo tool for code coverage [online]. [cit. 2015-03-04]. URL: http: //www.eclemma.org/jacoco/ [27] Active Objects. ATLASSIAN. [online]. [cit. 2015-03- 18]. URL: https://developer.atlassian.com/docs/ atlassian-platform-common-components/active-objects [28] JSON [online]. [cit. 2015-03-08]. URL: http://www.json.org/ [29] Highcharts javascript library [online]. 2015 [cit. 2015-03-08]. URL: http://www.highcharts.com/ [30] FreeMarker template language [online]. 2015 [cit. 2015-03-04]. URL: http://freemarker.org/ [31] Nunit testing tool [online]. 2002 [cit. 2015-03-08]. URL: http://www. nunit.org/ [32] GSON Java library [online]. [cit. 2015-03-08]. URL: https://code. google.com/p/google-gson/ [33] jquery [online]. [cit.2015-03-02]. URL: http://jquery.com/ [34] FOWLER, Martin. Continuous Integration [online]. 2000, 2006 [cit. 2015-03-04]. URL: http://www.martinfowler.com/articles/ continuousintegration.html 56

Appendices 57

Appendix A Terms and Abbreviations Term ACS ACT ASDP AST CI CMMI EC2 FTL HCI NPI ORM PDCA RUP QA SDP STL UML UI VCS W3C Definition Automation and Control Solutions ACS Center of Technologies ACS Software Development Process Automated Software Testing Continuous Integration Capability Maturity Model Integration Elastic Compute Cloud FreeMarker Template Language Human Computer Interaction New Product Introduction Object-Relational Mapping Plan Do Check Act Rational Unified Process Quality assurance Software Development Process Software Testing Life-cycle Unified Modeling Language User Interface Version Control Systems World Wide Web Consortium Table A.1: Terms and abbreviations 58

Appendix B Package Diagram Figure B.1: Package diagram 59

Appendix C Installation Instructions Following installation instructions explain how to install Unit Testing and Code Coverage plugin for the Bamboo CI server. Prerequisites Bamboo version at least 5.2 running on the machine Administration account on the Bamboo server Access to the Bamboo machine Compiled plugin as a JAR file XSLT files for the results transformation Configuration template files (optional) Pre-installation Steps 1. Create results folder at the location given in the directories.properties file in JAR (default C:\results\UnitTests&CodeCoverage) 2. Create folder for predefined configuration files and put the files into the folder (default C:\UserXml\input) 3. Create folder for XSLT files and put the files into the folder (default C:\UserXml\xslt) 60

C. I N S TA L L AT I O N I N S T R U C T I O N S Plugin Installation 1. Open the main Bamboo page in the browser 2. Log in as an administrator 3. Navigate to Bamboo Administration 4. In Section ADD-ONS select Manage add-ons 5. Select Upload add-on 6. Select the plugin JAR file and select Upload 61