Development of a Continuous Process Test to Increase the Software Code Quality of a Engine Control System

Transcription

1 Development of a Continuous Process Test to Increase the Software Code Quality of a Engine Control System Patama Ramchune The Sirindhorn International Thai-German Graduate School of Engineering (TGGS) King Mongkut s University of Technology North Bangkok (KMUTNB) Bangkok, Thailand patama.ram@gmail.com Sebastian Grobosch VEMAC GmbH & Co.KG Aachen, Germany grobosch@vemac.de Abstract Nowadays, embedded systems such as embedded control systems in vehicle are becoming more and more widely used. However, as the number of the embedded control system grows and as human put more and more responsibility on the systems, then how can people ensure that embedded systems are safe? In case of errors or failures in the embedded control systems, it could cause injury or loss of human life. So the embedded control system should be tested and the amount of errors or failures which is reported from the testing should be reduced. During the past few years, quality of an engine control system was increased continuously by many companies introduce using Continuous Integration (CI) in development process. The goal of using CI approach is continuously run the tests for whole systems from coding until build code in simulator in order to have a feedback and report of the result. The test can be done automatically and periodically. The result of the test will be reported to developer, so they can indicate the solution immediately. The principal objective of this paper is to demonstrate the implementation of using Continuous Integration (CI) for embedded systems of ECU(Engine Control Unit) product in order to improve the quality of the system for both coding and product usability. Keywords:- Continuous Integration, Code quality, Testing, Embedded System I. INTRODUCTION Within companies in automotive industry field, the products are very important since developer could determine the safety of the user. The quality of software should be tested and the amount of errors in software should be reduced. To avoid the bigger error and possible mistake in the future, the developer should involve integrating the test of software early and often. Performing test early and often can reduce rework and thus cut down cost and time. In this case, Continuous Integration (CI) can be applied in order to give the developer feedback about the quality of the software continuously. Our goal is to integrate the testing process into the development process of VeRa Project which is an embedded system for engine domain in order to have a feedback and report the result. The test can be done automatically and periodically by using CI approach. The repetitive tasks of usability testing can be done automatically by creating a bundle of command and executing periodically. Beside the code quality, we also focus on how to determine the measurement and what need to be reported to developer. In order to provide the reader with an excellent insight, the paper is structured as follows. We will first present in section 2 the system overview of our test structure. Section 3 describes the implementation of the test. Section 4 contains results of an implementation. Section 5 then presents the conclusion and future works for our system. II. SYSTEM OVERVIEW This paper analyese the use of Continuous Integration software called Jenkins. With various of plug in and large community, we can create command of the test process for both usability test and code test. Basically, the workflow of the Continuous Integration is checking-building-testingreport and back to checking again. In this way, we could schedule the building time everyday and acquire the test result periodically. Using Continuous Integration for the project can help developer catch problems and alert faster so that they could solve the problem. In this paper, the specific problems and requirements of the VeRa Project, which was developed by VEMAC Company, are analyzed. VeRa 3.0 is an ECU for prototype fleeds and small series. VeRa Desk is a user interface to compile the generated model code and to flash the code into hardware. The figure 1 shows the overview of VeRa project by VEMAC. Start from user creates MATLAB/Simulink model which is executed on the VeRa hardware. The code generating of models is done via the Simulink coder (Real-Time Workshop or RTW) or TargetLink. In order to create the feature models to access the I/O port of the hardware, a custom configuration of VeRa Simulink library is build. The library is a blockset of the inputs/output such as analog input, digital inputs, digital outputs, PWM output and additional blocks to configure the OSEK operating system such as OSEK task.

2 Matlab/Simulink Create model from VEMAC provided block VeRa Desk (User-Interface) Organization/Management of projects control the tool chain User Project Compiler/linker (Diab/WindRiver) VeRa Hardware Boot loader: VeRa BASE Fig. 1. MATLAB/Simulink Block (e.g. GPDI, OSEK-task) MATLAB/Simulink block generation xml-file VeRa OSEK Overview of work flow Manual create.mex(32 (.c,.h) and.tlc files Our goal is to find the error as fast as the problems occur by test the entire project continuously. Normally, many major software company have the separated test department. As VEMAC do not have additional department, testing is done separately by developer on their own unit not complete system. For this reason, testing environment for whole project is need to be set up as well as collect the create result and notice to the developer. The ideal method should be build up the test environment which developer can also do testing during the development process. The code we need to test is the user interface (VeRa Desk) which is written in C#, the bootloader (VeRa BASE) which is written in C and interface for Matlab/Simulink Models, provides OSEK operating system and hardware driver (VeRa OSEK) which is written in C. Moreover, usability of the project such as code building test, user interface installation in VM and the process of C-model code with the user interface also need to be tested. Software development requires a measurement mechanism for feedback and evaluation. Moreover, developer might need to have standard or tools that developer and the team could work. Then, select the suitable code metrics for measurement and follow the codinstandard are also required. For these reasons, we need to research on which code metrics or stardard and how much should satisfy and possible for the quality test. We can define the test process by create a test job in Jenkins, schedule it and make a report. We separate the test into 2 parts as shown in figure 2 which are usability test and code test. Start Start Jenkins and required environment Fig. 2. Usability test - Code build - Install user interface in VM - Build project in user interface Code test - Code metrics - Code Convention Overview of work flow Report and notification End III. IMPLEMENTATION CI service called Jenkins is used for automate the build process. For usability test, all of the test could work properly in the prepared environment in VMware for consistency. We automate the GUI action script for install and build process in order to automatically test installation and build code. Then get the output and set the result on defined criteria. Moreover, for code quality, the code metrics and code rules such as MISRA C and VEMAC s coding convention can define by using Sonar Tools. Then Sonar will perform the test and provided visualize the result in a single website. Code quality need to be tested by look inside the code metrics, coding style and rules. In addition, the code always updated by the developer, most updated code from SVN need to be tested that they are still working by test build in Eclipse. Then check if there are any problems of building VeRa Desk installer from the latest revision. After that, the new installer from the last test could install in the defined VM and be able to build the project properly. Moreover, the model code could be able to build properly in VeRa Desk. A. Continuous Integration Continuous Integration involves making change to software and then building and applying quality testing process.[4] Defects appear in the compile result, coding conventions, design, build scripts, the environment, server and so on. So testing for complete process and code is required for software quality. CI forces the defects to emerge early, rather than waiting for software to be fully produced.[5] Effective use of CI server could be better than working unplanned. CI server: There are several CI server, the most popular one is Jenkins. Jenkins can automate the building of software regularly and trigger test pulling in the results and falling based on defined criteria. Jenkins is an application that monitors executions of repeated jobs, such as building a software project or job, execute batch file or testing software. Testing environment: Building the test environment needs to set up the required program and environment for the test. VM is used in order to make a consistent test. Then VM environment also need to set up the required program before taking snapshot. B. Usability test The test deals with the usability of software. The idea is perform the action on user interface such as clicking or typing in order to check the availability of user interface and including code compiling via compiler or simulator. The Jenkins feature could use as the Windows command prompt. As there are many commands for the test, batch file was introduce to make a bundle of the commands which contain the process of test. Mostly, the batch file use for perform run VMware, process the test and get the result from VM respectively. Some processes cannot accomplish via command line, in this case auto GUI script is needed for automate mouse or keyboard action and record of the test result.

3 Currently, we test on the following features: 1) Test code build in Eclipse: In this test, the new version of code from SVN need to be tested in Eclipse. There are three codes that need to be tested which are BASE (Boot loader), OSEK (interface for Matlab/Simulink Models provides OSEK operating system, hardware driver) and Toolbox. The test perform by compile and test those code and check that the required file were created. 2) Test create VeRa Desk installer: After developer build and commit code to SVN, the next step is building VeRa Desk installer file. For create the installer file, the special scriptdriven installation system was used for create script to handle event of build the installer file. The test perform by build the installer via the installer script. In this step, we test that 3) Test install VeRa Desk in VM: After update code from SVN and create VeRa Desk installer file, the installation of VeRa Desk could be tested if it could install successful or not. However, it could be tested in various OS like Windows 7 and Windows XP by using the installer file from last section. In addition, we need to test that the current VeRa Desk could compile or build the VeRa Project properly. 4) Test build model code in VeRa Desk: The above test jobs are about VeRa Desk installation which is the compiler of VeRa Project. Nevertheless, VeRa Project which contain the model code from code generator (Simulink Coder or TargetLink) could to be tested in the VeRa compiler if need. VeRa Desk was used for manage the project tool chain. The test perform by checking that the model code could be able to build in VeRa Desk. For the test process, begin from start VeRa Desk and open the project. Then the GUI script will run to automate the test. As VeRa Desk does not support the command line feature, we have go create the GUI script in order to perform the build process. Figure 3 show the process of project build test. After that, create result and send back to Host PC via share folder. The test could also avialable for several VeRa Projects and run continuously by define the project run in Jenkins. Start testing new project n If finish all project y publish result Fig. 3. Start VeRa Desk and Open the project success send success message in output file Automate GUI test and create output file for the result of the test Waiting for the result Create result file and send back to Host PC fail send fail message in output file and also send back the error message from VeRa Desk console Overview GUI test for project build in VeRa Desk The entire test could be the same way. Figure 4 shows the sequence of usability test for each time. First we start job test in Jenkins by using batch file which contain command line for automate the test. All the GUI test are perform in VM in order to have the same environment in every test. After the test finish then retrive the test and store in xml file which can show the visualize report using XSLT and HTML form. Start Test Code build in Eclipse C. Code test Test create VeRa Desk installer Fig. 4. Test Install VeRa Desk in VM Overview of work flow Test Build project in VeRa Desk To make sure that the code is well running together with other, developer might need to have a standard or tools that developer and the team could work. We use the tool called Sonar for show the code metrics and code rules such as MISRA C and VEMAC s coding convention. 1) Code metrics: Software development requires a measurement mechanism for feedback and evaluation. To improve their software development, developer need a definition of clear improvement goals. Measurement on a software development process is performed by applying software metrics. For example, we use the following metric for inspect. End A source line of code LOC is the oldest, easiest and most basic metric that use to measure the size of a computer program.[1] LOC are used to create time and cost estimates. The LOC estimate becomes the baseline to measure the degree of work performed on a project such as the growth or progress of the project. A comment line Source Code understandability Metrics can be determine by using comment line. Comments are usually added with the purpose of making the source code easier to understand. The degree of commenting within the source code measures the care taken by the programmer to make the source code and algorithms understandable. Poorly commented code makes the maintenance phase of the software life cycle an extremely expensive adventure.[1] Comment density is assumed to be a good predictor of maintainability and survival of a software project. [6] Code duplicate Duplicate code is a computer programming term for a sequence of source code that occurs more than once, either within a program or across different programs owned or maintained by the same entity.[7] Duplicate code is dangerous and considered a mark of poor or lazy programming style. Good coding style is generally associated with code reuse or using function. Code duplicate frequently create long and repeated of

4 code. The long routines of code can make the difficulty of understanding. Cyclometric complexity Cyclometric Complexity defines how much complex does the computer program. Meaning that it defines the overall degree of hardness in working of coding program. The good code could be simple and readable. Less complexity could be apply for in the code quality because other developer will have to read and understand them in order to maintain or extend the code. The cyclomatic complexity number (CCN) is a measure of how many paths there are through a method. It serves as a rough measure of code complexity and as a count of the minimum number of test cases that are required to achieve full code-coverage of the method.[8] 2) MISRA C Rules: There are 143 rules for MISRA-C (2004), SONAR support about 39 useful rules.[2] Some of the rules cannot be checked by static analysis. For example, All usage of implementation defined behaviour shall be documented. (MISRA 3.1). However, the additional rule could be define by extend custom rules by Xpath. 3) Extended rules from VEMAC Coding convention using XPath: VEMAC provides coding convention to define the style, template and layout. Apart of the template, VEMAC defines the coding convention for the developer. For example, naming, indence, blangket or comment should be controlled. Sonar provides a quick and easy way to add new coding rules directly via the web interface for certain languages including C and C#. In this way, we can extend or define more coding rule in order to meet the requirement. Show as example on figure 5, the braces control can be detected and show in Sonar by using the following example of XPath. //listatement [@tokenvalue="{" and (@tokenline=//ifstatement/@tokenline)] //statement [@tokenvalue="{" and (@tokenline=//forstatement/@tokenline)] //statement [@tokenvalue="{" and (@tokenline=//whilestatement/@tokenline)] D. Experiment Setting Building the test environment needs to set up the required program and environment for the test. Moreover, Virtual Machine (VM) is used in order to make a consistent test. 1) Host PC and CI tools: Host PC is the machine that runs the tests. It requires Continuous Integration tools which are Jenkins (version 1.5) and Sonar (version 3.4.1). The CI tools needs to set up the test jobs in order to build the test processes. The particular job contains the steps of test process by using bundle of command line for excute the test steps. This may involve simply compiling the source code, running the batch file, measuring code coverage or code quality metrics or even generating document or notification. Moreover, we have to define the test schdule in order to have a continuous test. VMware also needs to be installed in order to set up a usability test environment. VMware provide the ability of using command line which suitable to use with Jenkins via batch file. In this way, we can run the jobs process which defined by Jenkin in VM environment. For code test, we have to configue the require component in Sonar such as update code version control, coding language or coding convention. 2) Virtual Machine: For usability test, we need to be confidence that the code could run in compiler and simulator properly. Virtual Machine can provide the consistent environment everytime we test. For this reason, the compiler and simulator need to be installed in Virtual Machine. E. Report result 1) Dashboard: Every test results are show in Jenkins dashboard. Developer can take a look on the overall from all system and also take a look deeper in each test if there are any problems. Moreover, sonar dashboard shows the code test informatioin which is defined code metrics such as line of code, number of file, duplicated line, some alert or the graph which show the growth of the project. Sonar also provided deeper detail by showing history of the code metrics and location of violation code. 2) XML, HTML, XSLT: There are additional files that especially created for report the result of GUI testing.after the test finish, batch file will generate output and append in form of XML. After that, convert XML file into HTML file for showing by using XSLT as a style sheet. XML was designed to transport and store data, with focus on what data is. In this way, we can use XML data for report or more advance query. Jenkins can display the reports on build job page and make them easily accessible to all team members. 3) notification: notification is the most obvious and common form of CI notification. Notification such as failed builds and some information can actively encourage developer to take matters into their work and fix broken builds more quickly. In this system, we also send some information about the test to the developer after the fail build. Fig. 5. Example of violation aleart in coding test IV. RESULT AND DISCUSSION We have run the test and collect the sample on weekday for one month (April 2013).

5 Usability test was test in VM for concurrency. The testing overview are show in Jenkins dashboard, from here, developer can take a look on the overall from all system and also take a look deeper in each test if there are any problems. We also provide the custom result from the GUI test in VM by using HTML form. Figure 6 shows the table of test result on code build for VeRa Desk. structure definition or decode syntax. During the test, there are some bugs found while running test by sonar runner, which show from output console in Jenkins. That could help developer get alert and find the solution. Since the testing processes are the same everytime, using CI could reduce the testing process for software developing. We will get the result as soon as the test can detect the error. To help the developer get the notification, alert need to be active. Jenkins will send the notification after the error found and also shows the information about the problem location or reason of failure. Jenkins also provides us the current status and the testing history. Figure 7 shows the job status indicated by the ball colour and the job static indicated by the weather icon. Fig. 6. Example of result table from code compiling test The usability of the system could be tested with automate GUI script. However, one of the problems about automate action for the test is concurrence of the script. Although the test enviroment was set consistency by using VM and snapshot but sometime the error occur because of the limit of PC resource. The entire result of code test are show in Sonar which contain the information of code metrics and code style. We also extend the code style by using Xpath in order to meet the VEMAC coding convension. Dashboard shows the default code metrics such as line of code, number of file, duplicated line, some alert or the graph which show the growth of the project by line of code and the metrics changes in each version. Moreover, using Sonar can lead developer to see the growth and history of the project by static analysis. Some of the source files cannot pass the code quality test with the specific function such as ASM. In this case, those source need to be skipped in order to continue the test. Because we skip some file, it cause the test did not cover all the source. Moreover, although there are many MISRA C rules (147 rules) but not all the rule could be test because some of the rules cannot be checked by static analysis. The result of code test at the initial phase, we found that the most violated rule are regarding to VEMAC coding convension. It could imply that the developer should consider more about their coding style. As VeRa Project was developed for several versions, it might possible that many files were build before the coding convension was defined and the files were never edited again. For this reason, code testing shows there are a lot of coding convension violation and also shows the bugs which according to the version compatible such as Fig. 7. Job status indicated by the ball and Job static indicated by the weather icon V. CONCLUSION AND FUTURE WORK From the concept in above section, we implement the test into development process. The test environment was applied in development process. The host machine was set up and configure the required program such as Jenkins for Continous Integration server, Sonar for centralize the range of code quality or Vmware for VM. All the tests were created from Jenkins and the steps of the test method were defined and set for each test. The steps of test method were defined by using batch file to create bundle of commands. The results can be found from dashboard as well as the error will be sent to the resposibility person in order to investigate and solve the problems. Using Continuous Integration in the project could help developer done the repetitive tasks on testing by admit the machine to do the tasks instad of human. Developer could get the rapid feedback in a simple visualization report. They could check that the whole system still working. Not only the usability of the system but also the coding inspection of each commit version. Developer could examine the part of code which could tend to be a problem. Moreover, they could see the static measurement growth of the project and archive too. Overall, we have imagined a further improvement for comprise error handling and other functionality with the following feature: The test could be extend the amount of test with another component such as PCLine or Arcade. The test sould perform only when the codes are change, this should be an aptimistic way for the machine resource.

6 Extend more coding rules which sonar could not cover by using Xpath. However, it need to have more research on which rule could be useful in order to improve the code quality. Of course, the collected measurements might be further research to improve the performance of the project. GUI test needed to be improved. Even the test already have the exeption and be able to detect some common mistake but there are more catagories of failure. So if the test could define more mistake, it could provide the right point of failure to the developer. VI. ACKNOWLEDGEMENT I would like to acknowledge the following people for their support and assistance with the research. From VEMAC GmbH Aachen, the special thank goes to my helpful advisors, Dipl.-Ing. Sebastian Grobosch. The advices and support that they gave truly helped the progression and smoothness of the research. The project during the program would be nothing without the enthusiasm and imagination from you. Next persons are my supervisor and my advisor from RWTH Aachen University, Prof. Dr.-Ing. Stefan Kowalewski. I am truly grateful for a chance and coordination between RWTH and KMUTNB-TGGS that made this master thesis program possible. The last but not least is Dr. Kamol Limtanyakul and Dr. Chayakorn Netramai, advisor from KMUTNB-TGGS who encouraged and guided me during the research period. REFERENCES [1] M Squared Techonologies. RSM Metrics Definitions. [Online]. Available: [2] Motor Industry Research Association. MISRA Development Guidelines for Vehicle Based Software. Nuneaton, November [3] D. Buchi and K. Baltzer. Continuous Integration for Embedded System. bbv Software Services AG, [4] A. M. Berg. Jenkins Coninuous Integration Cookbook. Packt Publishing Ltd., [5] M. Fowler. (2006, May). Continuous Integration. [Online]. Available: martinfowler.com/articles/continuousintegration.html [6] N.E. Fenton. Software Metrics: A Rigorous and practical Approach. PWS Publishing Company, [7] D. Spinellis. (2006, April). The Bad Code Spotters Guide - Repetition. [Online]. Available: [8] T. J. McCabe. A complexity measure. IEEE Transactions on Software Engineering, vol. SE-2, no.4, December [9] J. F. Smart. Jenkins: The Definitive Guide (creative common version). OReilly Media, [10] VeRa 3.0 The ECU construction kit. VEMAC Gmbh & Co.KG., [11] R. B. Grady. Practical Software Metrics for Project Management and Process Improvement. Prentice-Hall, Englewood Cliffs., [12] W. Humphrey. Introduction to the Personal Software Process. Addison- Wesley, [13] S. Brocklehurst and B. Littlewood. New ways to get accurate reliability measures. IEEE Software, vol. 9, pp. 3442, [14] A. Lindgren. MISRA C-Some key rules to make embedded systems safer. IAR Systems, 2011.