Unit Testing improves Software Quality Unit Testing and the Classification Tree Method

Transcription

1 Building a safe and secure embedded world Unit Testing improves Software Quality Unit Testing and the Classification Tree Method > White Paper Frank Büchner

2 Preface This White Paper is intended to give an overview on unit testing and the classification tree method. The documentation contains supplementary information, detailed notes, special examples, etc. For more information about current products and product versions or to access our update service, go to or The first part of this White Paper offers some general thoughts about unit testing. A simple example using the Tessy tool then shows how this might look in practice. The second part describes the classification tree method, which can be used to systematically identify test case specifications for unit testing. 2

3 Contents 1 Unit testing Introduction Module testing and unit testing in standards Modules vs. units What is Unit testing? When does unit testing make sense? Who should conduct unit testing? How is unit testing performed? Automatation is indispensible Example with Tessy The test unit The unit interface Initial test cases Conducting the test Troubleshooting and regression testing 10 2 The classification method Purpose Characteristics of the classification method Applying the classification method Identifying test-relevant aspects Classification Repeating to obtain a classification tree Specifying test cases in the combination table Example Tools to support the classification tree method 13 3 Summary 14 4 Literature 14 3

4 1 Unit testing 1.1 Introduction Module testing and unit testing in standards International standards for developing security-critical (embedded) software generally require module testing or unit testing (and software integration testing). Technical standards such as the cross-industry IEC call for module testing. Module testing is designed to show that modules provide the intended functionality and no unintended functionality. The newest edition of the standard (from 2010) recommends code coverage measures that must be measured independently of the Safety Integrity Level (SIL). The ISO standard (from 2011) requires unit testing. This standard also lists methods that should be used to identify test cases for unit testing and recommends identifying code coverage measures found in IEC The process-oriented standard Automotive SPICE uses the term software units and requires unit testing as part of their verification. In IEC 62304, which describes the software life cycle of medical device software, process step 5.5. calls for Software Unit Implementation and Verification. The industry-specific standard DIN EN (Railway applications) refers to the lowest software components as modules and requires them to be tested. ISO (Safety of machinery) also prescribes module testing in its V-model. RTCA/DO 178 (avionics) uses testing of low-level requirements and testing of software components that are verified using the requirements. Depending on the potential impact of errors in the software, DO-178 requires measurement of code coverage measures. All standards put a high value on test planning, test documentation and the option of repeating previous (successful) tests (regression testing). This also includes unit tests and module tests Modules vs. units A unit (as a test object in unit testing) is usually the smallest unit that can be sensibly tested in isolation from the rest of the application. This is consistent with ISO 26262, which defines a unit as an atomic level software component that can be subjected to stand-alone testing. For software written in C, a unit is therefore equivalent to a function as defined in C. IEC defines a software module as a construct that consists of procedures and/or data declarations. A module of this kind can also be understood as a function as defined in C, even when the definition would allow several functions in a module. Software components in DO-178 can be regarded as modules as defined in IEC 61508, which would mean that testing them is module testing. Thus, depending on the standard, the terms module and unit are used to refer to the test object. However, if you apply the definitions to a specific programming language, the end result is the same test object for the programming language C, namely a function as defined in C. To prevent a mix-up with the source modules in C, a function as defined in C is referred to as a unit in the following, not a module What is Unit testing? Unit testing tests a single function (as defined in C) in as comprehensive and isolated a way as possible. The unit is tested at its interface against its specification. We are therefore talking about a black box test in which the inner structure of the unit is not the decisive factor. The unit interface consists of the set of variables that the function reads (i.e. the input variables) and the set of variables that this unit writes (i.e. the output variables or result variables). The function s return value, if it has one, is part of the results. Of course, a variable can be an input variable and an output variable at the same time, such as when a variable is incremented by the unit. Functions that the test unit calls are also part of the interface in a broader sense. The unit s interface produces the structure of the test cases because a test case comprises 4

5 concrete values for input variables and the values for output variables that you expect when conducting a test with the available input values When does unit testing make sense? Unit tests reduce the complexity of testing. They enable us to focus on testing part of a functionality in an entire application and verify whether the unit correctly performs this functionality (although errorsensitive inputs are also checked, not just normal inputs). It can be very difficult or even impossible to define test cases for a unit that is very high in the call hierarchy so that called units that may be significantly lower in the call hierarchy are also adequately tested. Error handling of called units can be particularly difficult to test if the calling unit always uses the called unit correctly. This is why testing in isolation makes so much sense, because it avoids these problems. Since unit testing is supposed to ensure that a unit fulfills its intended function (and nothing more), security-critical units are especially predestined for unit testing. Since unit testing checks the relationship between the values of input and output variables, unit tests are particularly useful when converting input values to output values requires very complicated or extensive calculations. Examples of this are units that sort, analyze data, make complex decisions, etc. To identify units like these, you can use metrics such as McCabe s cyclomatic complexity metric. It is also good to test units that the application uses very frequently, since it is important that these units function correctly. However, there are many test questions that unit testing cannot answer by itself. One example is verifying time interactivity between units Who should conduct unit testing? It s a generally accepted fact that developers and software developers are not well-suited to testing their own code, especially when it comes to completeness of implementation or the treatment of unexpected inputs. After all, if a developer has failed to implement a functionality that the specification calls for, he or she is not likely to think about testing that missing functionality either. On the other hand, experience has shown that testers who conduct unit testing for code they did not create themselves spend most of their testing time getting to know the interface. They have to find out how interface variables convert test inputs (such as color information like green ) to concrete values, and they have to judge whether the values in the result variables are right or wrong. One way to resolve this dilemma could be for testers who did not write the code to define the required test cases based on the specification (including the expected results). They can use abstract data for this (test with color green ). They then hand the set of test cases over to the software developer to conduct. The developer now executes the test cases one by one, as long as it is simple enough for him or her to convert the abstract data provided ( green ) to concrete test data (such as an RGB value). If a test case does not provide the desired result, the developer can immediately investigate the problem and, if necessary, change the implementation. Executing the tests can thus be considered another step in processing the software, similar to a compiler checking the syntax. Many standards require an organizational separation of development and testing, and the method proposed here does not quite meet that requirement. In addition, developers are generally considered too valuable a resource to waste on testing. As a result, this type of distributed testing is rarely encountered in practice (for now) How is unit testing performed? There are different approaches you can take to test a unit. One option is to use the original application (original binary). Another is to use an application that has been created specifically for testing (test application). Original-binary testing The original-binary approach usually uses the entire application for testing, exactly as it will be in the finished product (fully translated, bundled and allocated). The process is similar to manual testing: A debugger is usually used as the execution environment for the application. A breakpoint is added at the 5

6 beginning of the unit to be tested. The next step is to start the application. After it reaches the breakpoint, the test input data for the unit is supplied and the test runs. The test result is produced at the end of the unit. Another test case can be executed as soon as the application has returned to the breakpoint at the beginning of the unit. Test-application testing The test-application approach attempts to only use the test unit from the original application. The other parts that are needed to have an executable application, known as the test drivers (for example, main() and the startup code), need to be supplied separately. The executable test application is produced from them along with the test unit. Every test unit requires its own special test application. Ideally, you should use placeholder functions (stubs) for functions that are called by the test unit. This ensures that the test unit can be tested in a reproducible way because it isolates the test unit from the behavior of the functions it calls. The special test applications may already contain the test data. That data is then transferred to the test system as soon as the test application is loaded. This can lead to problems in test systems with limited storage resources because the test application may not fit into storage when there is a large number of test cases. If test data is kept on the host, and only the test data that is needed for a specific test case is transferred to the target system, this problem does not normally occur, even if the number of test cases is very large. The drawback to this is slightly higher communication costs. Once the special test application is created, you can proceed as you would in the original-binary approach. Pros and cons The advantage of the original-binary approach is that it tests the test unit at its future address. This approach to unit testing does not require any additional effort for translation or linking because it uses the original application. It also allows you to conduct unit testing for applications that are already in a target system s ROM. In a case like that, you do not even need a compiler. (You still need a debugger though.) The original-binary approach also comes with some disadvantages. For instance, it can take a very long time for the normal application to call the test unit, which must happen for the test to take place. It is possible that the test unit will never be called under normal operation, and you will have to take special precautions to be able to execute the test, such as flipping a switch or triggering an interrupt. In an original-binary test, you cannot use placeholder functions for functions called by the test unit unless you take special steps. This is why all called units in the application are always used when the test unit is tested. As a result, test cases may show an error for a certain unit after an application has been republished, even though no changes were made to that unit. This is usually caused by a change at another point that led to an unexpected side effect. In addition, original-binary testing does not allow you to use any test input data you want. For example, if a unit receives its test data through a pointer, you can only supply as much test data as fits in the storage area that the application has provided Automatation is indispensible All standards require tests to be repeatable. The reason for this is obvious: You need to be able to repeat tests after making changes in the test object (or the development environment). Manual testing is too impractical for this purpose because it requires far too much time and effort. Instead, you need to automate at least the execution part of testing with appropriate tools. Tools can also complete other work, such as identifying the interface, managing test data, creating the test application, conducting the test, evaluating it, determining test coverage, and, last but not least, creating test documentation. 6

7 1.2 Example with Tessy The test unit As an example, let s look at a unit that solves the following problem. Say we have a number range that is specified by a start value and a length. We want to determine whether another value lies in this range (start <= value < (start + length)). We only want to look at whole numbers. This unit could be implemented in C as follows: Figure 1 Test unit This implementation has at least one obvious error. If (value == (start + length)) is true for the test value, the value will be mistakenly identified as being in the range, i.e. range_start == 5, range_len == 2 and v1 == 7 will yield yes instead of no The unit interface Tessy uses that unit s source code to automatically determine which variables in the unit are to be read or written. This identifies the unit interface. 7

8 Figure 2 The unit interface tells you the passing direction (IN / OUT) of variables The test unit has the number range (in the structure) and the test value as an input variable (IN). The result (the output) is the function s return value (OUT) Initial test cases The interface provides the structure of a test case. This lets you create ad-hoc test cases and fill them with data. But actually, test cases should be identified using the classification tree method. This is explained in part two. Figure 3 Entering test data for two ad-hoc test cases 8

9 We have created two ad-hoc test cases. The test cases or test data can be exported to Excel or another program and displayed there. (Test cases can also be imported from Excel.) Figure 4 Test cases displayed in Excel Conducting the test Generating the test application Tessy can generate the source code needed to create a special test application and translate and bind it with the source code from the test unit. It can do this with a cross-compiler for the desired target system or the Gnu C compiler. Loading the application Tessy loads the application to the target system for the test. This usually happens through a suitable debugger that Tessy controls. The debugger can control an instruction-set simulator, an in-circuit emulator with or without connected hardware, an OCDS / BDM / JTAG debugger or similar as a test target system. The tests can therefore be executed on real hardware if desired. Conducting all tests Tessy keeps the test data separate from the test application. For this reason, Tessy starts by transferring the test input data needed to conduct the first test case to the test target system. Tessy then conducts the test and determines the output data. It repeats this process for all remaining tests. Test evaluation and documentation Tessy uses a comparison of output data and expected results to specify the test result (passed or failed), which is displayed in green or red. You can have Tessy create test documentation as a PDF, Word or HTML file. There is also an XML option, which is intended for machines. Figure 5 Test results with details from the second test case Measuring code coverage Tessy can measure a test s Entry Point Coverage, Statement Coverage, Branch Coverage, Decision Coverage, Modified Condition / Decision Coverage (MC/DC), and Function Coverage. 9

10 1.2.5 Troubleshooting and regression testing Tessy enables a smooth transition from testing to debugging. A test case that yielded an unexpected result can be repeated in such a way that test-unit debugging begins immediately. Tessy achieves this by setting a breakpoint at the test unit. There are essentially two reasons why a test would yield an unexpected result: Either the expected result is incorrect or there is a flaw in the implementation of the unit. Once the (presumed) cause of the problem is identified and corrected, Tessy can be automated to repeat the test, and not just for the test case in question but for all the test cases that have been (successfully) conducted before it. This is particularly important if the software was modified to correct the error, because previous tests would then have been based on a different implementation of the unit, which would mean they are no longer valid. 2 The classification method 2.1 Purpose The classification tree method transforms a (functional) specification into a set of error-sensitive, lowredundancy test case specifications. This methodical process is intended to ensure that no important test cases are overlooked, i.e. the resulting set of test case specifications should contain all relevant test cases. Under this assumption, the method also provides a test completion criterion. 2.2 Characteristics of the classification method The classification tree method is generally applicable, i.e. it can be used for functional problems outside of software development as well. The classification tree method is not automatic; a person needs to apply it. This means, of course, that the result depends on the thinking and estimations of that human being. It is therefore advantageous for the user to be familiar with the problem and its environment. Different users are likely to identify different sets of test case specifications as a result for the same functional problem. Human error cannot be fully ruled out either. This contrasts with the systematic process that is expected of the user of this method and which leads him or her to specify test cases with a high probability of revealing an error. A key advantage of the classification tree method is that it presents test ideas visually, making them easy to understand in situations such as reviews. Reviews are an effective way of increasing the certainty that no test case specifications are missing. The classification tree method integrates several known approaches for deriving test cases, such as the equivalence partitioning method and the idea to test the boundaries of a range. 2.3 Applying the classification method Identifying test-relevant aspects The classification tree method starts with an analysis of the functional specification, or to put it another way, with a human user thinking about that specification. The goal here is to recognize what are called the test-relevant aspects of the problem. The variety of values that these aspects can take on guarantees a variety of different meaningful test cases. Example of a test-relevant aspects For a distance measuring device that measures distances with ultrasound, the temperature of the medium (air) is a test-relevant aspect because it influences the speed at which sound travels through the air. For this reason, it makes sense to conduct tests at different temperatures to check whether the temperature has been correctly incorporated into the distance calculation from the travel time. Air temperature does not play a role for a distance measuring device that uses radar instead of ultrasound (it even works without air). Thus, tests at different temperatures would not be meaningful and temperature is not a (meaningful) test-relevant aspect. This example also shows that it makes sense for the person using the method to have some familiarity with the problem environment. 10

11 2.3.2 Classification Once all test-relevant aspects are found, you need to look at the values that each aspect can accept. The next step is to divide the value range you found into classes, just like in the equivalence partitioning method. All values in the class should be considered to be equivalent for the test, i.e. the test result is not affected by which value from a class you use for the test Repeating to obtain a classification tree You can divide each class into sub-classes based on other aspects. Repeatedly forming sub-classifications by using the classes they belong to will ultimately produce a classification tree. The problem forms the root of the tree, which grows from the top down. Classifications are framed in boxes. Dividing the problem up into different aspects that can subsequently be refined separately, as described above, reduces the complexity of the original test problem significantly. Figure 6 Schematic for a classification tree with a combined table Specifying test cases in the combination table The leaves of the classification tree form the head of the combination table. A row in the combination table specifies a test case by marking the leaf classes of the tree from which a value is used for this test case. However, the method is not intended as a way to mechanically specify all possible combinations of leaf classes as test cases. Instead, the user should use a test case to cover multiple aspects in order to arrive at a minimal non-redundant set of test case specifications. Like when creating the tree, the number of test cases specified (and which ones) depends on the user s judgment. Clearly, a tree with many leaf classes requires more test cases than a tree with a small number of leaves. As a rule of thumb, the number of leaves should be the same as the number of test cases needed for reasonable coverage of a classification tree. 11

12 2.4 Example For the functional specification in part one ( Does a value lie in a range? ), you want to identify test cases using the classification tree method. The location of the range (given by a start value and length) and the position of the value that is to be identified as lying in the range or not can be considered test-relevant aspects, because in light of the length, for example, it makes sense to test using ranges of different lengths. A specification normally offers a basis for classifying test-relevant aspects. In the present specification, however, this is not the case for the start value of the range. As a result, all we can do is make a trivial division of the start values into negative and positive. (If the specification said something like, If the start value is even, the length doubles, it would be clear that we would need a class with even start values and a class with odd start values.) Zero gets its own class, but you could also assign zero to the class with positive (or negative) values; users should use their own best judgment here. For the length aspect, the same considerations apply as for the start value. The only difference is that it makes very good sense to create a separate class for the length of 0, because this ensures that there will be at least one test case in which the range has a length of 0. These considerations lead to a preliminary classification tree for the range s start and length aspects; the aspect for the position of the value is not part of the tree yet. Also preliminary is the specification in the combination table of two test cases (strictly speaking, they are ranges). This produces the following figure: Figure 7 Rudimentary classification tree for the range and two specifications The first specification combines a positive start value with a positive length. The range with a start value of 5 and a length of 2, which was used in the two ad-hoc test cases in part one, matches this specification. In that respect, this range specification is not extraordinary. The second test case is different. It combines a negative start value (such as 5) with a negative length (such as 2), which leads directly to the question of what values should be in a range like this. Should it contain 4 or 6 or is it better if it has no values at all? The problem specification should actually provide the answer to this question, but it does not. As a result, this question cannot and should not be decisively answered here. However, it is important to add that systematically applying the classification tree method can also reveal problems in the specification. If you extend the rudimentary classification tree to include the aspect of position, the two classes will obviously be inside or outside the range. It is equally obvious that the outside class will need to be further classified according to whether the value is below or above the range. Taking into consideration that preference should be given to tests on the borders of the range and that the largest positive number and smallest negative number could be interesting start values for the range gives rise to the following tree and a matching set of test case specifications. 12

13 Figure 8 Complete classification tree with test case specifications Test case specification 6 describes the combination of a negative start value and a negative length. A value should be chosen that is within the range, and, more specifically, on the border opposite the start value. This assumes that the problem specification has been extended to allow for this. Test case specifications 11 and 12 combine the largest possible positive start value with a positive length. Here, the start value is in the range, which test case specification 11 checks. Test case specification 12 makes sure that no other values, such as a very large negative value, are mistakenly identified as being in the range through wrap-around. In addition to the test case specifications given for the classification tree shown, there are other sets of test case specifications that make just as much sense. It is easy to imagine other classification trees too. However, it is important for users of the classification tree method to clarify what they hope to achieve through the test by carefully choosing the names of classes, classifications and test cases. 2.5 Tools to support the classification tree method There is a special tool you can use to apply the classification tree method the Classification Tree Editor (CTE). It offers a syntax-controlled graphical editor for creating classification trees. You can also manage large classification trees by moving refinements (sub-trees) to additional drawing areas. A navigator window makes it easier to orient yourself and move around in the trees. Descriptions and attributes can be added to elements in the tree. You can also use color in your tree. CTE helps compile test case specifications in the combination table by using the structure of the tree to automatically prevent invalid specifications (for instance, the start value cannot be positive and negative at the same time). The trees and test case specifications that you create can be documented and printed in different formats. Tessy includes CTE, and test cases specified in CTE can be transferred to Tessy. 13

14 3 Summary Unit testing is just one of many ways to ensure and improve software quality. Unit testing enables you to test small units that may be hidden in the overall system under extreme conditions, such as with invalid inputs, which would not otherwise be possible during system tests. Separately testing individual units reduces the overall complexity of testing. Errors can be directly assigned to a specific unit, and you do not need to search the entire application for them. The subsequent application is made up of separately tested individual components, which reduces the likelihood of errors during your system test. Once you have compiled an adequate set of test cases for testing a unit (the classification tree method is particularly well-suited for this purpose), you will have a good foundation for revising the unit. If, after revising the unit, all of the test cases still provide the expected result, it is safe to assume that the revision is okay. It is impractical to deploy unit testing without the help of certain tools, especially if you want to conduct regression tests. Tessy, CTE and the classification tree method come from the software technology lab of Daimler AG. They have been used in practice for a number of years, especially in the automotive industry. 4 Literature [1] [2] Grochtmann, M., Grimm, K.: Classification Trees For Partition testing, Software testing, Verification & Reliability, Volume 3, Number 2, June 1993, Wiley, pp [3] Wegener, J., Pitschinetz, R.: Tessy Another Overall Unit Testing Tool, Quality Week [4] Grimm, Klaus: Systematisches Testen von Software: Eine neue Methode und eine effektive Teststrategie. München, Wien, Oldenburg, GMD-Berichte Nr [5] Liggesmeyer, Peter: Software-Qualität: Testen, Analysieren und Verifizieren von Software. Heidelberg, Berlin, Spektrum Akademischer Verlag. [6] Broekman, B., Notenboom, E.: Testing Embedded Software. Addison-Wesly, About the Author: Frank Büchner, Principal Engineer Software Quality, Hitex GmbH Frank Büchner has a degree in IT and has been working in the Embedded Software Division at Hitex for over 25 years. He has been involved in a variety of tasks through the years, from embedded software development to his current position as Principal Engineer Software Quality. Frank has been dealing with the topic of testing and software quality for several years. He shares his knowledge through regular talks and specialist articles. 14

15 Building a safe and secure embedded world Hitex Head Office, Germany Hitex UK Hitex GmbH Greschbachstraße Karlsruhe Germany Phone: Fax: info@hitex.de Hitex (UK) Ltd Millburn Hill Road University of Warwick Science Park Coventry CV4 7HS United Kingdom Phone: Fax: info@hitex.co.uk WP-UNITTEST-E Consulting Engineering Testing Training Tools Software Components Systems Manufacturing