Unit Testing with JUnit and CppUnit

Transcription

1 Unit Testing with JUnit and CppUnit Software Testing Fundamentals (1) What is software testing? The process of operating a system or component under specified conditions, observing or recording the results, and making an evaluation of some aspect of system or component. (IEEE Definition) Why do we test? To ensure the quality and satisfaction of the product To gain the confidence in the correctness of the product Testing vs. debugging Testing is to show that a program has bugs Debugging is to locate and correct the error or misconception that cause the program failures 2

2 Software Testing Fundamentals (2) Software Testing Techniques White-box testing The tester has access to the details of the program under test and performs the testing according to such details. Examples: path testing, branch testing Black-box testing Black-box testing treats the program under test as a black box. No knowledge about the implementation is assumed. Examples: equivalent partitioning, boundary value analysis 3 Software Testing Fundamentals (3) Testing should begin in the small and progress toward testing in the large Unit testing Concentrates on each unit (i.e., component) of the software Integrated testing Building a system from its components and testing the resultant system for detecting component interaction problems System testing Concern with testing an increment to be delivered or entire system Acceptance testing Performed by the clients/users to confirm that the product meets the business requirements 4

3 Unit Testing Unit Testing Is normally considered as an adjunct to the coding step Focuses verification effort on the smallest unit of software design the software component or module Using the component-level design description as a guide Provide a release criterion for a programming task Unit Testing in the OO Context Smallest testable unit is the encapsulated class Conducting class testing (i.e., unit testing) Methods within the class are tested The state behavior of the class is examined 5 Unit Testing: Methodology void Foo() { ; Function under test Test case(s) void testfoo() { x.foo(); CPPUNIT_ASSERT( ); ;

4 Unit Testing: Methodology Example #1 int Max(int x, int y) { ; void testmax() { int max = x.max(10,5); CPPUNIT_ASSERT(max == 10); ; Unit Testing: Methodology Example #2 int Max(int a[], int size) { ; void testmax() { int a[5]={3,4,5,1,0; int max = x.max(a,5); CPPUNIT_ASSERT(max == 5); ;

5 Unit Testing: Methodology Example #3 int Sum(int a[], int size) { ; void testsum() { int a[5]={3,4,5,1,0; int sum = x.sum(a,5); CPPUNIT_ASSERT(sum == 13); ; Unit Testing Example: Statement coverage (1) void f() { ; Statement coverage: are all statements of f() executed? void testf() { x.f(); CPPUNIT_ASSERT( ); ; 10

6 Unit Testing Example: Statement coverage (2) void f(bool x) { if (x) { ; There are unexecuted statements void testf() { x.f(false); CPPUNIT_ASSERT( ); ; 11 Unit Testing Example: Statement coverage (3) void f(bool x) { if (x) { ; All statements are executed void testf() { x.f(true); CPPUNIT_ASSERT( ); ; 12

7 Unit Testing Example: Branch coverage (1) Unexercised void f(bool x) { ; if (x) { Brach coverage: are all control transfer exercised? All statements void testf() { are executed x.f(true); CPPUNIT_ASSERT( ); ; 13 Unit Testing Example: Branch coverage (2) void f(bool x) { if (x) { ; All control transfers are exercised Two test cases void testf() { x.f(true); CPPUNIT_ASSERT( ); x.f(false); CPPUNIT_ASSERT( ); ; Is reusing x a good idea? 14

8 Unit Testing Example: Branch coverage (3) void f(int x) { do { x--; while (x>0); Statement ; coverage? Yes Branch coverage? No void testf() { x.f(0); CPPUNIT_ASSERT( ); ; 15 Unit Testing: Difficult to Test? void LoadFromFile() { ifstream fin( FileName ); ; Sometimes, you need to rewrite your code to make it easier to test FileName should be parameterized void testloadfromfile() { x.loadfromfile(); // The file is not under // control. // How to do assert // for the file? ;

9 Unit Testing: Difficult to Test? Better void LoadFromFile(string filename) { ifstream fin(filename.c_str()); ; void testloadfromfile() { x.loadfromfile( a_file ) ; // The file is under // control ; Unit Testing: Difficult to Test? void LoadFromFile(ifstream &fin) { ; Though, the file is now assigned by the tester. The assert may still be difficult to implement, because the LoadFromFile() function is large. Sometimes, you need to break a large function into smaller ones. ; Another possibility void testloadfromfile() { ifstream fin( a_file ); x.loadfromfile(fin) ; // The file is under // control

10 Unit Testing: Difficult to Test? void LoadFromFile(ifstream &fin) { if (condition #1) LoadPart1(); ; LoadPart2(); Breaking a large function into smaller ones. Test each smaller function first. Testing for LoadFromFile(). void testloadfromfile() { void testloadpart1() { void testloadpart2() { ; Advantages of Unit Testing Write test = understand better what has to be done Silly bugs appear immediately = less time spent on debugging Provide a working specification of your functional code = tests are good documentation Speed-up the development write test once, use test to find errors many times Gain confidence in your code Repeatable and deterministic Regression tests incorrect changes discovered immediately; unautomated refactoring enabled Encouraged by extreme Programming community 20

11 Old-fashioned Low-level Testing Stepping through a debugger drawbacks: 1. not automatic 2. time-consuming Littering code with stream output calls drawbacks: 1. makes code ugly (ok, nowadays you could use aspects to avoid it;-)) 2. generates too much information 21 What is xunit? An automated unit test framework Provides the Driver for unit(s) Provides automatic test runs Provides automatic result checks Available for multiple languages: JUnit (from Kent Beck (XP) and Erich Gamma(Gang of Four)) CppUnit httpunit NUnit 22

12 The Goals of JUnit To write a framework within which we have some glimmer of hope that developers will actually write tests. The framework has to use familiar tools, so there is little new to learn. The second goal of testing is creating tests that retain their value over time. Someone other than the original author has to be able to execute the tests and interpret the results. Creating a setup or fixture is expensive A framework has to enable reusing fixtures to run different tests. 23 The Design of JUnit Patterns Generate Architectures Used to present the design of JUnit. The idea is to explain the design of a system by starting with nothing and applying patterns, one after another, until you have the architecture of the system. Presentation flow (1) Present the architectural problem to be solved (2) Summarize the pattern that solves it (3) Show how the pattern was applied to JUnit. The details can refer to JUnit A Cook's Tour 24

13 The Patterns Used in JUnit 25 How to Use JUnit (1) Write a test case (Fixture) Create your own test case as a subclass of JUnit TestCase Add an instance variable for each known object in the fixture Override the setup() method to initialize object(s) under test Override the teardown() method private to release Collection object(s) collection; under test Run the test protected void setup() { Define a public test???() method collection for exercising = new ArrayList(); the object(s) under test Verify the result protected void teardown() { Assert expected result of the test collection.clear(); case using assertequals(), asserttrue, Clean up the fixture public void testemptycollection() { through the teardown() method public class BookTest2 extends TestCase { asserttrue(collection.isempty()); 26

14 How to Use JUnit (2) Suite management A test suite is a collection of test cases that are intended to be return suite; used to show that a program under test has some specified set of behaviors public static Test suite(){ Write a static suite() containing all the test???() in the fixture return new TestSuite(BookTest.class); class Optionally define a main() method that runs the TestCase in batch mode Error vs. Failures Error: unanticipated problem like an ArrayIndexOutOfBoundsException public static void suite(){ TestSuite suite = new TestSuite(); suite.addtest(new BookTest("testEquals")); suite.addtest(new BookTest("testBookAdd")); public static void main (String[] args) { junit.textui.testrunner.run(suite()); Failure: is anticipated and can be checked with assertions 27 JUnit FAQ: Best Practices When should tests be written? Tests should be written before the code. Test-first programming is practiced by only writing new code when an automated test is failing. When all the tests pass, you know you're done! When a bug is reported, first write unit test(s) to expose the bug(s), then fix them. This makes it almost impossible for that particular bug to resurface later. Good tests tell you how to best design the system for its intended use. Test-driven development is a lot more fun than writing tests after the code seems to be working. 28

15 JUnit FAQ: Best Practices Do I have to write a test for everything? No, just test everything that could reasonably break. Investments in testing are equal investments in design. If defects aren't being reported, and your design responds well to change, then you're probably testing enough. If you're spending a lot of time fixing defects and your design is difficult to grow, you should write more tests. If something is difficult to test, it's usually an opportunity for a design improvement. 29 JUnit FAQ: Best Practices How simple is too simple to break? If it can't break on its own, it's too simple to break. Example getx() method cannot break unless the compiler is also broken. Therefore, don't test getx(). setx() method is also too simple to break. However, if it does any parameter validation, you likely need to test it. 30

16 JUnit FAQ: Best Practices How often should I run my tests? Run all your unit tests as often as possible Ideally every time the code is changed. Make sure all your unit tests always run at 100%. Frequent testing gives you confidence that your changes didn't break anything. For larger systems, you may just run specific test suites that are relevant to the code you're working on. Run all the tests of the a system at least once per day (or night). 31 JUnit FAQ: Best Practices What do I do when a defect is reported? Write a failing test that exposes the defect When the test passes, you know the defect is fixed! This is a learning opportunity Perhaps the defect could have been prevented by being more aggressive about testing everything that could reasonably break. Why not just use print? It requires that output be scanned manually every time the program is run to ensure that the code is doing what's expected. Tests should retain its value over time. Why not just use a debugger? The same as that of using print. 32

17 JUnit Primer: Testing Idioms Testing Idioms Code a little, test a little, code a little, test a little... Begin by writing tests for the areas of code that you're most worried about breaking. Write tests that have the highest possible return on your testing investment. When you need to add new functionality to the system, write the tests first. If you find yourself debugging using System.out.println(), write a test case instead. The next time someone asks you for help debugging, help them write a test. Don't deliver software that doesn't pass all of its tests. 33 CppUnit Cookbook (1) Simple Test Case (Ordinarily, you will not use such simple test) Subclass the TestCase class. Override the method runtest(). When you want to check a value, call CPPUNIT_ASSERT(bool) and pass in an expression that is true if the test succeeds class ComplexNumberTest : public CPPUNIT_NS::TestCase { public: ComplexNumberTest( std::string name ) : CppUnit::TestCase( name ) { void runtest() { CPPUNIT_ASSERT( Complex (10, 1) == Complex (10, 1) ); CPPUNIT_ASSERT(!(Complex (1, 1) == Complex (2, 2)) ); ; assuming == has been overloaded bool operator ==( const Complex &a, const Complex &b ) { return a.real == b.real && a.imaginary == b.imaginary; 34

18 CppUnit Cookbook (2) Fixture - a known set of objects served as a base for a set of test cases To add new tests Add member variables for each part of the fixture Override setup() to initialize the variables Override teardown() to release any resources allocated in setup() class ComplexNumberTest : public CPPUNIT_NS::TestFixture { private: Complex *m_10_1, *m_1_1, *m_11_2; protected: void setup() { m_10_1 = new Complex( 10, 1 ); m_1_1 = new Complex( 1, 1 ); m_11_2 = new Complex( 11, 2 ); void teardown() { delete m_10_1; delete m_1_1; delete m_11_2; ; 35 CppUnit Cookbook (3) How do you write and invoke individual tests using a fixture? Write the test case as a method in the fixture class Create a TestCaller which runs that particular method class ComplexNumberTest : public CPPUNTI_NS::TestFixture { private: Complex *m_10_1, *m_1_1, *m_11_2; protected: void setup() {... void teardown() {... void testequality() { CPPUNIT_ASSERT( *m_10_1 == *m_10_1 ); CPPUNIT_ASSERT(!(*m_10_1 == *m_11_2) ); void testaddition() { CPPUNIT_ASSERT( *m_10_1 + *m_1_1 == *m_11_2 ); ; CPPUNIT_NS::TestCaller<ComplexNumberTest> test("testequality", &ComplexNumberTest::testEquality ); CPPUNIT_NS::TestResult result; test.run( &result ); 36

19 CppUnit Cookbook (4) Use TestSuite class that runs any number of TestCases together CPPUNIT_NS::TestSuite suite; CPPUNIT_NS ::TestResult result; suite.addtest( new CPPUNIT_NS::TestCaller<ComplexNumberTest>( "testequality", &ComplexNumberTest::testEquality ) ); suite.addtest( new CPPUNIT_NS::TestCaller<ComplexNumberTest>( "testaddition", &ComplexNumberTest::testAddition ) ); suite.run( &result ); TestSuite can contain any object implementing the Test interface CPPUNIT_NS::TestSuite suite; CPPUNIT_NS ::TestResult result; suite.addtest( ComplexNumberTest::suite() ); suite.addtest( SurrealNumberTest::suite() ); suite.run( &result ); 37 CppUnit Cookbook (5) Run TestSuite using TestRunner Using Helper Macros Defined in cppunit/extensions/helpermacros.h which must be integrated for initiating and finishing a test suite (CPPUNIT_TEST_SUITE and CPPUNIT_TEST_SUITE_END) Rewrite the fixture to include the HelperMacros.h #include <cppunit/extensions/helpermacros.h> class ComplexNumberTest : public CppUnit::TestFixture {... TestSuite can be created with the HelperMacros CPPUNIT_TEST_SUITE( ComplexNumberTest ); //pass class name CPPUNIT_TEST( testequality ); // declare test case CPPUNIT_TEST( testaddition ); CPPUNIT_TEST_SUITE_END(); //end suite declartion 38

20 CppUnit Cookbook (6) //ComplexNumberTest.h Run TestSuite using TestRunner #include <cppunit/ui/text/testrunner.h> CPPUNIT_TEST_SUITE( ComplexNumberTest ); #include "ComplexNumberTest.h" CPPUNIT_TEST( testequality );... CPPUNIT_TEST_SUITE_END(); int main( int argc, char **argv) { CPPUNIT_NS::TextUi::TestRunner runner; runner.addtest(complexnumbertest::suite()); bool wassuccessful = runner.run(); return wassuccessful? 0 : 1; If all the tests pass, you'll get an informative message. If any fail, you'll get the following information: The name of the test case that failed The name of the source file that contains the test The line number where the failure occurred All of the text inside the call to CPPUNIT_ASSERT() which detected the failure 39 CppUnit Cookbook (7) CppUnit macros CPPUNIT_ASSERT Check whether the passed expression returns the value True CPPUNIT_ASSERT_EQUAL Check whether the first parameter is like the second one CPPUNIT_ASSERT_THROW Check whether the passed expression throws an exception of the passed type CppUnit Examples See LinkedListTest 40

21 Test-driven Development (TDD) Test-driven development (TDD) is an evolutionary approach to development which combines Test-first development Write a test before you write just enough production code to fulfill that test Refactoring The goal of TDD Think through your design before your write your functional code To write clean code that works TDD is primarily a design technique with a side effect of ensuring that your source code is thoroughly unit tested 41 Unit testing and Test-driven Development (TDD) How to write JUnit/CPPUnit tests? Do it TDD! Workflow: 1. Think about functionality to be implemented & scenarios in which the unit to be tested will play a role. 2. Create a stub of the unit you want to implement. 3. Write a test for each scenario and/or use of the unit. 4. Make the unit fail the tests (nothing is implemented yet!). 5. Develop the unit until it passes every test. Encountered new scenario/use? make a test before implementing it! 42

22 References W.-K Chen, Object-Oriented Programming - Software testing JUnit A Cook's Tour. htm CppUnit Cookbook. kbook.html Krzysztof Pietroszek, Unit testing with JUnit and CPPUnit. 43