Automating C/C++ Unit Testing and Coding Standard Analysis with Parasoft C++test

Automating C/C++ Unit Testing and Coding Standard Analysis with Parasoft C++test

Introduction Preventing errors is the key to delivering reliable C/C++ software as rapidly and efficiently as possible. It is also the key to keeping your work focused on the tasks you enjoy most. The less time you spend chasing bugs, reproducing errors, correcting code, and retesting the modifications, the more time you have for more interesting tasks, such as researching, designing, and implementing new technologies. The best way to prevent C/C++ errors is to regularly perform proven error prevention practices such as coding standard compliance and unit testing from the start of development. However, these practices are typically implemented in a way that is time-consuming and disruptive to developers work. Some industrystandard coding standards can usually be checked automatically, but developers still need to sort through noise, correct the meaningful violations, then recheck the modified code. Checking organizational and application-specific coding standards usually requires manual code reviews. And although unit tests can be executed automatically, developers must first determine how to effectively verify each class's functionality and robustness, then design and implement the necessary harnesses, test cases and stubs. When error-prevention practices require this much effort, error prevention simply isn t feasible for most developers and development groups. Parasoft C++test removes these barriers to error prevention by automating unit testing and coding standard compliance. With C++test, you can start unit testing without building a test harness, stubs, or test cases, plus you can check coding standards without manually reviewing code. C++test analyzes C and C++ classes, functions, or components, then automatically generates a test harness, test cases, and stubs (if necessary) designed to verify the code under test. Test cases and stubs can be added or customized as needed through user-friendly GUI controls or directly in C/C++ source, using a format similar to CppUnit. After executing the test cases, C++test reports coverage details for both automatically-generated and user-defined test cases. C++test also checks whether code follows an extensive range of coding standard rules designed to prevent application problems such as slow performance, memory leaks, security vulnerabilities, and so on. Additionally, it can check any number of custom rules that you design with its user-friendly graphical or automated rule-creation interface. With the click of a button, you can automate coding standard analysis and unit testing (including exception testing, functional testing, and regression testing), preventing software errors and ensuring that code is structurally sound, reliable, maintainable, and portable. With C++test, error prevention can become a seamless part of your development process. No difficult setup is required; C++test starts pinpointing problems immediately. If you have just completed a new class or function, you simply launch C++test and tell it which preconfigured or custom test scenario you want it to run. If you are developing in Microsoft Visual Studio, you can start tests directly from that IDE. Alternatively, tests can be scheduled to run automatically each night, so testing does not disrupt your daily work. In every case, C++test helps you quickly identify and prevent critical problems in your C/C++ code. In addition to making it easier for individual developers to perform unit testing and follow coding standards, C++test can standardize and manage C/C++ error prevention across a team and support the Parasoft AEP TM Methodology, a new methodology for improving software quality and reliability in a team environment. Implementing C++test as part of a team-wide AEP solution is a critical step in improving overall software quality and streamlining your team s software development process. 2

Figure 1: C++test GUI This paper explains how practices such as automated unit testing, coding standard compliance, and teamwide C/C++ error prevention can help you produce better C/C++ code in less time. Along the way, it illustrates how C++test makes these critical practices feasible for any developer or team. Automated Unit Testing "Unit testing" refers to testing software code at the simplest functional point, which is typically a single class or a function. Unit testing is typically performed by developers inside a development cycle, rather than in the QA phase. Using unit testing, you can ensure that the application building blocks are solid before they are integrated, thus improving the quality of the entire application. When you test early, you typically have the code fresh in your mind, so it is typically less difficult and time-consuming to identify and fix defects at this point. Manual unit testing typically involves writing test harnesses by hand, specifying input data, and supplying stubs for missing functions. C++test automates these tasks to make the unit testing process more efficient and consistent 3

Generally, unit testing can include: Exception testing (also known as stress testing or reliability testing), which is used to confirm that code is structurally sound and can handle the full range of feasible inputs and their combinations without throwing an unexpected exception. Functional testing, which is used to verify that software building blocks conform to their specification and that all of the intended functionality is included and working correctly. Creating functional tests at the unit level involves human input to specify particular input and state conditions, and expected output. Functional testing can be implemented as white box testing (testing with knowledge of internals and implementation of a unit under test), or black box testing, which is solely based on external behavior of a unit under test. Regression testing, which is used to verify consistent behavior of the code. This is typically done by developing a set of tests, verifying their correctness, and running them after code changes to catch deviations in code behavior. Regression testing may rely on both Exception testing and Functional testing. Unit testing dramatically improves software quality by helping you detect errors during implementation, the stage where it is easiest and most cost-effective to find and fix errors. First of all, unit testing helps you detect errors that application-level testing might not find. When you perform application-level testing, there is a good chance that the tests will not even reach many of the units, much less exercise them thoroughly enough to expose the functional and construction problems in those units. Unit testing is critical because it provides a prime opportunity to expose the errors that might get overlooked in later phases of testing. When you test one unit directly (apart from all other units), there is a much greater chance that your test inputs will thoroughly test that unit and expose any errors that might be lurking within that unit. The second way that unit testing facilitates error detection is by preventing bugs from spawning more bugs, which relieves you from having to wade through problem after problem to remedy what began as a single, simple error. Because bugs build upon and interact with one another, if you leave a bug in your code, chances are it will lead to additional bugs. If you delay testing until the later stages of development, you will not only spend more time fixing each bug, but you will also have more bugs to fix. If you test as you go, you will avoid this scenario. The result: a significant reduction in debugging time and cost. How C++test Automates Unit Testing The C++test Solution With the click of a button, C++test analyzes any C/C++ code from a single method to an entire project, and then automatically generates a test harness, test cases, and any necessary stubs. Preconfigured test modes make it simple to tailor testing to your goals, such as generating test cases that test corner conditions and identify exceptions, generating a black-box regression test suite (focusing only on method inputs and outputs), or generating a white-box regression test suite (considering object state, external variables, and execution results as well as method inputs and outputs). By running these test cases in C++test, you verify class robustness and identify inputs that could cause the program to enter an inconsistent state or terminate. You can customize the preconfigured test modes, as well as specific test generation settings. Two types of unit test cases can be generated: source test cases (test cases that are implemented in C or C++ source code and can be validated and extended by adding source code, similar to CppUnit) and native test cases (C++test proprietary format test cases that can be set up, reviewed, and edited via GUI controls). Source test cases are implemented and saved in C or C++ source code (the language used depends on the original source being tested). You can extend and modify generated test cases with a text editor or within a supported IDE. These test cases use a format similar to the popular CppUnit format. C++test source tests offer more extensive capabilities than CppUnit, including the ability to test C code and provide programmatic access to private and protected data and member functions within the test framework. Exist- 4

ing CppUnit test cases can be imported into C++test and used in concert with automatically-generated source test cases. Native test cases are implemented in C++test proprietary format and are set up, analyzed, and modified using C++test GUI controls. Native test cases are well-suited to users who prefer to edit and extend test cases graphically (rather than programmatically). When you use native unit test cases, you can add and edit test cases using C++test's graphical test case editor and object repository. This allows you to extend your test suite without coding. You can also customize C++test's use of test case generation settings, as well as use test or stub data from external data sources. Source tests are recommended for most testing scenarios. Native tests may prove more convenient if: You are not comfortable modifying or writing C/C++ code for test cases. External data sources need to be used in the testing process, and interface APIs for them cannot be integrated in the source code of source tests. Both types of automatically-generated test cases can be validated and leveraged for functional testing, and additional test cases can be added to provide more comprehensive testing, improve test coverage, and verify functionality. Source test cases can be added by extending the automatically-generated test source code and/or importing existing CppUnit test cases. Native test cases can be added using a graphical test case editor with data source and range value support, as well as a graphical object editor. Custom stubs can be used with both types of test cases. By saving all available test cases and leveraging them for automated regression testing, you establish a regression testing infrastructure that immediately identifies unexpected functionality changes and exceptions introduced by code modifications. C++test s automated test case generation allows you to create a more effective test suite in less time. Test case development is traditionally the most time-consuming part of the unit testing process. With C++test, you do not need to write any code to generate a foundational set of test cases that exercise each class, and you can easily add more test cases in the method you prefer. Moreover, C++test s automated test case generation helps you prevent errors in two key ways: It enables you to instantly generate and execute test cases as soon as you are done writing or modifying a class. This helps you find and fix problems before you (or a team member) unwittingly introduce additional errors by adding code that builds upon or interacts with the problematic code. It enables fast, thorough exception testing by automatically generating the required range and type of test cases, which is impractical to achieve by manually designing test cases. In addition, C++test tracks test coverage to help you gauge the effectiveness of your current test suite and give you the information necessary to achieve the greatest possible coverage. Supported coverage types include: Line Coverage Cumulative Line Coverage Basic Block Coverage Branch (Decision) Coverage Path Coverage Simple Condition Coverage Modified Condition/Decision Coverage (MC/DC). Conforms to the international technical standard DO-178B (RTCA), which specifies the criteria for software certification, including real time embedded systems, for mission-critical equipment and systems within the aviation industry. Another way that C++test facilitates unit testing is by providing a robust stub functionality. Unit tests, to be effective, cannot test code in isolation. Units must be tested with external functions and elements in mind to see how they will work in real world situations. Using C++test generated or user-defined stubs, you can 5

confirm that interactions with external resources operate correctly. If the class or function under test references external resources, C++test automatically creates and uses stubs, which you can customize and extend based on your unique testing needs. Available stub types include: Original definition: The original function. User definition: A custom stub definition that you define by extending a function. Table definition: A custom stub definition that you define by entering input/outcome correlations in a table. Virtual definition: An automatically-generated stub definition for virtual methods. Safe definition: An automatically-generated "safe" replacement for a "dangerous" function (for example a system I/O routine such as rmdir(), remove(), rename(), etc.. Auto-generated definition: An automatically-generated stub definition. By automating the unit testing process, C++test makes unit testing feasible for even the most time starved developers, and affords greater accuracy and effectiveness than manual unit testing. For developers working with embedded systems, using C++test significantly reduces target system debugging time. For both types of development, embedded and non-embedded, using C++test translates to higher quality applications, less development time, as well as lower development, support, and maintenance costs. Automated Coding Standard Compliance Coding standards are language-specific programming rules that greatly reduce the probability of defects being introduced into software. Coding standards originated from industry experts, such as Scott Meyers, who analyzed the cause of each error they encountered and correlated these errors to specific coding practices; they took these correlations between errors and coding practices and designed a set of coding standards that prevent the most common and serious errors. By following these coding standards, you improve code quality, maintainability, and reusability. How C++test Automates Coding Standard Compliance C++test statically analyzes code to check compliance with specified coding standards. Coding standards in C++test are implemented as rules. C++test includes over 700 rules that implement coding standards developed by C/C++ experts such as Scott Meyers, Martin Klaus, and Herb Sutter, as well as organizations such as Ellemtel, Motorola, and MISRA. Certain rules prevent functionality, security, or performance problems; some detect possible bugs; others make code easier to maintain and reuse. Teams can also enforce specific project and organizational requirements, as well as prevent applicationspecific errors from recurring, by creating and checking custom rules. Custom rules can be created with the RuleWizard module, then checked automatically within C++test. With RuleWizard, rules are created graphically (by creating a flow-chart-like representation of the rule) or automatically (by providing code that demonstrates a sample rule violation). C++test provides preconfigured rule sets and allows users to define their own rule sets, including custom rules, to implement a coding standards policy specific to their organization. To help teams determine which rules to comply with, built-in rules are categorized by topic (for instance, security, optimization, initialization, Effective C++ [after Meyers book], and so on) as well as ranked by severity. 6

Figure 2: Configuring a Custom Rule Set Performing coding standard analysis with C++test helps development teams: 1. Detect definite or potential defects in the source. 2. Prevent the usage of code that could compromise security. 3. Follow organizational design guidelines and specifications (application-specific, use-specific, or platform-specific) and error-prevention guidelines abstracted from known specific bugs. 4. Improve code maintainability by improving class design and code organization. 5. Enhance code readability by applying common formatting, naming, and other stylistic conventions. Automating Error Prevention for Teams If some team developers are following coding standards and performing unit testing, the quality of the code in the team s shared code base will undoubtedly improve. However, by taking error prevention one step further and implementing it as part of a team-wide error prevention solution, you can also ensure that the team delivers reliable software as rapidly as possible and that the entire team works more efficiently. 7

C++test can be easily scaled to manage and standardize C/C++ error prevention across a team. The Team Configuration Manager module ensures that all team members have access to the appropriate team test settings and test files. This team support paired with the automation of key error prevention practices such as coding standards compliance and unit testing enables teams to implement and practice the Parasoft Automated Error Prevention (AEP) Methodology TM. How AEP Works The Parasoft AEP Methodology is a new methodology for improving software quality and increasing the efficiency of the software development lifecycle. As shown in Figure 3, a variety of different error prevention practices work together to stop errors at each phase of the software lifecycle. These practices include coding standard enforcement, unit testing, integration testing, load testing, and monitoring. When one of these practices exposes an error, that error is analyzed, then the process that allowed that error is modified so that it prevents the same type of error from recurring. This process creates a feedback loop where each error found serves as a catalyst for process improvement and error prevention. Figure 3: How the AEP Methodology places error prevention practices throughout the software development lifecycle Thus, the essence of AEP is the automation of the following procedures, which are combined to improve the development process and prevent software errors: 1. Detect an error 2. Isolate the cause of the error 3. Locate the point in the process that created the error 4. Implement practices to prevent the error from reoccurring 5. Monitor for improvements 8

For instance, imagine that you have an n-tier system, which includes a client, middleware, and a database. Assume that load testing revealed that the system stops working under a heavy load. After detailed analysis, you discovered a resource leak from open connections to the database. Normally, you would simply modify the code to close the connection. However, if you were to approach this situation from the perspective of AEP, you would also try to determine how to prevent the error from recurring. After isolating the problem as an open connection, you would determine that the error was introduced into the process because a developer wrote code to open but not close a connection. You might then try to stop this error from recurring by implementing a practice that ensures that code written to open a connection is always accompanied by code to close that connection. One way to implement this practice is through a coding standard that requires code that opens a connection always follow a certain template that ensures the connection is closed. In C++, the template might use constructors and destructors to open and close the connection. You could try to enforce the practice by having the team conduct code reviews. However, this is inefficient because the team would need to manually review and analyze all of the code to determine whether all of the connections were closed. A more efficient strategy is to integrate into your process a tool that automatically checks coding standards, then have that tool analyze your code and automatically identify violations of this coding standard. Your team then has a practical way to prevent leaks by ensuring that all connections are closed. This is the idea behind AEP. You found an error during load testing, then isolated the error's source as a resource leak from an open connection to the database. You found that the code was incorrect in a way that could occur throughout the code, defined a coding standard to specify how code should be written in the future, and automated the process to ensure that this standard is actually followed. In this way, the AEP feedback loop helps prevent an entire class of performance problems. The success of the AEP methodology depends on the entire team correctly performing the required error prevention practices. Parasoft has developed a recommended workflow (illustrated in Figure 4) which ensures that all team members follow the appropriate practices thoroughly and efficiently. Figure 4: The team workflow designed to support AEP 9

How C++test Supports AEP C++test helps teams implement and follow the recommended workflow. C++test automates key error prevention practices (such as coding standard checking and unit testing) required throughout the workflow so that they can be performed as painlessly as possible. In addition, the automated management of team test settings and files establishes an infrastructure that makes this team workflow feasible. In a model Parasoft AEP solution powered by C++test, every developer has a version of C++test installed on his or her local workstation. To promote the necessary uniformity and enforce team behavior, all C++test installations are configured to use the same settings. The Team Configuration Manager module ensures that all team members have access to the most recent team test settings and test files. The team architect or manager can configure the appropriate team settings on one C++test installation, then tell the Team Configuration Manager where to access the settings and related test files. Developers point their machines to the Team Configuration Manager location, and the Team Configuration Manager ensures that all developer machines access the appropriate settings and files. When the master version of a file is modified, added, or removed, the Team Configuration Manager makes the appropriate updates on all of the team s C++test installations. Developers use local installations of C++test to test their "in progress" code and ensure that it meets the team s quality guidelines before they add it to the source control system. These tests can be configured to run automatically each night, or they can be run directly from the IDE. When the developer is confident in the quality of a class, he or she adds it to the source control system. At this point, C++test automatically adds all related test case files to the source control system and the Team Configuration Manager program ensures that the files are made available to other developers as needed. To verify that all code in the source control system satisfies the team s quality standards, another instance of C++test (which is installed on one of the team s server machines) runs automated nightly tests on the code available in the source control system. The Team Configuration Manager ensures that these tests use the standard team settings and can access all necessary test assets. It also collects results, and allows developers to import results into their C++test GUIs for review and follow-up. C++test also sends information from these team-wide tests to a relational database that runs on the same server machine as the team-level C++test installation. The Group Reporting System (GRS) reporting engine then analyzes these results and provides role-based analyses to managers, architects, and developers. Team members can use this information to assess the effectiveness of the current error prevention practices and determine what additional practices would be helpful to implement. At that point, C++test can be used to implement and monitor the process improvements that the team members decided to adopt (for example, C++test could be used to create and check a new error prevention rule). 10

Learning More C++test is available now at http://www.parasoft.com. To learn more about how C++test and other Parasoft technologies can provide your team a fast and easy way to deliver reliable C/C++ applications, contact Parasoft today or visit http://www.parasoft.com. In addition, Parasoft packages customizable solutions that provide a framework for applying C++test as part of a team-wide, full-lifecycle error prevention strategy. Based on the Parasoft AEP Methodology introduced in this paper, these solutions ensure consistent and uniform behavior across development teams by providing infrastructure, technologies, configuration, and training to support full-team operation. For details on these solutions, contact Parasoft. About Parasoft Parasoft is the leading provider of innovative solutions for automated software test and analysis and the establishment of software error prevention practices as an integrated part of the software development lifecycle. Parasoft s product suite enables software development and IT organizations to significantly reduce costs and delivery delays, ensure application reliability and security and improve the quality of the software they develop and deploy through the practice of Automated Error Prevention (AEP). Parasoft has more than 10,000 clients worldwide including: Bank of America, Boeing, Cisco, Disney, Ericsson, IBM, Lehman Brothers, Lockheed, Lexis-Nexis, Sabre Holdings, SBC and Yahoo. Founded in 1987, Parasoft is headquartered in Monrovia, CA. For more information visit: http://www.parasoft.com. Contacting Parasoft USA 101 E. Huntington Drive, 2nd Floor Monrovia, CA 91016 Toll Free: (888) 305-0041 Tel: (626) 305-0041 Fax: (626) 305-3036 Email: info@parasoft.com URL: www.parasoft.com Europe France: Tel: +33 (1) 64 89 26 00 UK: Tel: + 44 (0)1923 858005 Germany: Tel: +49 7805 956 960 Email: info-europe@parasoft.com Asia Tel: +886 2 6636-8090 Email: info-psa@parasoft.com 2005 Parasoft Corporation All rights reserved. Parasoft and all Parasoft products and services listed within are trademarks or registered trademarks of Parasoft Corporation. All other products, services, and companies are trademarks, registered trademarks, or servicemarks of their respective holders in the US and/or other countries. 11