IBM Software Group Rational Developer for System z Unit Testing with zunit Jon Sayles / IBM - jsayles@us.ibm.com IBM Corporation
IBM Trademarks and Copyrights Copyright IBM Corporation 2013, 2014. All rights reserved. The information contained in these materials is provided for informational purposes only, and is provided AS IS without warranty of any kind, express or implied. IBM shall not be responsible for any damages arising out of the use of, or otherwise related to, these materials. Nothing contained in these materials is intended to, nor shall have the effect of, creating any warranties or representations from IBM or its suppliers or licensors, or altering the terms and conditions of the applicable license agreement governing the use of IBM software. References in these materials to IBM products, programs, or services do not imply that they will be available in all countries in which IBM operates. This information is based on current IBM product plans and strategy, which are subject to change by IBM without notice. Product release dates and/or capabilities referenced in these materials may change at any time at IBM s sole discretion based on market opportunities or other factors, and are not intended to be a commitment to future product or feature availability in any way. IBM, the IBM logo, the on-demand business logo, Rational, the Rational logo, and other IBM Rational products and services are trademarks or registered trademarks of the International Business Machines Corporation, in the United States, other countries or both. Other company, product, or service names may be trademarks or service marks of others. 2
Agenda Topics: Unit Testing Value Proposition, zunit Terms, Concepts and Vocabulary Demonstration Next Steps 3
What is Unit Testing? Unit Testing is a software testing method by which individual units of source code, sets of one or more computer program modules together with associated control data, usage procedures, and operating procedures are tested to determine if they are fit for use. Wikipedia An individual unit of software is a single test-able logic construct or routine within a call-able program: Date validation Credit Card number look-up Tax computation Co-pay calculation This method of testing is sometimes called white Box testing (See Slide Notes for more on White Box Testing) 4
Black Box Testing Black Box Testing examines the functionality of an application (what the software does) without regard to its internal semantics (how the program does what it does) Analyze Input Black Box Analyze Output Classic Example - Batch File Maintenance Testing Analyze Data Files Run File Update Program Verify Changes to Data Files QSAM DB2 or IMS VSAM QSAM DB2 or IMS VSAM 5
White Box (Unit) Testing and zunit White Box Testing is a method of testing software that tests the internal workings of an application, as opposed to its functionality or its outputs. During White Box testing, code is run with preselected input values for the validation of preselected output values. Test Harness White Box Tests Address Validation Test Initialize Street Address Initialize City Initialize State Initialize Zip Code Call Program using Evaluate test results Unit Test Software Program Date Validation Routine Internal Table Lookup Address Validation State Tax Calculation Software Program IBM has implemented White Box testing as zunit 6
zunit and the xunit Standard xunit is an industry-standard, program testing framework - that helps create and automatically run repeatable, self-checking unit test cases. xunit provides: Assertion-style test validation capabilities and result reporting. Automated execution validation - instead of independent/interactive testing activity At run-time, the xunit framework distinguishes between failures and errors: A failure is an anticipated problem (e.g., actual output does not match expected). An error is an unexpected, catastrophic problem (e.g., protection exception, null pointer) junit is an instance of the xunit architecture for Java zunit is IBM s implementation of the xunit architecture for System z zunit provides the same industry-standard output reports as junit You can use the same XML-reporting dashboards and products that you currently use with junit to review and analyze results 7
Why bother to Unit Test? By testing individual logic routines in your programs: You can move through the lifecycle more quickly, because you have precise feedback about separate logic routines So you can better understand cause & effect And you know that your code works and you know how your code works, which gives you confidence to make enhancements and modifications Because you execute zunit Tests through JCL: The testing can be automated The end-to-end process takes less time than interactive debugging. And it can be more systematic By isolating and verifying code fragments Unit Testing allows you to understand cause & effect in your program logic All of the above benefits allow you to catch errors earlier in the lifecycle 8
The value of early and extensive testing 80% of development costs are spent identifying and correcting defects ** During the Coding or Unit Testing phases $80/defect During the BUILD phase $240/defect 9 During Quality Assurance or the System Test phases $960/defect Once released into production **National Institute of Standards & Technology Source: GBS Industry standard study Defect cost derived in assuming it takes 8 hours to find, fix and repair a defect when found in code and unit test. Defect FFR cost for other phases calculated by using the multiplier on a blended rate of $80/hr. $7,600/defect + Law suits, loss of customer trust, damage to brand
What do you use to Unit Test? Unit Testing is typically done using a product or a framework that provides: 1. Standardized process, methodology and a component framework (architecture) 2. Tools for code generation or some method of simplifying the development Unit Tests 3. Automation - running scripts - or in the case of zunit - running JCL and not testing interactively The industry leadership position in the Unit Testing space is occupied by the xunit component framework xunit has been adopted by almost all of the major computing languages/vendors/tools providers Including IBM Thus zunit 10
How do you Unit Test - with zunit Using zunit you: 1. Develop Unit Tests for your programs 2. Execute the Unit Tests through JCL or interactively 3. Interpret/Analyze your Unit Test results The Unit Tests you develop are COBOL programs or PL/I procedures RDz generates 95+ percent of the COBOL and PL/I Unit Testing code RDz provides the test harness (Test Runner) as an installed run-time And RDz provides both an interactive and batch test execution workflow The zunit development process is: Wizard-driven Evolving - as customers adopt, and feedback next-generation enhancements and suggestions 11
zunit Support - as of RDz v9.1 The call-able (zunit targeted) program is a z/os Load Module that can be written in any z/os language - including but not limited to: COBOL PL/I 4GL z/os applications, generated to; COBOL or PL/I; CA-Easytrieve, SAS, IBM Report Writer, CA-Telon, NETRON/CAP, CA-Meta COBOL, etc. Assembler (see slide notes) (what this means ) You zunit test at the Load Module level: The underlying language used to create the targeted business logic is irrelevant. Note that this is differs significantly from junit and the other xunit-based language-dependent frameworks, and product offerings In RDz v9.1 - zunit supports Test Case + stub module generation, plus automated testing and result XML generation for: COBOL batch call-able subroutines - with or without QSAM/VSAM and DB2 file I/O IMS and CICS are under development 12
zunit Capabilities (High-level Feature/Function) zunit Test Runner Runs on z/os Installed and configured on z/os as part of RDz Host install and customization Fetches and runs the Test Suite referred to in a zunit configuration file zunit Wizard used to generate Test Cases RDz client feature Eclipse based wizards allow creation of: Template Test Cases are generated in COBOL or PL/I Simple pass/fail assertion API (RDz v9.1) Complete COBOL test cases: Identify the interface or set of copy book(s) Generate XML Schema to represent the interface Generate XML files where you would specify test input and expected output Generate a Test Case based on the XML file (Optionally) Generate stubs for called programs RDz viewers/editors for unit test XML results 13
zunit/unit Testing is Not a Silver Bullet IBM recognizes this - and provides multiple testing solutions, each optimized for different problem areas within the software testing space: GUI 3270 Regression Testing Performance Testing Interactive Testing Integration Testing Rational Function Tester Application Performance Analyzer RDz Integrated Debugger IBM Debug Tool Automated Unit Testing Test Driven Development zunit 14
xunit Components and Terminology All xunit frameworks share the following basic component architecture, with some varied implementation details. Test Runner Test Case Test Suite Assertions Test Fixture A test runner executes test cases or test suites and makes the results available in some programmatically-consumable way (e.g., XML,...). Test cases execute within a test fixture and make assertions during test execution. Test cases contain any number of tests where each test is a separate subprogram, function, method, or procedure in the test case. Test suites define a set of test cases for test execution. Assertions verify the behavior and state of a test case. A failed assertion should result in an exception being thrown. A test fixture is a context and/or environment that the programmer creates (setup()) before a test in a test case is executed, and discards (teardown()) immediately after the test is run. Test Execution Test Execution Test execution involves invoking the Test Runner with input that includes, among other things, which test Cases or test suites should be run. Launches Test Runner Reads from Test Suite Test Results (XML File) Which identifies Test Case There are hundreds of distributed xunit products and implementations (junit, JSUnit, NUnit, XMLUnit,etc.). They all contain code frameworks in the underlying supported language. Many but not all are TAP compliant. Which constructs Test Fixture Which executes 15 Assertions Against Your program and the program logic to be tested
zunit Component Implementation RDz substitutes generated z/os-specific resources for distributed technology classes, and frameworks. Test Runner Test Case Test Suite Assertions Test Fixture Test Execution A z/os systems program developed by the RDz team, and installed with RDz. The Test Runner orchestrates the unit testing process A generated COBOL or PL/I program compiled into a Load Module that calls your target Subroutine. Test Cases are executed (Called) by the Test Runner An XML file that provides a list of Test Cases to be executed by the Test Runner COBOL or PL/I conditions that test individual units of source code in your program A generated set of COBOL or PL/I programs that implement the xunit framework standard; for setting up, running and tearing down individual Unit Tests z/os JCL that invokes the Test Runner Test Execution //JCL Submit JCL EXEC PGM= Test Runner - RDz zunit run-time OPEN INPUT READ Test Suite Which identifies Test Case Load Module Test Suite - List of Test Cases (Load Modules) Test Case - Generated COBOL or PL/I program Test Results (XML File) The Test Runner CALLs (interacts with) Test Case Load Module Which contains Test Fixtures (COBOL subprograms or PL/I procedures Each Test Fixture sets up and CALLs (executes) individual Unit Tests and releases unit test resources (Tear Down) Test Fixture - Generated COBOL or PL/I programs Makes Assertions - Conditional Expressions coded in COBOL or PL/I Your program and the logic to be tested 16
zunit Test Case Development Wizard RDz provides a wizard-driven process to create your Test Cases and Test Fixture programs 17
zunit Generated Test Case The Test Case is a generated COBOL or PL/I module that contains: Nested programs with standard xunit names and functionality 1 M generated Test Fixtures (programs) - One program for each of the Tests defined using the wizard zunit Test Case Module USER.ZUNIT.LOAD(UNIT0001) UNIT0001(...) ADDTESTS(...) SETUP(...) TEARDOWN(...) TEST1(...)...... TESTn(...) You customize the Test Case programs, and add statements specific to your business logic and testing requirements: Initialize test input variables Evaluate test outcome RDz s zunit framework takes care of the rest 18
Agenda Topics: Unit Testing Value Proposition, zunit Terms, Concepts and Vocabulary Demonstration Next Steps 19
Demonstration What you will see: 1. Develop Unit Tests for your programs Use the zunit Automated Testing Framework to generate a Test Case Which includes Test Fixtures that contain Assertion logic that unit tests your program Compile & Link the Test Case 2. Execute the Unit Test 3. Interpret/Analyze results 20
http://www.youtube.com/watch?v=vjny7smtvqu&list=uudgvl1ysbledgitptykyjuq&index=1&feature=plcp IBM Rational Developer for System z version 8 21 21
Review - What you Saw - zunit Run-time JES Launches zunit Test Runner (Test Harness) READ Test Suite CALL (invoke) Test Case Invoke Test Fixture Submit JCL Interact with different nested COBOL subprograms or PL/I procedures within your Test Case to: - Setup unit test variable values - Run a test. - Evaluate the Assertion outcome - Release system resources Generate Results File // EXEC PGM=Test Runner //AZUCFG Test Suite //AZURES Test Results // Test Case Load Library Test Execution //AZUCFG DD DSN= Names the Test Case module(s) Test Case Test Fixture - SETUP Linkage Variables Test Suite - Run Unit Test Assertion Logic (CALL) - Evaluate outcome - Release resources z/os Test Results //AZURES DD DSN= Stores test results in XML PDS member 22 Target Subroutine Load Module to be unit tested Testable logic routine Testable logic routine Testable logic routine
Agenda Topics: Unit Testing Value Proposition, zunit Terms, Concepts and Vocabulary Demonstration Next Steps 23
Review - zunit Test Case Development Wizard RDz provides a wizard-driven process to create your Test Cases and Test Fixture programs 24
Review - xunit Vocabulary / zunit Implementation Test Execution: JCL that invokes the Test Runner (a supervisor program or test harness that runs your Test Suite) Test Runner: The zunit supervisor program invoked by Test Execution, that: Reads a Test Configuration Test Suite Calls your Test Cases Test Suite: an XML configuration file that defines which Test Cases the Test Runner should execute (Note - a Test Case == a Load Module) Test Case: An RDz-generated COBOL or PL/I program that: Is called by the Test Runner Uses a Test Fixture to Call your Subroutine and write test outcomes to a Test Results file Test Fixture: A set of generated programs nested inside your Test Case. Test Fixtures run Tests that utilize Assertion logic to Call your Subroutine, and evaluate the result. Subroutine: The target call-able program to be Unit Tested Test Results: An output dataset that describes the results of each test run in xunit standard (XML) file format Assertion: A simple pass/fail predicate used to test one part of your Subroutine s logic ** Test Execution, Test Runner, Test Suite, Test Case, Test Fixture, Assertion == xunit vocabulary 25
26
Links - to additional Unit Testing content http://lisacrispin.com/wp-content/uploads/2011/11/agile-testing-quadrants.png http://1.bp.blogspot.com/-rqltptssy_o/uz9cyqzflqi/aaaaaaaaayo/9kix6agwsau/s1600/testingtrianglepished.png http://en.wikipedia.org/wiki/white-box_testing http://en.wikipedia.org/wiki/black-box_testing http://en.wikipedia.org/wiki/xunit http://en.wikipedia.org/wiki/test-driven_development 27