Requirements Based Functional Testing Traci Mapps and Mike Santos SLI Global Solutions State Certification Testing of Voting Systems National Conference June 4-5, 2014 1
SLI Overview Accredited by the US National Institute of Standards and Technology as a Voting System Test Laboratory Accredited by the United States Election Assistance Commission as a Voting System Test Laboratory In depth experience with most voting products used throughout the United States State, Federal and International certification testing experience Established expertise and knowledge in testing election systems to certification standards and statutory requirements 2
What is Requirements Based Functional Testing? A testing approach in which test suites/modules are derived from requirements. It includes functional tests and also non-functional attributes such as performance, reliability and/or usability. Considered to be an effective testing approach, used to significantly reduce the number of undetected discrepancies being released into production. A testing approach which enables the determination of a necessary and sufficient set of tests which verify that the design and code of a system meets stated requirements. 3
What is Functional Test? sliglobalsolutions.com Functional Testing verifies that the system behaves correctly and functions according to the requirements which specify the intended use of the application. Functional Testing determines if each component: o Performs in accordance to the specifications o Responds correctly to all conditions that may be presented by incoming events/data o Moves data correctly from one business event to the next o Initiates events in the order required to meet the business objectives of the system. 4
What is a Requirement? sliglobalsolutions.com A requirement is a capability or function that must be delivered by a system component or components. A single documented functional need that a particular design, product or process must be able to perform. It is a statement that identifies a necessary attribute, capability, characteristic or quality of a system for it to have value and utility to a customer. 5
Characteristics of a Good Requirement Completeness: The requirement is complete to the extent that all of its parts are present and each party is fully developed. For example, no TBDs, no reference to nonexistent functions, inputs or outputs. Consistency: The requirement is consistent to the extent that its provisions do not conflict with each other or with governing specification and objectives. Suitability: The requirement is appropriate or suitable for testing voting systems. Feasibility: The requirement is feasible from a cost and schedule standpoint. Also can a system truly meet the requirement. Testable: The requirement is testable to the extent that one can identify an economically feasible technique for determining whether or not the requirement has been met. Requirements must be specific, unambiguous, and quantitative wherever possible. 6
Characteristics of a Good Requirement Unambiguous: A requirement is unambiguous only if every portion of requirement stated therein has only one interpretation Bounded: The boundaries scope and context for the requirement should be identified Normalized: Requirements should not overlap nor refer to other requirements or the capabilities of other requirements Unique: Each requirement should be stated only once Correct: A Requirement is correct only if every portion of requirement stated therein is one that the software shall meet 7
The Good, the Bad and the Ugly Bad Example The system must be user friendly. How do you measure user friendliness? Good Example The user interface shall be menu driven. It shall provide dialog boxes, help screens, radio buttons, dropdown list boxes and spin buttons for user inputs 8
Test Cycle 9
Test Cycle Requirements Definition Requirements-based tests can be no better than the requirements they are testing. Requirements must be correct, complete, unambiguous and logically consistent. Functional Decomposition/Partitioning Functional decomposition/partitioning is the breakdown of a system into its functional components or functional areas and assignment into a given partition for functional testing. 10
Test Cycle Test Planning Define the scope, schedule and deliverables Test Suite/Module Design and Validation Write test suites and modules to validate the product performs in accordance to the requirements. 11
Test Cycle Traceability (Traceability Matrix) Document the links between the requirements and the work products developed to implement and verify those requirements. Test Suite/Module Execution Execution of the test suites/modules against the system being tested and documentation of the results defined in the test plan. 12
Test Cycle Defect Tracking Defects detected during the testing process are written against the pertinent requirement(s) and tracked to resolution. Coverage Analysis Determine the coverage of the requirements against test suites/modules and outstanding defects. Determine the percentage of the requirements that are untested, performing to specification and not performing to specification (defects or discrepancies). 13
Why Requirement Based Functional Testing? Industry research shows that the root cause of 56 percent of all errors identified in projects are introduced in the requirements phase. Implementing formal requirements can reduce risk. Requirements based testing can increase efficiency, reduce the risk of failure and improve overall quality. From a Certification point of view, requirements based testing creates a consistent baseline from which different voting systems can be consistently evaluated to the same criteria. Complete traceability between requirements, test suites/modules, and defects provides comprehensive information to help ensure complete test coverage and suitability for certification. The result is an effective and auditable base for certification testing. 14
Questions 15