Quality Assurance Program

Transcription

1 CLEAR BALLOT GROUP ClearVote 1.0 Quality Assurance Program Abstract: This document outlines the general quality assurance policies followed by Clear Ballot Group to test the ClearVote product Clear Ballot Group

2 ClearVote Quality Assurance Program Part Number: Copyright , Clear Ballot Group All rights reserved. This document contains proprietary and confidential information, consisting of trade secrets of a technical and/or commercial nature. The recipient may not share, copy, or reproduce its contents without express written permission from Clear Ballot Group. Ballot Resolver, Clear Ballot Group, ClearAccess, ClearAudit, ClearCount, ClearData, Clear Design, ClearVote, Image-to-Ballot Traceability, MatchPoint, ScanServer, ScanStation, Speed Accuracy Transparency, Visual Verification, Visualization of Voter Intent, and Vote Visualization are trademarks of Clear Ballot Group. ScandAll PRO is a trademark of FUJITSU LIMITED. All rights reserved. Other product and company names mentioned herein are the property of their respective owners. 7 Water Street, Suite 700 Boston, MA (857) Document history Date Description Version Authors 08/10/2015 Initial release. 1.0 Ed Smith Iris Friedman 08/20/2015 Minor corrections Ed Smith Iris Friedman ClearVote Quality Assurance Program 2

3 Table of contents Chapter 1. Abstract About this document Scope of the Quality Assurance program at Clear Ballot Group Items in scope of the Clear Ballot Group quality program Items outside scope Intended audience Reference documents 7 Chapter 2. ClearCount system ClearCount requirements Documentation of quality procedures Quality responsibilities Tests and verification Documentation feedback and process review 10 Chapter 3. COTS hardware: laptop computers and scanners Hardware specification review Scanner review and requirements Laptop review and requirements Router or switch requirements Hardware responsibilities Test plans and special tests Scanner tests Laptop tests Future development and COTS integration 13 Chapter 4. Integration with third-party EMS PDF review and creation of the BDF BDFs and election creation BDF testing and validation 14 ClearVote Quality Assurance Program 3

4 4.4 Integration responsibilities 14 Appendix A. Test plans 16 Appendix B. Test case reports 17 ClearVote Quality Assurance Program 4

5 1.1 About this document Chapter 1. Abstract This section defines the general description of the Quality Assurance Program at Clear Ballot Group. It contains the following sections: About this document Scope of the QA Program History and intended audience 1.1 About this document This document outlines the general quality assurance policies that Clear Ballot Group follows in producing the ClearVote system. ClearDesign is the first ballot design, proofing, and production system built from the ground up using the latest technological advances in a generation. ClearDesign makes it intuitive and simple to set up and manage elections and ballots. ClearDesign has a browser-based user interface that is consistent, streamlined, and easy to use. Election department staff can quickly create elections and then generate, modify, and proof all their ballot styles. In addition, each jurisdiction can customize the terminology used in ClearDesign to match what you they, rather than forcing their teams to learn new terminology. ClearAccess is an accessible voting and ballot marking application that allows voters with sight or mobility limitations to vote in an unassisted manner. The software runs on a touchscreen computer installed with an EZ Access keypad and Origin Sip-and-Puff device. The voter may make ballot selections either by touching the monitor, pressing buttons on the keypad, or using the Sip-and-Puff. Ballot selections may be presented on the computer's display, played back over audio headphones, or presented both on the screen and in audio format. Once the voter has finished voting, selections are printed to a paper ballot. From its conception, ClearCount was designed to inspire confidence in the user. The vote and ballot visualizations that allow marginal votes to be identified and corrected by authorized officials without requiring extensive retabulation provides a new level of visibility and transparency to reviewing election results. However, those new features would not be valuable unless they were embedded in a program that had been designed, analyzed, and verified to include the highest possible level of quality. ClearAudit, the audit product from Clear Ballot Group, has been used in the field, and feedback from election officials and Clear Ballot Group staff has been incorporated into ClearVote when applicable for a central count system. ClearVote Quality Assurance Program 5

6 Chapter 1. Abstract The Clear Ballot Group validates that ClearVote is designed and developed so that it meets functional requirements for a central-count system and is suitable for election purposes. 1.2 Scope of the Quality Assurance program at Clear Ballot Group ClearVote combines proprietary software for ballot design, tabulation, accessible voting sessions, and results reporting with COTS hardware (scanners and laptop or desktop computers). There are many aspects to the quality program at Clear Ballot Group Items in scope of the Clear Ballot Group quality program The major areas of the Quality Assurance rogram at Clear Ballot Group are as follows: The requirements definition, design, and process documentation of the ClearVote software program. Determining the specifications that COTS hardware must meets to provide the level of quality (workmanship) necessary for a central count system. The requirements and process for integrating other EMS information into ClearCount (primarily creating and validating the Ballot Definition File (BDF)). The requirements and process for ballot layout in ClearDesign. The requirements and process for integrating other EMS information into the ClearVote system (primarily creating and validating the Ballot Definition File (BDF)). The validation and verification of the performance, and functioning and integration of the voting system, which consists of COTS hardware, BDFs (adapted from PDFs from other vendors' EMS systems), and internally developed software. Validation and verification of the process for installation of ClearCount and ClearAccess on COTS hardware along with the necessary steps to secure the system as would be required in a jurisdiction. ClearVote Quality Assurance Program 6

7 1.3 Intended audience Items outside scope The following items are out of scope of the Clear Ballot Quality Assurance Program: Hardware development, manufacture, and testing. Pre-delivery tests or maintenance records for third-party hardware. Development of third-party Election Management Systems. Information on the hardware manufacturers' quality programs can be found in their documentation. See the Appendix to the Technical Data Package Summary for the list of documents provided from Fujitsu (scanners), Toshiba (laptop computers), and Hewlett Packard (laptop computers). 1.3 Intended audience Clear Ballot Group s Quality Assurance Program has been evolving since the first Quality Engineer was hired in early The company s internal wiki and bugtracking system (Trac) were the first repositories of QA processes and progress. The Quality Assurance Program has been continually improved, and this document has been created and refined. This document is intended for state election officials and their delegated Voting Systems Test Laboratory, as part of the TDP required to certify the ClearVote system for use in their state. 1.4 Reference documents Table 1-1. Reference documents Document Software Design and Specification Election Preparation and Installation Guide Description Technical details on all components of ClearVote software, including coding standards, software items, databases, interfaces, and URLs. Administrative, set up, and troubleshooting issues. An appendix contains a list of error messages for all ClearCount components. Election Administrator's Guide System Functionality Description Describes requirements for all ClearVote functionality, including the BDF. ClearVote Quality Assurance Program 7

8 Chapter 1. Abstract Document System Hardware Specification Configuration Management Plan System Security Specification Election Database Specification Test and Verification Specification Approved Parts List Technical Data Package Summary Description Describes requirements and validation of COTS hardware used for the ClearVote system. Describes the configuration management for the ClearVote system; also lists all third party packages used. Describes the databases and their intersection with user roles and permissions. Details each database table, the characteristics of the data within the table, and the source of the data. Describes the Quality Assurance and testing processes at Clear Ballot Group Lists all components of the ClearVote system and the tested hardware included in the Technical Data Package. Lists all the documents included in the Technical Data Package. ClearVote Quality Assurance Program 8

9 2.1 ClearCount requirements Chapter 2. ClearCount system ClearCount performs tabulation and reporting functions within the ClearVote system.because these functions are foundational, this section describes ClearCount in greater detail. ClearCount system consists of several components: ScanServer software Scripts and executables run from the ScanStation, but located on the ScanServer Tabulator (runs on the ScanStation) Election Administration reports and other browser-based tools, including the Election Administration tool, User Administration tool, and the Ballot Resolver ClearCount software makes use of some third-party open source software (such as Python, Ubuntu, MySQL, JavaScript). The full list is found in ClearVote Configuration Management Plan. In general, the quality verification process is the same for all components. Exceptions are noted in the following sections. 2.1 ClearCount requirements The ClearCount System Functionality Description provides the general functional requirements for the ClearCount system. The VVSG 1.0 requirements in regards to accuracy (Volume 2, Appendix C.5) were also adapted when designing and performing stress tests. ClearAudit field tests, discussions with election officials, and research into election industry best practices, also informed the process. VVSG 1.0 requirements for security, accuracy, error recovery, and integrity as described in Volume 1, Section 2.1 were taken into account in all cases. 2.2 Documentation of quality procedures The Trac system, Clear Ballot Group s internal ticketing system, has been the system of record for not only functional problems but also for design issues, enhancement requests, and tasks. The accompanying internal wiki contains information on functional and process standards. As the TDP documentation took shape, information from the Trac tickets and wiki were incorporated into the System Functionality Description, the Configuration Management Plan, and the other documents listed in Reference documents. ClearVote Quality Assurance Program 9

10 Chapter 2. ClearCount system As the Test and Verification Specification details, this information is also incorporated into a series of test plans and test case reports. Each minor or major new build goes through significant regression testing to verify its acceptability. Bug fix builds have bug fixes verified by QA. The QA staff also performs smoke testing based on which components changed. 2.3 Quality responsibilities The CEO, CTO, Director of Product Management, and the Quality Engineers are all responsible for the definition and enforcement of quality in the ClearVote system. 2.4 Tests and verification The Test and Verification Specification lists all the test plans (describing the test approach) and test case reports (describing the test steps) for the ClearVote system. They are also listed in the appendices to this document. Known issues present in the ClearVote build submitted for certification are listed in the Test and Verification Specification. 2.5 Documentation feedback and process review The assembly of the TDP documentation and corresponding testing and analysis of the Clear Ballot software identifies areas where the Quality Process requires review. The Director of Product Management convenes a series of meetings to institute additional levels of feedback and peer review into the Quality Assurance Program. These meetings recur periodically, and updated processes are documented on the Clear Ballot Group internal wiki page as well in subsequent versions of this document. ClearVote Quality Assurance Program 10

11 3.1 Hardware specification review Chapter 3. COTS hardware: laptop computers and scanners As described in ClearVote System Hardware Specification, system hardware components undergo extensive validation and verification. 3.1 Hardware specification review Scanners must meet stringent requirement for reliability, ease of use, availability of support, and durability. Clear Ballot staff reviewed the specifications for all proposed hardware and conducted in-depth discussions with Fujitsu, the manufacturer of the scanners Scanner review and requirements The major scanner requirements are: Scanners must be able to accept all ballot weights and paper sizes required. The scanner must produce acceptable quality images (200 or 300 dpi) as JPGs for correct analysis by the Tabulator. Scanner settings such as contrast must be configurable. The scanner software must be able to rename the images when provided with the target card bar code and to deposit the scanned images into a designated folder. Scanner performance must be acceptable for extended periods of time of continuous use. Scanner performance must degrade only slightly (and ideally not at all) when scanning folded ballots. Accuracy of results must not be affected. Scanners must have sensitive and accurate capabilities for detecting size and width irregularities, multifeeds, and so on. Recovery must be clear and immediate. Since the ClearVote system is designed to be scalable depending on the size of the jurisdiction, certifying different scanner models at different price points was also a requirement Laptop review and requirements ClearVote Quality Assurance Program 11

12 Chapter 3. COTS hardware: laptop computers and scanners As described in the ClearCount Approved Parts List, the requirements for the laptop computers used involve the operating system, CPU, and memory. Different models are therefore selected for the ScanServer and for the ScanStation and Election Administration Station clients. The servers and clients were used in an extended series of in-house and field tests, including a million ballot performance and stress test that exercised each laptop for approximately ten hours as it tabulated the ballot images for 100,000 ballots (ten ScanStations were used in the test). ClearDesign and ClearAccess also utilize laptop computers. ClearAccess also utilizes all-in-one form factor computers. Selection and test of these computers follow the same process as used with ClearCount computers, with specific ClearDesign and ClearAccess requirements added to the selection considerations Router or switch requirements The router or switch used to connect the systems in the closed network of the ClearVote system must be gigabit router with a minimum of 4 ports. 3.2 Hardware responsibilities The Director of Product Management bears primary responsibility for the selection of the hardware, with input and assistance from the CTO and CEO. The Quality Assurance staff assisted with the testing that verified the hardware would meet our needs. 3.3 Test plans and special tests Scanner tests The Test Case Report Scanner Behavior contains an extensive suite of test cases that were performed on each scanner model being submitted for certification (Fujitsu fi-7180, fi-6670, and fi-6800). The Test Case Report System and Tabulation Regression contains a set of integration tests performed with larger test decks Laptop tests The Laptop Acceptance Test Plan provides information and references to relevant laptop tests, including installation, performance, and system tests. ClearVote Quality Assurance Program 12

13 3.4 Future development and COTS integration 3.4 Future development and COTS integration Clear Ballot Group continues to work with Fujitsu to guarantee that the functions required for the ClearVote are included in future scanner models, including them in the scanner configuration software. Clear Ballot Group staff are committed to making ClearVote available on the latest models of COTS laptop computers with the latest operating systems and continue to test (and submit for certification as appropriate) as new versions of Microsoft Windows become available. ClearVote Quality Assurance Program 13

14 Chapter 4. Integration with third-party EMS Chapter 4. Integration with third-party EMS The Software Design and Specification describes the Ballot Definition File creation process. This is a customized process based on each election s specifics, including vendor, ballot size, number of contests, choices per contest, languages on the ballot, and so on. The Election Database Specification spreadsheet contains information on the contents of the BDF and how it corresponds to the database tables in the election database. The following process does not apply to BDFs created through ClearDesign. 4.1 PDF review and creation of the BDF The development staff at Clear Ballot Group is responsible for analyzing the ballot PDFs sent from the jurisdiction and translating them into the BDF zip file. Each ballot style and precinct must be analyzed and the contests and choices translated into a set of virtual coordinates required for tabulation. Additional information, such as ballot size, may be required from the jurisdiction if the information cannot be interpreted from the PDFs. 4.2 BDFs and election creation A BDF is required for creating an election database that contains tabulated ballot data. If the BDF does not contain all the required tables, the election database is not created. Any test of the system that involves tabulation requires a valid BDF. 4.3 BDF testing and validation Any given BDF is tested through election creation and tabulation. In addition, some UI tests compare the contents of reports to the data in the BDF. Spot-checking of the ballot or ballot image content against some of the UI reports, such as the Contests report, serves as a validation that the BDF content is correct. 4.4 Integration responsibilities The development team at Clear Ballot Group is responsible for the testing and direct verification of the BDF creation process and the validation of the BDF content. ClearVote Quality Assurance Program 14

15 4.4 Integration responsibilities The QA staff validates the BDFs through election creation and the review of tabulation results and UI reports. ClearVote Quality Assurance Program 15

16 Appendix A. Test plans Appendix A. Test plans The following test plans describe the test scope and approach for various components or functions of the ClearVote system. See the Test and Verification Specification for more information. Laptop Acceptance Test Plan Scanner Acceptance Test Plan System Test Plan System Error Handling Test Plan Performance and Stress Test Plan UI Reports Test Plan Ballot Adjudication and Resolution Test Plan Tabulator Functional Test Plan Installation and Hardening Test Plan User Access and Election Admin Test Plan ClearVote Quality Assurance Program 16

17 Appendix B. Test case reports The test case reports contain the test steps for each ClearCount test scenario. Test instructions and a tab to place test data are also included in each spreadsheet. See the corresponding test plan document (see Appendix A) for more information about the test approach for that system function or component. Test case reports include the following. See the Test and Verification Specification for more information. The test case report for UI 1007 is deliberately excluded; the report was removed from the ClearCount UI. Test Case Report scanner behavior Test Case Report installation and hardening Test Case Report Performance and stress testing Test Case Report system and tabulation regression Test Case Report system error handling Test Case Report Tabulator Regression tests Test Case Report Tabulator error handling Test Case Report users election admin and access levels Test Case Report Ballot Adjudication and Resolution Test Case Report UI 1001 Dashboard Test Case Report UI 1002 Statement of Ballots Cast Test Case Report UI 1003 Statement of Votes Cast Test Case Report UI 1004 Contests Report Test Case Report UI 1005 Statement of Votes Cast with Precincts Test Case Report UI 1006 Vote Visualization Test Case Report UI 1008 Election Activity Log Test Case Report UI 1009 Causes of Unreadable Ballots Test Case Report UI 1010 Ballot Images Test Case Report UI 1011 ScanStation Report Test Case Report UI 1012 Ballot Information Page Test Case Report UI 1013 Annotated Ballot Page Test Case Report UI 1014 Box Report Test Case Report UI 1015 Precinct report ClearVote Quality Assurance Program 17

18 Appendix B. Test case reports Test Case Report UI 1016 Ballot Locator Test Case Report UI 1017 Human Resolutions Test Case Report UI 1018 Ballot Styles Test Case Report UI 1019 Counter Group Statement of Ballots Cast with Counter Groups ClearVote Quality Assurance Program 18