Software Verification and Validation

Transcription

1 Software Verification and Validation Georgia L. Harris Carol Hockert NIST Office of Weights and Measures 1 Learning Objectives After this session, using resources and references provided, you will be able to List examples of software error impact Identify which criteria in ISO/IEC address requirements for computer systems Identify examples of software errors and non-conformities 2 1

2 Software Verification & Validation What and Why 3 What is Verification and Validation? Validation is ensuring "you built the right product (SUITABLE) Validation is testing to confirm that it satisfies stakeholder needs. Examples for metrology: Correct SOP selected, Correct equations, Results are good Verification is ensuring "you built the product right." (ACCURATE) Verification is testing to confirm that a product complies with its requirements and specifications. Examples for metrology: Equations match the SOP, Calculations produce the expected answer 4 2

3 Why do Verification and Validation? Ensure measurement result accuracy Minimize impact of software errors Ensure records are maintained as evidence Meet Requirements Accreditation to ISO/IEC FDA all software (including use of spreadsheets) must follow good software engineering practices 5 Health Impacts: U.S. Food & Drug Admin. The FDA s analysis of 3140 medical device recalls conducted between 1992 and 1998 reveals that 242 of them (7.7%) are attributable to software failures. Of those software related recalls, 192 (or 79%) were caused by software defects that were introduced when changes were made to the software after its initial production and distribution. Software validation and other related good software engineering practices discussed in this guidance are a principal means of avoiding such defects and resultant recalls. NIST OWM has seen failures in proficiency tests due to lack of adequate software control and validation on a regular basis. Guidance for Industry and FDA Staff General Principles of Software Validation,

4 Financial Impacts: U.S. NASA Automated Software Verification & Validation: An Emerging Approach for Ground Operations, David G. Bell and Guillaume P. Brat, NASA Ames Research Center, 2008 IEEE. 7 Does it affect you? 8 4

5 Does it Affect You? 9 Requirements 10 5

6 17025:2005 Sections to Consider c Procedures exist to protect client s information Procedures to control software Quality system reviewed and approved by authorized personnel (electronic signatures) Authorized editions of appropriate documents all locations. (Intranet, NT file Share) Altered or new text shall be identified (electronic document) Procedures shall describe how changes in documents, including software are controlled Records (electronic media) shall be stored and maintained so that they are retrievable Procedures to protect and back-up electronic records Retain records for the retention period (old versions of software also) Observations shall be recorded at the time they are made. (electronic) Electronic records shall avoid loss to original data (audit trails) Lab shall have instructions on the use and operation of equipment (and software) Calculations (spreadsheet) and data transfers (tables) shall be subject to checks a Software shall be validated a Laboratory configurations of COTS software shall be validated b Procedures are established to protect data c Computer and automated equipment are maintained Equipment & Software shall comply with specifications Each item of equipment & software shall be uniquely identified Records shall be maintained of equipment & software When correction factors are used, procedures shall ensure software is updated Software shall be safeguarded from adjustments Reports may be issued electronically j Reports may contain electronic signatures Reports may be transmitted electronically. 11 Laboratory Computer Systems Assessment Internal Audit Quality and Technical 12 6

7 Requirements, References and Records Handbook Requirement ) or c. have policies and procedures to ensure the protection of its customers confidential information and proprietary rights, including procedures for protecting the electronic storage and transmission of results; Laboratory Documentation References QM Section 4.2 SAP 10 Title SAP 4 Title Form 10, Title Evidence of Compliance Policy is in place; audit shows that it is being followed; Reviewed 3 copies of Form 10 to see they were completely filled out. (See Attachment A). No information has been inadvertently released 13 ISO/IEC 17025: Key Concepts for Software Documentation Quality management system; Standard Admin. Procedure 10 Procedures for Verification & Validation Checklist = Form A Validation Process & Records Data sets Correct Updates Calibration data (values & uncertainties) Controls & Maintenance Inventory, status Security, locked cells, passwords Approvals It s not just about data sets! 14 7

8 Laboratory Document Control and Approval 17025:2005, Section 4.3 Document Control Procedures for development, V&V, and approval Inventory of Software (Master List) Technical assessment Records of V&V 15 Document Control: Inventory & Tracking Sample File available: trology/lab-resources.cfm 16 8

9 Quick Quiz:? How many major sections of ISO/IEC cover Software Verification and Validation? What is objective evidence? 17 Software Verification & Validation Administration and Review Process 18 9

10 Focus: Spreadsheet Verification & Validation Software Engineering Life Cycle Specs, Design, Build, Test, V&V, Use + Documentation Requirements (17025, FDA) Software QA (SAP 10) 17025:2005 Auditing Forms Spreadsheet Verification & Validation Software Inventory Form A 19 Standard Administrative Procedure 10 (SAP 10) Responsibility and Authority Software Engineering Essentials FDA RP 13 Risk Analysis SSFM NPL Best Practice Guide Methods for Controlling and Evaluating Software Inventory SAP 10, Form A 20 10

11 Software Life-Cycle Requirements Consider RISK Assessments Change Control Process Figure 1. Software Life Cycle Design Construction Purchase Testing Documentation Retirement Installation & Validation Operations & Maintenance Time 21 Risk Assessment For additional information on Risk Analysis practices and procedures, see Validation of software in measurement systems (Software for Metrology Best Practice Guide No. 1), National Physical Laboratory (NPL), January Spreadsheets may not be an appropriate solution

12 Testing Phase (SAP 10 has more ) 1. Analysis without computer assistance 2. Other validated computer program 3. Experiments & tests 4. Standard problems with known solutions (data sets) 5. Confirmed published data and correlations 23 Installation & Acceptance 1. Verify complete installation 2. Ensure correct operation within system Operating System updates, patches, software updates and potential changes 3. Document approval for use Who has authority for review and implementing new system 24 12

13 Operations & Maintenance Phase 1. Fix remove errors 2. Improve changed specifications; process improvements 3. Adapt operating environment 25 Software Verification & Validation Techniques and Tools for Testing 26 13

14 Verification & Validation Techniques SAP 10 Descriptions Form A Sample Items What is evidence? 27 SAP 10: Descriptions Form A: Review and Evidence A.- Software Inspection B.- Mathematical Specification C.- Code Review D.- Numerical Stability E.- Component Testing F.- Numerical Reference Results G.- Embedded Data Evaluation H.- Back-to-Back Testing I.- Analysis Without Computer Assistance J.- Security 28 14

15 Example & Demonstration 29 References & Guides ISO/IEC 17025:2005 Laboratory Standard Administrative Procedure 10 and Form A NPL Guides, March 2004 Software Support for Metrology Best Practice Guide No. 1, Validation of Software in Measurement Systems, Includes Risk Assessment methods. Software Support for Metrology Best Practice Guide No. 7 Development and Testing of Spreadsheet Applications, see especially section 6, Checklists for spreadsheet development and testing General Principles of Software Validation; Final Guidance for Industry and FDA Staff (FDA, January 11, 2002) NCSLI Recommended Practice 13, Computer Systems in Metrology (February 1996) 30 15

16 Thank you!