UIDE FOR NDEPENDENT OFTWARE ERIFICATION ALIDATION

Transcription

1 UIDE FOR NDEPENDENT OFTWARE ERIFICATION ALIDATION prepared by/préparé par ESA property. reference/réference ESA ISVV Guide issue/édition 2 revision/révision 0 date of issue/date d édition December 29, 2008 status/état - Document type/type de document Technical Note Distribution/distribution ESTEC Keplerlaan AZ Noordwijk - The Netherlands Tel. (31) Fax (31) ESA ISVV Guide Issue dec 2008

2 Issue 2 Revision 0 Page ii Disclaimer: This ISVV Guide is the second issue of the document. The ISVV Guide is provided as is: the European Space Agency gives no warranty nor guarantees whatsoever as to its completeness, adequacy or suitability and shall not be held liable for any direct, indirect nor consequential damages. Use of the ISVV Guide by Readers/Users is made fully at the latter's own risk. Reproduction of all or part of this ISVV Guide is authorised providing the source is referenced. Feedback: Readers/users of this guide are invited to provide comments to ESA on eventual inconsistencies in the guide or suggestions for its improvements. The comments shall be send by to: INFO-ISVV-GUIDE@ESA.INT

3 Issue 2 Revision 0 Page iii APPROVAL Title titre ESA Guide for Independent Software Verification and Validation issue issue 2 revision revision 0 author auteur Maria Hernek Sabine Krueger date date December 29, 2008 approved by approuvé by Kjeld Hjortnæs date date CHANGE LOG reason for change /raison du changement Issue/issue revision/revision date/date 2 0 December 29, 2008 CHANGE RECORD Issue: 2 Revision: 0 reason for change/raison du changement page(s)/page(s) paragraph(s)/paragraph(s) Overall improvement of document to optimise the information and introduce results from R&D and ISVV process optimisation activities. ALL

4 Issue 2 Revision 0 Page iv contents: 1.0 Introduction Background and Motivation Purpose Outline What is Independent Software Verification and Validation? Objectives of ISVV ISVV Process Overview Definition of Scope and Budgeting Non-Disclosure and Security Roles and Responsibilities ISVV supplier ISVV Customer Other roles Types of Independence ISVV Process Management Activity Overview Activity Inputs and Prerequisites ISVV level definition Documents and Code from Software Development Activity Outputs Activity Management Initiating and Terminating Events Completion Criteria Relations to other Activities Task Descriptions ISVV Process Planning ISVV Process Execution, Monitoring and Control Methods ISVV level definition Activity Overview Activity Inputs and Prerequisites Activity Outputs Activity Management Initiating and Terminating Events Completion Criteria Relations to Other Activities Task Descriptions System Level ISVV level definition Software Technical Specification ISVV level definition Software Design ISVV level definition Software Code ISVV level definition Technical Specification Analysis Activity Overview Software requirements verification Activity Inputs and Prerequisites...27

5 Issue 2 Revision 0 Page v 5.3 Activity Outputs Activity Management Initiating and Terminating Events Completion Criteria Relations to other Activities Task Descriptions Software Requirements Verification Design Analysis Activity Overview Architectural Design Independent Verification Software Detailed Design Independent Verification Software User Manual Analysis Activity Inputs and Prerequisites Activity Outputs Activity Management Initiating and Terminating Events Completion Criteria Relations to other Activities Tasks Description Architectural Design Verification Detailed Design Verification Software User Manual Verification Code Analysis Activity Overview Source Code Verification Integration and Unit Test Specification and Data Verification Activity Inputs and Prerequisites Activity Outputs Activity Management Initiating and Terminating Events Completion Criteria Relations to other Activities Tasks Description Source Code Verification Integration Test Specification and Test Data Verification Unit Test Procedures and Test Data Verification Independent Validation Activity Overview Identification of Test Cases Construction of Test Procedures Execution of Test Procedures Activity Input and Prerequisites Activity Outputs Process Management Initiating and Terminating Events Completion Criteria Relations to other Activities Task Descriptions Identification of Test Cases Construction of Test Procedures... 62

6 Issue 2 Revision 0 Page vi Execution of Test Procedures Annex A. Definitions and acronyms...65 A.1. Definitions...65 A.2. Acronyms...68 Annex B. ISVV activity outputs...70 B.1. ISVV Plan Outline...70 B.2. Requests for Clarification...73 B.3. ISVV Report (with ISVV Findings)...74 B.4. Progress Reports...74 B.5. ISVV Findings Resolution Report...74 Annex C. Review Item Discrepancy Form Example...75 Annex D. Summary of ISVV tasks, activities and methods and techniques...77 Annex E. ISVV Levels and Software Criticality Categories...83 E.1. ISVV Level definition overview...83 E.2. Error Potential Questionnaire...85 E.3. Procedures for Performing Simplified FMECA...86 Annex F. Methods...88 F.1. Formal Methods...88 F.2. Inspection...88 F.3. Modelling...89 F.4. Data Flow Analysis...91 F.5. Control Flow Analysis...91 F.6. Real-Time Properties Verification...91 F.7. Reverse Engineering...93 F.8. Simulation (Design execution)...93 F.9. Software Failure Modes, Effects and Criticality Analysis (SFMECA)...93 F.10. Static Code Analysis...94 F.11. Traceability Analysis...94 Annex G. Checklists...95 G.1. Requirements Review Checklists...95 G.2. Architectural Design Review Checklist...98 G.3. Detailed Design Review Checklist G.4. Software User Manual Review Checklist G.5. Code Inspection Checklist G.6. Unit and Integration Test Review Checklist G.7. Validation Checklist G.8. Model conformance with applicable standards Annex H. Software Validation Facility Annex I. References...111

7 Issue 2 Revision 0 Page vii figures: Figure 1 ISVV Process Activities...3 Figure 2 ISVV tasks in parallel to SW supplier review milestones...5 Figure 3 ISVV tasks after SW supplier s review milestones...6 Figure 4 ISVV process management in context...12 Figure 5 ISVV Process Management Tasks...12 Figure 6 ISVV level definition in context...17 Figure 7 ISVV level definition tasks...18 Figure 8 Technical Specification Analysis in context...25 Figure 9 Technical Specification Analysis activity...26 Figure 10 Software Requirements Independent Verification...26 Figure 11 Design Analysis in context...30 Figure 12 Software design analysis...31 Figure 13 Software Architectural Design Independent Verification...32 Figure 14 Software Detailed Design Independent Verification...33 Figure 15 Software User Manual Independent Verification...33 Figure 16 Code Analysis in context...44 Figure 17 Code Analysis...45 Figure 18 Software Source Code Independent Verification...46 Figure 19 Integration/Unit Test Procedures and Test Data Verification...47 Figure 20 Independent Validation in context...55 Figure 21 Independent Software Validation...55 Figure 22 Subtasks to "Identification of Test Cases"...56 Figure 23 Subtasks to "Construction of Test Procedures"...58 Figure 24 Subtasks to "Execution of Test Procedures"...59 tables: Table 1: Competence requirements for ISVV personnel... 9 Table 2: ISVV levels Table 3: Dependency between ISVV level, input and analysis Table 4: Test case steps Table 5: RID Form Table 6: RID Problem Type Categories Table 7: RID Severity Classes...76 Table 8: ISVV levels Table 9: Matrix to derive ISVV level from Software Criticality Category and Error Potential Table 10: Error Potential Questionnaire Table 11: Mapping from error potential score to error potential level Table 12: UML 2 diagram types... 90

8 Issue 2 Revision 0 Page viii Foreword This ISVV Guide is the result of work carried out for the European Space Agency by the contribution of different companies. Most of the material presented in this Guidebook has been based directly or indirectly on a variety of sources (ESA, European space industry experiences and projects, technical literature sources). These sources as well as the contributors are too numerous to list here, nevertheless we would like to provide acknowledgment to this invaluable contribution from all participants who through different means (direct interviews, the ISVV e- mail address, specific R&D projects, etc) supported the improvement of the ISVV process and this document.

9 Issue 2 Revision 0 Page Introduction 1.1 Background and Motivation Independent Software Verification and Validation (ISVV) is an engineering practice intended to improve quality and reduce costs of a software product. It is also intended to reduce development risks by having an organisation independent of the software developer s perform verification and validation of the specifications and code of a software product. The global objective of this guide is to aid in establishing an improved and coherent ISVV process across the European space industry in a consolidation of existing practice. Special emphasis is placed on process efficiency while the tasks are complementary to the nominal SW supplier s verification and validation tasks. It is hoped that the guide will also be found useful in other industries (e.g. Automotive, telecom, railway, etc) where software is a component of safety and dependability critical systems. The guide defines the ISVV process with management, verification, and validation activities. It provides advice on ISVV roles, responsibilities, planning, and communication as well as methods to use for the various verification and validation tasks. 1.2 Purpose The purpose of this guide is to: Define a uniform, cost effective and reproducible ISVV process across projects, and to guide in adapting it to each specific project; Assist the industry in getting predictable cost and quality out of the ISVV process; Clarify the benefits of applying ISVV; Improve ISVV project execution by highlighting the many different issues that need to be clarified and considered in the various phases of the project; Disseminate best practices with respect to recommended methods for the different verification and validation activities; Present a summary of the required capabilities of the Independent SVF in preparation of the development and utilization of a specific one for each project. The assumed readership of the ISVV Guide is primarily the customers and suppliers of ISVV services, but also software developers, system suppliers (primes) and system customers are likely to find the guide useful, be they verification / validation personnel, quality assurance managers or technical managers. The guide should be used in the preparation of a request for quotation for an ISVV service, in preparation for a bid, during planning, execution and re-planning of an ISVV project. 1.3 Outline The document consists of the following sections: The introduction of which this outline is a part describes the background and motivation for ISVV as well as the purpose of the ISVV guide. Section 2.0 elaborates on the topic of ISVV, describing types of independence, the objectives of ISVV as well as its relationship to development verification and validation. It also provides an overall view of the ISVV process. Section 3.0 describes the ISVV process management activity, detailing on ISVV roles, responsibilities, tasks and other aspects of management. Section 4.0 describes the ISVV level definition activity, and how it can be used to identify the scope of ISVV. Sections 5.0, 6.0, 7.0 describe the verification activities of Technical Specification Analysis, Design Analysis, and Code Analysis respectively. Section 8.0 describes the Independent Validation Activity. Finally, there are a number of annexes providing more detailed information related to the various ISVV activities.

10 Issue 2 Revision 0 Page What is Independent Software Verification and Validation? 2.1 Objectives of ISVV As with any verification and validation activity, the objective of ISVV is to find faults and to raise confidence in the software subject to the ISVV process. The emphasis of either of these objectives may vary, depending on the maturity of the software, budget, time, the maturity of the software supplier, as well as the distribution of responsibility between the software developer s V&V and the ISVV supplier s V&V. Raising the confidence is particularly important for critical software, whose failure may lead to hazardous events, loss of life and exceptional costs, damage to health, environmental damage, grave economic losses, or loss of reputation. ISVV is therefore usually targeted to find faults in critical and/or safety or dependability related components (including resilience recovery capabilities). This is also the main emphasis of this guide. However, in other cases, ISVV may target other quality attributes, including security, reusability, and usability. ISVV should provide added value over the verification and validation carried out by the software developer. The approach of the ISVV supplier thus has to be complementary, introduced by having different: organisational missions and values, objectives, processes, methods, tools, people. The ISVV supplier focuses on finding possible weaknesses and faults, trying to break the software, with a destructive attitude. ISVV shall aim to using methods and tools different from those of the development organisation. In some cases, one method is an alternative to another; in others, methods are complementary and not substitutable. The ISVV activities shall emphasize on the manual verification and validation activities of the SW supplier (eg. the ISVV supplier could also perform automatic code verifications using different tools than the ones used by the SW supplier). These are the ones that could be better complemented by ISVV. When performing ISVV, other supporting processes should be performed in parallel to the different ISVV tasks: e.g. Configuration Management process, the documentation process, the SPA process, the Project management process, the risk management process, the problem resolution process.

11 Issue 2 Revision 0 Page ISVV Process Overview The ISVV Process consists of 6 activities: two management activities, three verification activities, and one validation activity. The activities are shown in the figure below: MAN. Management MAN.PM.ISVV Process Management MAN.VV. ISVV level definition IVE. Independent Verification IVE.TA.Technical Specification Analysis IVE.DA.Design Analysis IVE.CA.Code Analysis IVA. Independent Validation IVA.Validation Figure 1 ISVV Process Activities ISVV Process Management (MAN.PM) is concerned with issues such as roles, responsibilities, planning, budgeting, communication, competence, confidentiality etc. It involves responsibilities of both the ISVV customer and the ISVV supplier. ISVV level definition (MAN.VV) is an activity supporting both ISVV Process Management and the Verification and Validation tasks. It provides important input for ISVV planning and how the available budget can be best utilized. The objective of the ISVV level definition task is to limit the scope and guide subsequent verification and validation activities as well as the methods proposed to be used to perform them. Technical Specification Analysis (IVE.TA) is verification of the Technical Specification, i.e. software requirements. The activity ends with a Technical Specification Analysis Review (TAR). Design Analysis (IVE.DA) is verification of the Software Architectural Design and the Software Detailed Design. The activity ends with a Design Analysis Review (DAR). Code Analysis (IVE.CA) is verification of the software source code. The activity ends with a Code Analysis Review (CAR). Validation (IVA) is testing of the software to demonstrate that the implementation meets the Technical Specification in a consistent, complete, efficient and robust way. The activity ends with an Independent Validation Review (IVR). Figure 2 and Figure 3 relate the ISVV activities to the software development processes and the review milestones defined by [ECSS-E-40B:2003]. Two different life cycle process dependencies are shown: a) when the ISVV activities are performed in parallel to the review milestones, and b) when they are performed after the review milestones. The four ISVV reviews are indicated in the figures to highlight the connection with the other milestones. The figures indicate possible early and likely start times as well as end times of the activities. More specific guidance is provided with the individual activity. Scheduling the ISVV project has a strong dependence on the progress of the software development projects. Delays in software development activities will cause corresponding delays in ISVV activities 1. 1 A scheduled date for the start of activities may be provided, with the understanding that this date may have to change if deliverables from the software development projects are late.

12 Issue 2 Revision 0 Page 4 ISVV suppliers shall follow the same life cycle as the SW supplier: e.g. supplier s iterative or incremental life cycle imply iterative or incremental ISVV tasks; or supplier s joint architectural design with detailed design, affects ISVV activities in similar way. Repetition of ISVV tasks as well as verification of new or largely modified parts of products should be carefully planned beforehand in cooperation with ISVV customer to ensure most coverage as well as highest efficiency. But not all correspondent ISVV tasks have to be performed when following de same life cycle. Examples of this might include: a) single ISVV tasks to be performed after a specific milestone, etc. This guide does not identify specific initiating events for ISVV activities. A general recommendation is that input documents and code from the software suppliers should be sufficiently mature. This is normally achieved at the delivery of review data package or after the implementation of project reviews: PDR, DDR, and CDR. The Independent Validation activity may already start with the identification of test cases as early as PDR, when stable documentation becomes available. To carry out the independent software validation testing effectively requires completion of the software development validation testing against the TS this usually finishes at CDR. Note: MAN.VV activities are drawn as VV in the figures. ISVV projects should aim at providing all ISVV findings before SW-QR, but this is to be defined per project. In some cases, documents may be mature earlier, or it may be desirable to have ISVV providing input to the development review along with other review comments. In this case, ISVV activities have to start earlier.

13 Issue 2 Revision 0 Page 5 Time ISVV Software Development Systems Engineering Requirements & Architecture Engineering VV VV TS Analysis VV AD Analysis Design and Implementation Engineering DD Analysis SUM analysis Validation Verification Delivery and acceptance VV Code Analysis Independent Validation SRR PDR DDR CDR QR TAR DAR CAR IVR Figure 2 ISVV tasks in parallel to SW supplier review milestones AR

14 Issue 2 Revision 0 Page 6 Time ISVV Software Development Systems Engineering Requirements & Architecture Engineering VV VV TS Analysis VV Design and Implementation Engineering Design Analysis Validation Verification Delivery and acceptance VV Code Analysis Independent Validation SRR PDR DDR CDR QR TAR DAR CAR IVR Figure 3 ISVV tasks after SW supplier s review milestones AR Each of the ISVV activities is described in detail in the following sections. The ISVV Process Management activity is given special treatment, but otherwise the main structure of an activity description is: Activity overview Activity inputs and prerequisites Activity outputs Activity management Initiating and terminating events Completion criteria Relations to other activities Task descriptions Methods Every activity is broken down into tasks and sometimes sub-tasks. Each task description (with the exception of project management tasks) is described in a table format with the following fields: Title: Name of the task Task ID: Task identifier Activity: Identifier and name of the activity Start Event: Start constraint for the task (might be tailored depending on the characteristics/objectives of specific ISVV projects) End Event: End constraint for the task (might be tailored depending on the characteristics/objectives of specific ISVV projects)

15 Issue 2 Revision 0 Page 7 Responsible: Identification of the organization responsible for the task execution, the ISVV supplier or the ISVV customer. Objectives: Main objectives to be accomplished by the task Inputs: Inputs to the task Sub Tasks (per ISVV Level): Task breakdown into subtasks, organised per ISVV level. Outputs: Outputs of the task A specific ISVV project may include all, one, or some of the previously referred verification and validation activities and their associated tasks and subtasks. There are dependencies between the activities; output of previous activities is often used as input for later activities 2. Users of this guide are encouraged to propose advanced techniques beyond the ones in the guide providing they can justify their performance. 2.3 Definition of Scope and Budgeting The budget for ISVV should reflect the criticality of the software to be scrutinised. The higher the criticality of the system (and the software) is, the bigger the effort for the verification and validation would be. This is the objective of the so-called ISVV level definition (see definition in Annex D), which identifies the ISVV Level of software items at various stages (software requirements, component, unit), both reducing the number of items subject to ISVV and determining which verification and validation tasks to carry out for each individual item. The ISVV costs are mainly composed by man-hour costs, travelling costs and tool costs. Manhour costs are usually dominant. Cost drivers are: Volume of specifications and code to be analysed. The ISVV Level definition is used to scope volume, so that it fits with available budget. The rigour with which an ISVV task is to be carried out. This depends on the ISVV Level. Methods to be used. Some methods are more labour intensive than other, but on the other hand they may also increase the value of the results. Repetitions in the development cycle. SW development is usually carried out in an iterative way (e.g. Incremental). Typically this means that ISVV activities have to be repeated on different versions of specifications and source code. This may have a big impact on costs, and for fixed-price contracts it is thus crucial that the number of repetitions is defined before the project starts. Complexity of the project. E.g., a project with many SW suppliers will require more management hours Travelling costs depends on whether ISVV supplier staff is required to be present at milestone reviews or not. This needs to be clarified in the Statement Of Work issued by the ISVV customer. Tools support specific methods for specific verification and validation tasks. The use of tools may greatly increase the efficiency of carrying out ISVV tasks, thereby reducing man-hours. 2 If one or more of the activities are defined to be outside the scope of the ISVV project, some of the tasks of the activity may nevertheless have to be performed for the prerequisites (in terms of required input) of other activities to be fulfilled. The dependencies are described as part of the individual activity descriptions.

16 Issue 2 Revision 0 Page Non-Disclosure and Security Spacecraft software is high value intellectual property. It is therefore important that access to documents and code (both source and executable code) is strictly controlled when handed over to the ISVV supplier 3. The ISVV supplier and other stakeholders involved in the ISVV process must fulfil requirements both with respect to non-disclosure and with respect to secure handling of information. The ISVV customer must in cooperation with the software suppliers (or the intellectual property owner) and other stakeholders (system supplier/customer) determine the confidentiality requirements, including: whether there should be different confidentiality classes for documents and what those classes are; requirements for distribution and storage of confidential documents; requirements for personnel authorised to handle confidential documents. A Non Disclosure Agreement shall be established among the ISVV supplier, the ISVV customer and the SW supplier in order to preserve the confidentiality of the data. It is not recommended that the ISVV supplier performs ISVV activities at SW supplier s site. There are several reasons for this: independence might be jeopardised, increase in travelling cost, difficulties in tool and environmental share, timeframe of analyses may be limited etc. The ISVV customer must also identify the documents required for the ISVV process and their confidentiality class. The ISVV supplier should have an information security management system in place to ensure that distribution, storage, and handling of data fulfils the confidentiality requirements (e.g. based on [BS :2002]). 2.5 Roles and Responsibilities Independent Software Verification and Validation is a service provided by an ISVV supplier to an ISVV customer. Their roles and responsibilities are described in subsequent sections ISVV supplier Independent software verification and validation requires special competence. Requirements on the competence on the individual should cover formal education, experience, as well as personal traits. There is still no consensus on what the requirements for ISVV personnel should be, but an example is provided here below: Formal education ISVV personnel should have a university degree in software engineering, computer science or similar. It will also be beneficial to have some formal background in hardware electronics and systems engineering as well as the domain itself, e.g. aerospace. Personnel should also have received proper training in quality assurance and quality control and testing. 3 The software developer will most likely require the ISVV supplier not to be involved in any kind of competing software development.

17 Issue 2 Revision 0 Page 9 Experience Personal traits ISVV personnel should have at least 5 years of working experience with software development of which at least 2 should be related to verification, validation or quality assurance. ISVV personnel should also have at last 2 years of experience within the space domain. If models are produced, the ISVV team is required to have a good knowledge of the modelling tool suite if not proficiency to fully utilise the verification and simulation capabilities of the tool suite and to correctly interpret the results. ISVV personnel should be creative destructors, rigorous, process mindful, objective, should have a critical attitude and be result-oriented. ISVV personnel should also have: - strong communication skills - be pragmatic, direct, able to simplify problems - have a critical mindset, able to explore and test, try - have generic knowledge about the domain environment and current and upcoming technologies Table 1: Competence requirements for ISVV personnel The ISVV personnel must thus exhibit creativity at breaking the system, at being destructive but respecting and considering the full scope of the mission requirement set. ISVV suppliers should also have the attitude of being the USER of the SW product, e.g.: with the operator's view focusing for example either on successfully controlling the software when recovering from errors or on how the system enters into safe mode. An ISVV project is usually carried out by an ISVV team, not a single individual. The ISVV team should compose a mix of complementary competencies. The team as such should be familiar with all methods and tools to be employed for the analyses. In addition, the ISVV team manager should be experienced with project management, including the management of ISVV projects. The project manager must also be able to handle the contractual and human relations aspects of the project, and should also have sufficient personal authority to defend the findings of the ISVV team. In addition to what is mentioned in above Table 1, for some ISVV tasks (e.g. for safety and dependability verification, or the Independent validation tasks, etc) someone at the ISVV supplier should have general space system knowledge expertise. The ISVV supplier should have a suitable quality management system (fulfilling the requirements of [ISO 9000:2000]) as well as an information management system). The ISVV process has many uncertainties (availability of inputs from the software supplier, etc) and risky elements (maturity of the elements under ISVV, maturity of the SVF, etc) that should be properly managed and controlled by the ISVV supplier through a formalised risk management process ISVV Customer The ISVV customer has the following responsibilities: Define the ISVV objectives Perform the initial ISVV level definition Define the ISVV scope and budget Approve the ISVV supplier s ISVV Plan and control its implementation Ensure NDAs are signed up between ISVV suppliers and SW suppliers Filter ISVV findings Ensure close out and implementation of ISVV findings

18 Issue 2 Revision 0 Page 10 In European space projects, the ISVV customer has traditionally been either the prime or the end customer. The prime should not be the ISVV customer if the prime itself (or any of its subsidiaries) is also developing software subject to ISVV Other roles In addition, the ISVV process may have interfaces to other roles: Software supplier (software developer) Software validation facility supplier System supplier (system integrator, prime, software customer) System customer (system owner) One of the two latter roles is also likely to be the ISVV customer Responsibilities of Software suppliers The involvement of the software supplier in ISVV includes: Assisting the ISVV customer in responding to requests for clarifications from the ISVV supplier; Assisting the ISVV customer in assessing the findings of the ISVV supplier, their criticality and resolution; Investigating and following up software problem reports resulting from ISVV findings. All communication between ISVV supplier and any of the software suppliers (when allowed) shall be copied to the ISVV customer Interface with Software Validation Facility supplier The Software Validation Facility supplier is the party providing the Software Validation Facility for the ISVV supplier s independent validation activity. The involvement of the SVF supplier could be minimal, i.e. just providing the SVF for a given period, or it could involve tasks such as specification and execution of test procedures, and reporting of test results. It is the ISVV customer s responsibility to ensure the ISVV supplier gets (or gets access to) the SVF. The recommendation of this ISVV guide is that the SVF is provided to the ISVV supplier. This secures the ISVV project s access to the SVF, also in critical phases of the project where resource contention would otherwise easily occur. 2.6 Types of Independence In the context of ISVV, independence is intended to introduce three important benefits: Separation of concerns. Any person or organisation is likely to discover that their activity inevitably produces conflicting demands and interests. Clearly separating roles and responsibilities ensures that such conflicts do not arise and also gives confidence of this to other stakeholders. A different view. All persons have a limited horizon of understanding within which texts (both written and oral) are interpreted and produced. A second opinion contributes to complement the view of the other by identifying omissions, ambiguities, factual errors, logic errors etc. Effectiveness and productivity in verification and validation activities. Staff specialised in independent software verification and validation develops technical competence and

19 Issue 2 Revision 0 Page 11 motivation that should lead to more effective and productive work. This is especially the case with verification and validation methods that necessitate the application of sophisticated tools. ISVV implies independence, additional and complementary activities from the SW supplier s verification and validation. Many standards (e.g. [IEC :1998]) distinguish between: independent person, who may belong to the same department as the writer/developer, but should not have been involved in writing the specification or the code independent department, that requires verification to be carried out by people from a different department within the same organisation, and independent organisation that must be different legal entities with different management groups and preferably different owners The higher the criticality of the system (and the software see critical item definition in Annex A.1), the more independence is required. The IEEE Standard for Software Verification and Validation [IEEE 1012:1998] distinguishes between different types of independence addressing these concerns: technical independence, ("fresh viewpoint") by an independent person, is an important method to detect subtle errors overlooked by those too close to the solution. For software tools, technical independence means that the IV&V effort uses or develops its own set of test and analysis tools separate from the developer's tools managerial independence, allowed to submit to program management the IV&V results, anomalies, and findings without any restrictions (e.g., without requiring prior approval from the development group) or adverse pressures, direct or indirect, from the development group, and financial independence preventing situations where the IV&V effort cannot complete its analysis or test or deliver timely results because funds have been diverted or adverse financial pressures or influences have been exerted. In European space industry, full technical, managerial and financial independence is required for ISVV of critical software. The ISVV supplier is required to be an organisation independent of the software supplier as well as the prime (system integrator).

20 Issue 2 Revision 0 Page ISVV Process Management 3.1 Activity Overview The objective of ISVV Process Management is to define the overall ISVV plan and to control and monitor the ISVV process. As can be seen from Figure 4 it is a management activity of the ISVV process. MAN. Management MAN.PM.ISVV Process Management MAN.VV. ISVV level definition IVE. Independent Verification IVE.TA.Technical Specification Analysis IVE.DA.Design Analysis IVE.CA.Code Analysis IVA. Independent Validation IVA.Validation Figure 4 ISVV process management in context ISVV Process Management (PM) consists of two tasks as shown in Figure 5: ISVV Process planning ISVV Process monitoring and control The figure also shows the inputs and outputs of each task. Criticality analyses (System engineering) Software Development Plan Software product Assurance Plan Software Technical ISVV Process Planning ISVV Plan Request for Clarification Software Verification and Validation Plan ISVV process management ISVV Report (with ISVV findings) Documents and code (SW development) ISVV level definition ISVV Plan ISVV Process Monitoring and Control Progress reports ISVV findings resolution report ISVV Report (with ISVV findings) Figure 5 ISVV Process Management Tasks 4 4 Note that figure shows only the most important inputs and outputs.

21 Issue 2 Revision 0 Page Activity Inputs and Prerequisites The input work products are shown in above Figure 5 defining the ISVV Process Management Tasks. There are no particular prerequisites for starting the ISVV Management Activity ISVV level definition The ISVV level definition activity (see Annex E with the definition of the ISVV levels and software criticality categories) produces critical items lists defining the scope for subsequent verification and validation activities. The activities are either carried out by the ISVV customer or the ISVV supplier. In addition to the software criticality categorization it also identifies other factors that influence the ISVV level to be assigned to the software product subject of ISVV. The scope resulting from the ISVV level definition must be reflected in the ISVV plan. The initial ISVV plan is based (among other inputs) on the initial ISVV level definition task results and updated as later ISVV level definition refines the scope Documents and Code from Software Development The main input to the ISVV project is the software documents and code to be verified and validated. Additional documentation may also be required, e.g. system requirements, system architecture, safety analyses, etc. In addition, process documents such as development standards and quality assurance procedures may also be required. Documents and code to be verified or validated should be reasonably mature and stable before being subject to ISVV activities. This normally means that they have been submitted for or been through development reviews. Earlier versions of the documents and code may be provided to the ISVV supplier for familiarisation, especially if deadlines for the actual ISVV are short. All documentation available from the software development should be delivered to the ISVV supplier when published in order to favour his knowledge acquisition, even when no ISVV activity is due to be performed (eg: RB release, etc).. Other inputs from the SW supplier to be delivered to the ISVV Customer are needed for proper management of the ISVV activities. These inputs are: SW development plans and schedules, NDAs, managerial constraints like participation or synchronization with reviews, ISVV goals, critical items lists, etc. Then also part of these information shall be transferred/filtered to the ISVV supplier by the ISVV Customer. To ensure the efficiency of ISVV activity it is important that inputs are delivered and received in electronic searchable original format. 3.3 Activity Outputs The output work products are shown in above Figure 5 defining the ISVV Process Management Tasks. 3.4 Activity Management Initiating and Terminating Events The ISVV Management activity starts for the ISVV customer when it becomes clear that a software product for which the customer is responsible (either as developer or integrator) will require ISVV. This may be at an early stage in the ISVV customer s process of bidding for the development of the software or the system containing the software. The ISVV customer starts the activity when preparing a tender package for ISVV services.

22 Issue 2 Revision 0 Page 14 ISVV Management ends with the close of the ISVV contract, i.e. with the acceptance by the ISVV customer of all deliverables required by the contract and described in the ISVV Plan Completion Criteria The ISVV Management activity completes when all ISVV activities, tasks and subtasks defined by the ISVV Plan have been carried out and all deliverables have been accepted by the ISVV customer Relations to other Activities The ISVV Management activity shall manage all of the other activities of the ISVV project. The ISVV level definition activities provide important input to the ISVV Management activity for budgeting and planning. 3.5 Task Descriptions Note that responsibility for the ISVV Management tasks is shared between the ISVV Customer and the ISVV Supplier. Responsibility is defined at the subtask level ISVV Process Planning TASK DESCRIPTION Title: ISVV Process Planning Task ID: MAN.PM.T1 Activity: ISVV Management Start event: Start of project End event: End of project Responsible: ISVV Customer and ISVV Supplier Objectives: Plan the ISVV process Inputs: - From ISVV Customer: - Criticality Analyses (from System Engineering) - ISVV level definition - Software Development Plan - Software Product Assurance Plan - Software Verification and Validation Plan - From ISVV Supplier: - ISVV level definition Sub Tasks (per ISVV Level): - MAN.PM.T1.S1: Define ISVV objectives (ISVV Customer) The main objectives of the ISVV (what quality attribute(s) are critical?) must be defined by the ISVV Customer. See also section MAN.PM.T1.S2: Perform System Level ISVV level definition (ISVV Customer) The ISVV Customer should perform an analysis to identify the need for ISVV, its scope and the initial critical items list (see section 4.5.1). - MAN.PM.T1.S3: Define the ISVV scope and determine the ISVV budget (ISVV Customer) The ISVV Customer should determine the overall ISVV budget frame based on the mission costs, size of SW product (documents and code) and the ISVV scope and level (see sections 4.1). - MAN.PM.T1.S4: Perform Technical Specification ISVV level definition (ISVV Customer or ISVV Supplier) Perform the Technical Specification analyse to identify the ISVV scope, level, and critical software requirements list. This may be carried out by the ISVV Customer or the ISVV Supplier. See also section

23 Issue 2 Revision 0 Page 15 Outputs: - MAN.PM.T1.S5: Estimate ISVV scope and budget (ISVV Supplier) The ISVV Supplier should do an independent estimation of the ISVV budget. See section MAN.PM.T1.S6: Develop ISVV plan (ISVV Supplier) The ISVV Supplier must define an ISVV plan (a draft could be part of the proposal). The plan should be approved by the ISVV Customer. The developer s software development plan, software product assurance plan, and software verification and validation plan should be taken into account if available (overall coordination planning data is to be provided by the ISVV Customer). An outline of a sample ISVV plan is found in Annex B.1. - MAN.PM.T1.S7: Approve ISVV Plan (ISVV Customer) The ISVV Customer should approve the ISVV plan developed by the ISVV Supplier. An outline of a sample ISVV plan is found in Annex B.1. - MAN.PM.T1.S8: Determine confidentiality issues and prepare NDAs (ISVV Customer) It is the responsibility of the ISVV Customer to clarify confidentiality requirements and ensure these are kept throughout the project through the signing of Non-Disclosure Agreement with the ISVV Supplier and any of its sub-contractors (see Section 2.4). - MAN.PM.T1.S9: Approve scope definition resulting from ISVV level definition (ISVV Customer) All ISVV level definition results must be approved by the ISVV Customer. See also section ISVV plan (ISVV Supplier) ISVV Process Execution, Monitoring and Control. TASK DESCRIPTION Title: ISVV Process monitoring and control Task ID: MAN.PM.T2 Activity: ISVV Management Start event: Project start End event: Project end Responsible: ISVV Customer and ISVV Supplier Objectives: Execute, monitor, and control the ISVV process Inputs: - From ISVV Customer: - ISVV level definition (from System Engineering) - Software Development Plan - Software Product Assurance Plan - Documents and Code from Software Development - ISVV Findings Resolution Report - From ISVV Supplier: - ISVV level definition [MAN.VV] - ISVV Plan [MAN.PM.T1] Sub Tasks (per ISVV Level): - MAN.PM.T2.S1: Manage ISVV project (ISVV Supplier) The ISVV Supplier must manage the project in accordance with the ISVV plan. This includes schedule management ( the dependency of SW supplier s schedule changes), budget management, resource management, activity management, risk management, quality management, document management, and security management. - MAN.PM.T2.S2: Submit documentation and code to ISVV Supplier (ISVV Customer) It is the responsibility of the ISVV Customer to provide all documentation and code necessary for ISVV planning and for the verification and validation activities to the ISVV Supplier. See also section MAN.PM.T2.S3: Check received documentation (ISVV Supplier) Any documentation and code received from the ISVV Customer or other parties of the ISVV should be registered and checked by the ISVV Supplier. - MAN.PM.T2.S4: Familiarization with software and system product under ISVV (ISVV Supplier)

24 Issue 2 Revision 0 Page 16 Outputs: The ISVV Suppliers shall familiarize themselves with the system where the SW is to be working in, supplier s development environment, the details of the SW product to be ISVV ed, the software validation facility and tools in which it will be validated, etc. - MAN.PM.T2.S5: Submit the verification and validation testing environment (ISVV Customer) 3.6 Methods Both the development environment and the validation testing environment from the SW supplier shall be available to the ISVV Supplier either delivered by the ISVV Customer or to be acquired by the ISVV supplier. - MAN.PM.T2.S6: Perform verification and validation activities (ISVV Supplier) The ISVV Supplier must carry out the verification and validation activities as described in the ISVV plan. - MAN.PM.T2.S7: Request clarifications (ISVV Supplier) The ISVV Supplier may request clarification from the ISVV Customer. See Annex B.2. - MAN.PM.T2.S8: Respond to Requests for Clarification (ISVV Customer) Whenever the ISVV Supplier issues a Request for Clarification, the ISVV Customer should provide feedback in a timely manner (see Annex B.2) - MAN.PM.T2.S9: Report early ISVV findings (ISVV Supplier) The ISVV Supplier may provide early feedback on findings to the ISVV Customer. - MAN.PM.T2.S10: Review early ISVV Findings (ISVV Customer) The ISVV Customer shall review received early ISVV findings for criticality and impact on the software/system, and shall take action as appropriate. - MAN.PM.T2.S11: Produce ISVV report (ISVV Supplier) For each ISVV activity (as defined by the ISVV plan), the ISVV Supplier must produce an ISVV report in which all of the findings shall be reported and main findings shall be highlighted. See Annex B.3.The ISVV supplier shall perform an internal review of the findings before submission. - MAN.PM.T2.S12: Filter ISVV findings (ISVV Customer) The ISVV Customer shall review and filter the ISVV findings in order to optimise their disposition and eventual implementation. - MAN.PM.T2.S13: Draft disposition of ISVV findings (SW supplier) The ISVV findings shall be replied by the SW supplier to be later discussed during a review meeting (see below). - MAN.PM.T2.S14: Conduct Review Meeting (ISVV Customer) The ISVV findings and their resolution are discussed during a review meeting (or off line discussion meeting) with participation of all related parties. The meeting is the responsibility of the ISVV Customer. - MAN.PM.T2.S15: Produce ISVV findings resolution report (ISVV Customer) In response to each ISVV report, the ISVV Customer should produce an ISVV findings resolution report, describing how each finding is resolved. The reports should be distributed to the ISVV Supplier and to the end customer (see section 3.3). in case of discrepancy between ISVV supplier and software provider it is up to the ISVV customer to determine the verdict. - MAN.PM.T2.S16: Implement resolutions (ISVV Customer) The ISVV Customer is responsible for ensuring that the resolutions described in the ISVV findings resolution report are implemented. The ISVV Supplier is not responsible for following-up the findings. - MAN.PM.T2.S17: Update ISVV level definition (ISVV Supplier) The ISVV level definition is may be updated throughout the project to further limit the scope of subsequent verification and validation activities. This is the responsibility of the ISVV Supplier, although the ISVV Customer may also be involved. See sections and Requests for Clarification (ISVV Supplier) (Optional) - ISVV Plan (ISVV supplier update) - ISVV Report (with ISVV Findings) (ISVV Supplier) - ISVV findings resolution report (ISVV Customer) - Progress Reports (ISVV Supplier) Methods used for ISVV Process Management are not different from project management methods in general and will not be further described in this document.

25 Issue 2 Revision 0 Page ISVV level definition 4.1 Activity Overview The objective of the ISVV level definition 5 task is to limit the scope and guide subsequent verification and validation activities as well as the methods proposed to be used to perform them. As can be seen from Figure 6, it is a management activity of the ISVV process. MAN. Management MAN.PM.ISVV Process Management MAN.VV. ISVV level definition IVE. Independent Verification IVE.TA.Technical Specification Analysis IVE.DA.Design Analysis IVE.CA.Code Analysis IVA. Independent Validation IVA.Validation Figure 6 ISVV level definition in context ISVV level definition (MAN.VV) consists of four tasks as shown in Figure 7 below: System level ISVV level definition Software technical specification ISVV level definition Software design ISVV level definition Software code ISVV level definition The figure also shows the inputs and outputs of each task. 5 Activity formerly called Criticality analysis but re-named now for clarity reasons to avoid confusion with Safety and Dependability analysis activities.