EUROPEAN MIDDLEWARE INITIATIVE

Transcription

1 EUROPEAN MIDDLEWARE INITIATIVE DSA2.1 SOFTW ARE QU ALITY ASSUR ANCE PL AN EU DELIVERABLE: DSA2.1 Document identifier: EMI-DSA QA_Plan-v1.2.doc Date: 31/05//2010 Activity: Lead Partner: Document status: Document link: SA2 Quality Assurance CERN Final Abstract: This deliverable contains the definition of the global software QA processes, procedures, roles and responsibilities and the related metrics and measurement methodologies for the EMI (European Middleware Initiative) project. What described in this document applies to the software teams in their development, release and support activities. INFSO-RI Members of EMI collaboration PUBLIC 1 / 47

2 Copyright notice: Copyright (c) Members of the EMI Collaboration See for details on the copyright holders. EMI ( European Middleware Initiative ) is a project partially funded by the European Commission. For more information on the project, its partners and contributors please see This document is released under the Open Access license. You are permitted to copy and distribute verbatim copies of this document containing this copyright notice, but modifying this document is not allowed. You are permitted to copy this document in whole or in part into other documents if you attach the following reference to the copied elements: "Copyright (C) Members of the EMI Collaboration. ". The information contained in this document represents the views of EMI as of the date they are published. EMI does not guarantee that any information contained herein is error-free, or up to date. EMI MAKES NO WARRANTIES, EXPRESS, IMPLIED, OR STATUTORY, BY PUBLISHING THIS DOCUMENT. INFSO-RI Members of EMI collaboration PUBLIC 2 / 47

3 Delivery Slip Name Partner / Activity Date Signature From Maria Alandes Pradillo CERN/SA2 18/07/2010 Reviewed by Giuseppe Fiameni, Andrea Ceccanti CINECA/SA1, INFN/JRA1 08/08/2010 Approved by PEB 23/08/2010 Document Log Issue Date Comment Author / Partner First draft of the document. Missing Procedures. Maria Alandes Pradillo Added Procedures. Maria Alandes Pradillo New version after phone conference feedback. Maria Alandes Pradillo Added Metrics and Quality Factors. Maria Alandes Pradillo Added comments from official reviewers. Maria Alandes Pradillo Added Deadlines for Guidelines. Maria Alandes Pradillo Updated Executive Summary and Conclusions Alberto Aimar Final release PEB Document Change Record Issue Item Reason for Change 1.1 Deadlines for guidelines 1.2 Updated Executive Summary and Conclusions The different satellite guideline documents need to have clear release dates, which were missing Exec summary and conclusions were too short and not descriptive enough to be useful INFSO-RI Members of EMI collaboration PUBLIC 3 / 47

4 TABLE OF CONTENTS 1. INTRODUCTION PURPOSE DOCUMENT ORGANISATION APPLICATION AREA REFERENCES DOCUMENT AMENDMENT PROCEDURE TERMINOLOGY EXECUTIVE SUMMARY SQA FACTORS SQA MANAGEMENT SQA ORGANISATION SQA TASKS SQA Tasks Summary Documentation tasks Minimum Documentation Requirements Technical Development Plan Software Release Plan Software Release Schedule Software Maintenance and Support Plan QA Tools Continuous Integration and Certification Testbeds Security Assessment Plan Security Assessments Review Tasks Review of the Minimum Required Documentation Review the Technical Development Plan Review of the Software Release Plan Review of the EMI software components Review the Software Release Schedule Review the Software Maintenance and Support Plan Review of the EMI Production Releases Review the Security Assessments Review the SQAP Reporting tasks Periodic QA Reports Periodic Software Development Quality Control Periodic Software Maintenance Quality Control PROCEDURES MEETINGS HOW TO CARRY OUT QA REVIEWS SUPPORT TO PRODUCT TEAMS METRICS CRITICAL BUG METRICS BUG SEVERITY DISTRIBUTION BACKLOG MANAGEMENT INDEX FAILED BUILDS METRIC INTEGRATION TEST EFFECTIVENESS METRIC UP-TO-DATE DOCUMENTATION METRIC DELAY ON RELEASE SCHEDULE METRIC INFSO-RI Members of EMI collaboration PUBLIC 4 / 47

5 6.8. UNIT TEST COVERAGE METRIC MEMORY LEAKAGE WARNINGS METRIC CODE COMMENTING METRIC NUMBER OF SUPPORTED PLATFORMS METRIC TOTAL BUG DENSITY METRIC BUG DENSITY PER RELEASE METRIC TOTAL USER INCIDENTS METRIC TRAINING AND SUPPORT INCIDENT METRIC AVERAGE TIME TO DEAL WITH AN INCIDENT STANDARD PRACTICES AND CONVENTIONS CONCLUSIONS INFSO-RI Members of EMI collaboration PUBLIC 5 / 47

6 1. INTRODUCTION 1.1. PURPOSE The European Middleware Initiative is a close collaboration of the three major middleware providers, ARC, glite and UNICORE, and other software providers. It will deliver a consolidated set of middleware components for deployment in EGI, PRACE and other distributed computing infrastructures. The SQAP specifies the manner in which the EMI project is to achieve its quality goals [R1] in terms of software development. The SQAP applies to all EMI software components. The list of EMI software components can be found in the EMI components table from the project deliverable document: DNA1.3.1 Technical Development Plan [R9]. The SQAP will specify which documents are needed, which reviews are going to be carried out and the roles and responsibilities within the EMI project related to the software lifecycle DOCUMENT ORGANISATION The Software Quality Assurance Plan is organized as follows: Chapter 1 and 2 are the introduction and the executive summary respectively. Chapter 3 describes the management: who is responsible for quality and what tasks need to be carried out. Chapter 4 describes the documentation: which documents ensure the quality of the EMI software. Chapter 5 describes the procedures: how the software quality assurance plan is managed APPLICATION AREA The Software Quality Assurance Plan refers to the EMI software life cycle and basically affects SA1, SA2 and JRA1 activities REFERENCES Table 1: Table of References R1 R2 R3 R4 R5 R6 R7 R8 Eric J. Braude Software Engineering An Object-Oriented Perspective IEEE Standard For Software Quality Assurance Plans IEEE Guide for Software Quality Assurance Planning ISO/IEC Software Lifecycle process ISO/IEC-9126 Software Engineering Product Quality ITIL v3 LCG glite Testing Guidelines EMI DoW INFSO-RI Members of EMI collaboration PUBLIC 6 / 47

7 R9 R10 R11 R12 R13 R14 R15 The McCall Quality Model EMI Technical Development Plan Project Deliverable DNA (to be released) Atlassian Bamboo Valgrind framework The Eclipse Test & Performance Tools Platform EMI Incident Management System (GGUS) LCG-ROLLOUT mailing list DOCUMENT AMENDMENT PROCEDURE Amendments, comments and suggestions should be sent to the SA2.2 task leader. This document can be amended by the Quality Assurance team (SA2) further to any feedback from the other teams. Minor changes, such as spelling corrections, content formatting or minor text reorganisation not affecting the content and meaning of the document can be applied by SA2 without previous review. Other changes must be peer reviewed and submitted to the PEB for approval. When the document is modified for any reason, its version number shall be incremented accordingly. The document version number shall follow the standard EMI conventions for document versioning. All versions of the document shall be maintained using the document management tools selected by the EMI project TERMINOLOGY This section explains the main terms and acronyms used in the document. These are consistent with the definition in the EMI DoW [R8]. The Quality Assurance activities will use existing ISO standards for the definition of the common terminology and software lifecycle phases and will follow the guidelines of the good-practice methodologies described in CMMi and ITIL. API APT Application Programming Interface Advanced Packaging Tool INFSO-RI Members of EMI collaboration PUBLIC 7 / 47

8 BDII CMMi DoW GGUS EMT ITIL KLOC KPI LOC PEB PT PTB SDP SQA SQAP SQC WP YUM Berkley Database Information Index Capability Maturity Model Integration Description of Work Global Grid User Support Engineering Management Team Information Technology Infrastructure Library Thousands of Lines of Code Key Performance Indicator Lines of Code Project Executive Board Product Team Project Technical Board Software Development Plan Software Quality Assurance Software Quality Assurance Plan Software Quality Control Work Package Yellowdog Updater Modified Table 1: Table of Acronyms Incident Problem Problem Record Product Team An unplanned interruption to an IT Service or a reduction in the Quality of an IT Service. A cause of one or more Incidents. The cause is not usually known at the time a Problem Record is created, and the Problem Management Process is responsible for further investigation. A Record containing the details of a Problem. Each Problem Record documents the Lifecycle of a single Problem. Product Teams are teams of software developers fully responsible for the successful release of a particular software product (or group of tightly coupled related products) compliant with agreed sets of technical specification and acceptance criteria. INFSO-RI Members of EMI collaboration PUBLIC 8 / 47

9 Request Software Development Plan Software Measurement Software Metric Software Quality Assurance Software Quality Assurance Plan Software Quality Control Work Package A request from a User for information, or advice, or for a Standard Change or for Access to an IT Service. A project plan for a software development project. A measurement is an indication of the size, quantity, amount or dimension of a particular attribute of a product or process. For example the number of errors in a system is a measurement. A Metric is a measurement of the degree that any attribute belongs to a system, product or process. For example the number of errors per person hours would be a metric. A set of activities designed to evaluate the process by which software products are developed or manufactured. The software quality assurance plan is a document that describes the procedures, documents, roles and responsibilities that will ensure the quality of the software development process. A set of activities designed to evaluate the quality of developed or manufactured software products. In addition, in EMI, SQC also verifies that User Requirements are satisfied. The EMI project is composed of two Networking Work Packages (NA1 and NA2), two Service Work Packages and one Joint Research Work Packages (JRA1). Table 2: Table of Definitions INFSO-RI Members of EMI collaboration PUBLIC 9 / 47

10 2. EXECUTIVE SUMMARY The Software Quality Assurance Plan (SQAP) is a document that specifies tasks that are needed to ensure quality, responsibilities for quality monitoring, documentation required and procedures that will be followed to manage the software process. In practice the SQAP specifies the manner in which the EMI project is going to achieve its quality goals. The quality factors that have been agreed and their utility for the project are explained. Metrics and measurements are associated to the quality factors in order to evaluate the quality of the EMI software lifecycle. These focus on Product Operation, Revision and Transition. SQA tasks, roles and responsibilities of the EMI technical activities (SA1, SA2 and JRA1) are described as well as those of the EMI technical bodies (PTB and EMT). The SQA tasks follow the EMI DoW guidelines and cover documentation tasks, reporting tasks and reviewing tasks. The documentation tasks include the minimum documentation requirements for each software component and services and must be provided by every EMI product team. Other documents on plans and schedules are more used for the strategic and technical coordination controlled by the PTB and the EMT groups. Two documents are particularly important as they define and guide the general project-wide technical objectives and the plans to achieve them: Technical Development Plan provides the details of the technical development plan for all EMI services. It contains an initial status assessment, a list of known requirements, requirement prioritisation and a plan with agreed deliverables and milestones. Software Release Plan describes the release procedures and policies for the middleware services and the complete EMI reference distributions. It also contains the initial release schedule to be prepared by the EMT in collaboration with the PTB and the JRA1work package. Additional documents are needed to describe the software engineering tools and the repository management systems provided by SA2 to EMI and third-party users; and to describe the distributed certification test beds for internal and acceptance certification and its access and usage requirements. The reviews aim at helping to ensure the quality of the EMI software. Review tasks are under the supervision of SA2 as a whole and in particular of SA2.5, which is responsible for collecting and storing the reviews making sure they are conducted following the SQAP. The reviews will verify all artefacts and documents produced and create review reports and tables that summarize the status of the software management and development activities. This document also describes the procedures in place for managing the software quality assurance process. It provides details about the communications means (meetings, mailing lists, twiki areas, etc) to be used for QA activity, how the reviews will be performed and the interaction workflow among the different work packages and project bodies. It also describes the change management: details on how to change the SQA documents if the results of a review are not meeting the quality expectations. The last section presents the list of metrics used for the different reviews planned in the SQAP. The list of metrics includes metrics on the development and support of the products (external metrics) and metrics on the artefacts like documentation and code (internal metrics). These metrics can be modified throughout the lifetime of the project and will be possible to add new metrics or remove obsolete metrics according to the needs of the SQA process. INFSO-RI Members of EMI collaboration PUBLIC 10 / 47

11 For technical details, five satellites documents are being maintained to serve as guidelines for some of the software tasks (release, integration, certification, testing, etc). INFSO-RI Members of EMI collaboration PUBLIC 11 / 47

12 3. SQA FACTORS This section presents the quality factors [R9] that have been agreed to respect and their utility for the project. Metrics and measurements will be associated to the quality factors to be able to evaluate the quality of the EMI software lifecycle. They are grouped in three parts: Product Operation Usability: The software must be easy to use providing intuitive, effective and efficient interfaces to end users. Correctness: The software must answer to the objectives of the end users and satisfy their requirements. Reliability: The software must consistently perform according to its specifications and report any internal error that might influence its functionality. Availability: The software must allow the system where it has been deployed to remain operational even when faults occur. The system must remain operational even in presence of a high load for hardware resources and continue operating even at a reduced capacity depending of the number of user's requests it is receiving. Integrity: The software must guarantee a high level of security about the protection of data against loss, theft or unauthorized access. Product Revision Maintainability: The maintainability is an indicator of the ease by which a component, device or system can be maintained and repaired. Flexibility: The flexibility is an indicator of the ease by which a component, device or system can be adapted to changes. Testability: The testability is an indicator of the ease by which a component, device or system can be tested. Product Transition Portability: The software can be portable on different platforms with ease. INFSO-RI Members of EMI collaboration PUBLIC 12 / 47

13 4. SQA MANAGEMENT 4.1. SQA ORGANISATION Each product team member is responsible for the quality of his or her work. In addition, the following roles are part of the organisational structure that directly influences the quality of the software: SA2 Quality Assurance SA2 Quality Assurance activity leader: Alberto Aimar: responsible for the whole SQA layer of EMI. SA2.2 Quality Assurance plan task leader: Maria Alandes: responsible for the Software Quality Assurance Plan, which is a plan written at project level to declare commitment to follow an applicable set of standards, regulations, procedures and tools during the development lifecycle. SA2.3: Metrics and KPIs task leader: Eamonn Kenny: responsible for the definition, collection and revision of software quality metrics. SA2.4: Tools and Repositories task leader: Lorenzo Dini: responsible for the set up and maintenance of the tools needed in the QA process. SA2.5: QA Implementation review and support task leader: Jozef Cernak: responsible for the review activities and support to the PTs as far as QA is concerned. SA2.6: Test beds task leader: Danilo Dongiovanni: responsible for the set up and maintenance of test beds. JRA1 Middleware Development, Evolution and Integration JRA1 Quality Control task leader: Andrea Ceccanti: responsible for the Software Quality Control layer, which ensures both SQA and SQAP defined by SA2 are being followed by the development teams. JRA1 leader: Morris Riedel, responsible for the Middleware Development: Evolution and Integration WP. JRA1.2 leader: Massimo Sgaravatto, responsible for the Compute Area. JRA1.3 leader: Patrick Fuhrmann, responsible for the Data Area. JRA1.4 leader: John White, responsible for the Security Area. JRA1.5 leader: Laurence Field, Infrastructure Area. SA1 Maintenance and Support SA1 Quality Control task leader: Giuseppe Fiameni: responsible for the Software Quality Control layer, which ensures both SQA and SQAP defined by SA2 are being followed by the development teams. SA1 leader: Francesco Giacomini, responsible for the Software Maintenance and Support WP. SA1.3 Release Manager: Cristina Aiftimiei. The following boards and management roles may be involved to take decisions affecting the SQA procedures when there is a conflict or a critical change: Technical Director: decides on specific technical matters within the project and leads the Project Technical Board. INFSO-RI Members of EMI collaboration PUBLIC 13 / 47

14 PEB: responsible to assist the Project Director in the execution of the project plans and in monitoring the milestones, achievements, risks and conflicts within the project. It is led by the PD and is composed of the Work Package Leaders, the Technical Director and the Deputy Technical Director. PTB: is led by the Technical Director and composed of the Technical Area Leaders and a representative and is responsible to assist the Technical Director in defining the technical vision of the project and deciding on specific technical issues. The PTB members are the coordinators of the project Technological Areas. The PTB can invite experts (e.g. product team leaders, component developers, etc.) or delegate specific tasks to appointed working groups as required. EMT: is lead by the Release Manager and composed of the Product Team Leaders (or duly appointed representatives), a QA representative, a Security representative, representatives of the Operations teams of the major infrastructures (EGI, PRACE, etc.) and invited experts as necessary. The role of the EMT is to manage the release process and the 'standard changes, that is all the changes that are pre-approved based on established policies and do not need to be individually approved by the PEB or the PTB. This includes software defects triaging and prioritization, technical management of releases, integration issues, and user support request analysis SQA TASKS SQA Tasks Summary A summary of the SQA tasks is described below. They have been defined following the guidelines in [R7] EMI DoW, [R2] IEEE Standard for Software Quality Assurance Plans and [R3] IEEE Guide for Software Quality Assurance Planning. For a complete description of each task, please check the following sections: Documentation tasks: o o Definition of the Minimum Required Documentation for each software component. Definition of the Technical Development Plan: There is a project deliverable DNA1.3.1 Technical Development Plan. o Definition of the Software Release Plan: There is a project deliverable DSA1.2 - Software Release plan. o o o o Definition of the Software Maintenance and Support Plan: There is a project deliverable DSA1.1 - Software Maintenance and Support Plan. Definition of the QA Tools Documentation: There are several project deliverables for QA Tools Documentation: DSA 2.2.1, DSA2.2.2 and DSA Definition of the Continuous Integration and Certification Test beds: There is a project deliverable DSA2.4 - Continuous Integration and Certification Test beds. Security Assessments. Reporting tasks: o Periodic QA Reports: There are project deliverables DSA2.3.1 DSA2.3.4 every end of project quarter. INFSO-RI Members of EMI collaboration PUBLIC 14 / 47

15 o o Periodic Software Development Quality Control: There are project deliverables DJRA1.7.1 DJRA1.7.4 every end of project quarter. Periodic Software Maintenance Quality Control: There are project deliverables DSA1.3.1 DSA1.3.4 every end of project quarter. Review tasks: o Review of the Minimum Required Documentation for each software component. o Review the status of the Software Development Plan. o Review of the status of the Software Release Plan. o Review of the EMI software component releases. o Review the status of the Software Release Schedule. o Review the status of the Software Maintenance and Support Plan. o Review of the EMI Production Releases. o Review the Security Assessments. o Review the SQAP Documentation tasks This section defines the documents governing the development, verification and validation, use and maintenance of the software. The documents will be accessible from the SQAP twiki page under: Changes and updates to these documents will be notified using the SA2.2 activity mailing list Minimum Documentation Requirements Description: This document identifies the documentation governing the development, verification, validation, use and maintenance of the software. Deadline: It s part of the SQAP. Author: Maria Alandes Requirements: The Minimum Documentation Requirements for the software components and services are: Software Requirements Specifications Software Design Description Software Verification and Validation Plan Software Verification and Validation Report User Documentation Installation Guide Troubleshooting guide Software Requirements Specifications INFSO-RI Members of EMI collaboration PUBLIC 15 / 47

16 Description: The Software Requirements Specifications should define the requirements of the software component. Author: PTB Requirements: The PTB will decide whether a detailed document describing the software requirements is needed or not. Otherwise a tracking tool will be used to describe and track the requirements. Validation criteria (which tests are needed to verify the requirements) should be also included. Software Design Description Description: The Software Design Description should describe the software architecture of the software component. Author: PTB or a delegated person by the PTB. Requirements: The PTB will decide whether a detailed document describing the software design is needed or not per software component basis. Software Verification and Validation Plan Description: The Software Verification and Validation Plan document should describe the strategy that will be adopted in testing each software component. Author: Product teams and PTB. Requirements: The set of tests included in the Software Verification and Validation Plan should contain: Unit and Functional tests o o Unit Testing: Testing of individual software units. Functional Testing: Testing conducted to evaluate the compliance of a component with specified functional requirements Integration tests o o CLI: every command of the CLI, every available option. API: every function or method. GUI: correct functioning of graphical interfaces. System Testing: verify if the component works within the grid in interaction with other grid services. Deployment Testing: verify if the component installs and configures for upgrade and clean installations. Regression tests Performance and Scalability tests Definition of which set of tests are executed for each type of release (major, minor and revision). Complete and detailed guidelines on what should be included in the Verification and Validation Plan will be given as described in section 7. INFSO-RI Members of EMI collaboration PUBLIC 16 / 47

17 Software Verification and Validation Report Description: The Software Verification and Validation Report should contain the result of the tests specified in the Software Verification and Validation Plan. This report should be provided for each type of release as it is also specified in the Software Verification and Validation Plan. Author: Product teams Requirements: The Verification and Validation Report should at least contain: Test coverage of the code The test report should contain the test results of all the tests included in the Software Verification and Validation plan according to the type of release (major, minor or revision). Complete and detailed guidelines on what should be included in the Verification and Validation Report will be given as described in section 7. User Documentation Description: The User Documentation should contain all the necessary information needed by the users of the software. Author: Product teams. Requirements: The User documentation for each EMI software component should contain when applicable: Introduction to the software component describing the main functionality and architecture. Installation and configuration instructions. Access or log-on and sign-off the software. Information on software commands: required parameters, optional parameters, default options, order of commands, and syntax. Description of all error situations which can occur and how to react. Glossary with terms specific to the usage of the software component. References to all documents or manuals intended for use by the users. o o o o Installation Guide Known Issues System Administrator Guide (if any) Tutorials Developer s Guide i.e. API documentation, Build instructions. Description: The Installation Guide should contain the necessary instructions on how to install and configure the software. Author: Product teams Requirements: The Installation Guide for each EMI software component should contain: The list of supported platforms Software and Hardware requirements to install the software component INFSO-RI Members of EMI collaboration PUBLIC 17 / 47

18 Installation instructions o Supported installation tools o Installation command Configuration instructions o Default configuration variables o Mandatory configuration variables o Configuration command Known Issues during Installation and Configuration of the software component. Troubleshooting guide Description: The Troubleshooting Guide of each EMI software component should help users when tracking and solving problems. Author: Product teams Requirements: The Troubleshooting Guide for each EMI software component should describe the most common scenarios where users have problems in the following areas: Installation Configuration Administration Debugging Technical Development Plan Description: There is a project deliverable DNA1.3.1 Technical Development Plan. This document provides the details of the technical development plan for all EMI services. It contains an initial status assessment, a list of known requirements and their prioritization and a plan with deliverables and milestones for each Product Team. Deadline: PM3. Author: Balazs Konya Requirements: The Technical Development Plan should specify the following items for each software component: The software component description. The responsible product team. The work plan for the implementation of the software component including the list of the milestones concerning the release of each feature. The description of the activities and resources required and planned to meet specifications detailed in requirements documents Software Release Plan Description: There is a project deliverable DSA1.2 - Software Release plan. This document describes the release procedures and policies for the middleware services and the complete EMI reference INFSO-RI Members of EMI collaboration PUBLIC 18 / 47

19 distributions. It also contains the initial release schedule to be prepared in collaboration with the PTB and the JRA1WP. Deadline: PM3. Author: Cristina Aiftimiei Requirements: The Software Release Plan should specify the way software components are going to be released into the Production EMI Repository. It should define the following: How to manage revision, minor and major releases. How to manage external dependencies. The build process. The communication channels: mailing lists, meetings. The software delivery strategy to the Production Infrastructures: i.e. Where to upload software packages, which metadata information is required Software Release Schedule Description: The Software Release Schedule should define the dates for which certain versions of the software components need to be released in the Production EMI Repository. It has to take into account the priorities of the project that need to be aligned with the priorities of the different infrastructures using EMI. Deadline: Every three months according to [7] EMI DoW. Author: Cristina Aiftimiei Requirements: The Software Release Schedule should contain: List of software component to be released in the next three months. Estimated dates for the release of each component Software Maintenance and Support Plan Description: There is a project deliverable DSA1.1 - Software Maintenance and Support Plan. This document describes the Software Maintenance and Support processes, the roles and responsibilities and the main metrics to be used for Service Level Agreements. Deadline: PM3. Author: Francesco Giacomini Requirements: The Software Maintenance and Support Plan should describe the way to maintain and support the EMI middleware. It should define the following: How to handle incidents reported by EMI users using GGUS. How to handle requests coming from EMI users or other PTs. How to handle problems. How to handle changes. All these requirements should be measurable. INFSO-RI Members of EMI collaboration PUBLIC 19 / 47

20 4.2.8 QA Tools Description: There is a project deliverable for QA Tools Documentation DSA2.2.1, DSA2.2.2 and DSA This document describes the software engineering tools and the repository management systems provided by SA2 to EMI and third-party users. This document is updated and revised regularly. Deadline: PM3, PM10 and PM22. Author: Lorenzo Dini. Requirements: The tools documentation should define the following: How to sign software packages. How to upload software packages in the EMI repository. What technologies are supported to install the software packages: i.e. APT and YUM. Description of the tools/scripts used to generate metrics from the tracking tools. Up to date inventory of the tools used by the product teams. Dashboard or any other tool needed by Quality Control to monitor the software components Continuous Integration and Certification Testbeds Description: There is a project deliverable DSA2.4 - Continuous Integration and Certification Test beds. This document describes the distributed certification test beds for internal and acceptance certification and its access and usage requirements. Deadline: PM3. Author: Danilo Dongiovanni Requirements: The Continuous Integration and Certification Test beds document should define: The test bed structure with the list of available services and their versions. The procedure to access the test bed and run tests. The monitoring strategy of the test bed Security Assessment Plan Description: The Security Assessment Plan identifies which software components are going to be assessed and when the assessments are going to take place. Author: Elisa Heymann Deadline: PM6 Requirements: Description of the Security Criteria. List of components and prioritisation for their assessment Security Assessments Description: Vulnerability assessments will be carried out following the approach called First Principles Vulnerability Assessment (FPVA). FPVA is a primarily analyst-centric (manual) approach INFSO-RI Members of EMI collaboration PUBLIC 20 / 47

21 to assessment whose aim is to focus the analyst s attention on the parts of the software system and its resources that are mostly likely to contain vulnerabilities. FPVA is designed to find new threats to a system. It s not dependent on a list of known threats. Deadline: As defined in the Security Assessment Plan. Author: University of Wisconsin and Universitat Autonoma de Barcelona. Requirements: Architectural Analysis: identify the major structural components of the system, including modules, threads, processes, and hosts. For each of these components, identify the way in which they interact, both with each other and with users. The artefact produced at this stage is a document that diagrams the structure of the system and the interactions amongst the different components and with the end users. Resource Identification: identify the key resources accessed by each component and the operations supported on those resources. Resources include elements such as hosts, files, databases, logs, and devices. For each resource, describe its value as an end target or as an intermediate target. The artefact produced at this stage is an annotation of the architectural diagrams with resource descriptions. Trust and Privilege Analysis: identify the trust assumptions about each component, answering such questions as how are they protected and who can access them. Trust evaluation is also based on the hardware and software security surrounding the component. Associated with trust is describing the privilege level at which each executable component runs. The privilege levels control the extent of access for each component and, in the case of exploitation, the extent of damage that it can accomplish directly. A complex but crucial part of trust and privilege analysis is evaluating trust delegation. By combining the information from the first two steps, we determine what operations a component will execute on behalf of another component. The artefact produced at this stage is a further labelling of the basic diagrams with trust levels and labelling of interactions with delegation information. Component Evaluation: examine each component in depth. For large systems, a line-by-line manual examination of the code is infeasible, even for a well-funded effort. A key aspect of our technique is that this step is guided by information obtained in the first three steps, helping to prioritize the work so that high value targets are evaluated first. The artefacts produced by this step are vulnerability reports, perhaps with suggested fixes, to be provided to the software developers. Dissemination of Results: Vulnerabilities are reported to the PT leaders. Users are never informed of vulnerabilities until they are fixed and the fix is available to the users Review Tasks The reviews will help to ensure the quality of the EMI software. The metrics associated with each review are described in section 6. The review tasks will be under the supervision of the SA2.5 activity that is responsible for collecting and storing the reviews making sure they happen according to the SQAP. Reviews and Review Templates will be available from the SQAP twiki page under: The following reviews need to be carried out throughout the software lifecycle: INFSO-RI Members of EMI collaboration PUBLIC 21 / 47

22 Review of the Minimum Required Documentation Description: The Review of the Minimum Required Documentation should asses for each software component the status of the documents that are defined in section Frequency: To be included in every SA2 QA report. Author: Maria Alandes. Review checklist: follow the checklist below for each software component: All the documents exist and are up to date. The contents of the documents conform to the descriptions done in section 4.1. Output: It s a document containing a table with the EMI software components and the list of the minimum required documents. A link to the document will be included if it exists and, if applicable, any metrics or measurements associated with the document. If the document doesn t exist, it will be marked as NONE. The table will be published in the Periodic QA report and sent to JRA1 so that developers are aware of what it is missing as far as documentation is concerned for their software components Review the Technical Development Plan Description: The Review of the Technical Development Plan should check that the plan is up to date and that it reflects the real development plans of the project. Frequency: PM6, PM9, PM12, PM15, PM18, PM21, PM24, PM27, PM30, PM33, PM36. Author: Jozef Cernak. Review checklist: follow the checklist below for each software component: The software component deliverables and associated completion criteria is up to date. In case of an extended deadline, there is a justification and it s documented. The schedule and interrelationships among other software components is up to date. The responsibility for the component is up to date. Output: It s a document that contains a report on the status of every item in the checklist and the metrics and measurements associated with the Technical Development plan Review of the Software Release Plan Description: The Review of the Software Release Plan should check that the plan is up to date and that it describes the actual release process. Frequency: PM6, PM9, PM12, PM15, PM18, PM21, PM24, PM27, PM30, PM33, PM36. Author: Giuseppe Fiameni. Review checklist: follow the checklist below: The list of supported platforms corresponds to the actual set of platforms on which software components are released. Installation of external dependencies is well documented. Instructions to use the supported build systems are up to date. The list of supported delivery software formats is up to date (source and binary packages, tarballs, package lists, etc). INFSO-RI Members of EMI collaboration PUBLIC 22 / 47

23 The description of the process on how to handle changes is up to date. The communication channels are published with updated information about: Mailing lists developers should use for discussion, requests, announcements, etc. Meetings and the purpose of each of them. The process on how to deliver software to the Production Infrastructures is up to date and it s aligned to what the Production Infrastructures are expecting. Output: It s a document that contains a report on the status of every item in the checklist and the metrics and measurements associated with the Software Release Plan Review of the EMI software components Description: The Review of the software component should check that the software components are meeting the requirements described below. Frequency: Every week. Author: Andrea Ceccanti Review checklist: follow the checklist below: The software component contains the required code metrics. The software component meets the naming and packaging conventions. The software component builds successfully in the supported platforms. The necessary tests (unit and functional tests) have been carried out and have been successful according to the Test Plan of the component. The corresponding bug tracker items are consistent and reflect the release status of the component. Tickets assigned to each product team are dealt with. If applicable, security vulnerabilities have been addressed. Output: The status of each item in the checklist should be integrated and monitored in a dash board that will automatically collect and publish the relevant metrics and measurements associated to the software component. Alarms and notification on deviations should be monitored by Andrea Ceccanti Review the Software Release Schedule Description: The Review of the Software Release Schedule should check that the priorities of the project are taken into account and reflected in the scheduled releases. Frequency: PM6, PM9, PM12, PM15, PM18, PM21, PM24, PM27, PM30, PM33, PM36. Author: Giuseppe Fiameni Review checklist: follow the checklist below: Check whether the previous schedule has been kept. Check whether the new schedule takes into account what wasn t accomplished in the previous schedule. INFSO-RI Members of EMI collaboration PUBLIC 23 / 47

24 Check whether the new schedule is aligned to the Software Development Plan and the priorities of the project. Output: It s a document that contains a report on the status of checklist and the metrics and measurements associated with the Software Release Schedule Review the Software Maintenance and Support Plan Description: The Review of the Software Maintenance and Support Plan should check that the plan is up to date and describes the actual maintenance and support processes and that the SLAs are respected. Frequency: PM6, PM9, PM12, PM15, PM18, PM21, PM24, PM27, PM30, PM33, PM36. Author: Giuseppe Fiameni Review checklist: follow the checklist below: The process on how to handle incidents reported by EMI users using GGUS is up to date. The process on how to handle requests coming from EMI users or other PTs is up to date. The process on how to handle problems is up to date. Output: It s a document that contains a report on the status of checklist and the metrics and measurements associated with the Software Maintenance and Support Plan Review of the EMI Production Releases Description: the Review of the EMI Production Releases should check that the release is meeting the requirements described below. Frequency: for every EMI Production Release. Author: Cristina Aiftimiei Review Checklist: follow the checklist below: Release notes: they exist and they summarise the most relevant changes in the release. Repositories: they are updated with the new packages. Announcement to the relevant users: it s done after the release notes are published and the repositories updated. Status of the relevant bug tracker items: it s consistent. Test reports: they exist and they are successful. They also contain the minimum set of tests necessary to pass the certification (deployment and regression tests). Output: It s a verification report that should be part of the documentation shipped in the EMI release. The verification report summarises the status of the items in the check list and it also contains any metrics and measurements associated to the EMI release Review the Security Assessments Description: the Review of the Security Assessments should check that the different stages described in the First Principles Vulnerability Assessment (FPVA) approach are being followed. Frequency: Part of the QC reports. INFSO-RI Members of EMI collaboration PUBLIC 24 / 47

25 Author: Giuseppe Fiameni Review Checklist: follow the checklist below: The Architectural Analysis has been carried out and the output contains a diagram describing the interactions among components and end users. The Resource Identification has been carried out and the output contains the resource descriptions. The Trust and Privilege Analysis has been carried out and the output contains the trust levels and the delegation information for all the components and their interactions. The Component Evaluation has been carried out and the output contains identified vulnerabilities and their suggested fixes. The Dissemination of Results has been carried out Review the SQAP Description: the Review of the SQAP should check that the plan is being followed and that all the activities described in it are being carried out as planned. Frequency: Every month. Author: Maria Alandes Review Checklist: follow the checklist below: Check that the following tasks are carried out as defined in the plan: o o o Documentation tasks Review tasks Reporting tasks Check the list of responsibilities is up to date. Check the description of the SQA procedures is up to date. Check the list of metrics is up to date. Check the guidelines are up to date. Output: It s a document that contains a report on the status of the checklist Reporting tasks Periodic QA Reports are planned to be done throughout the lifetime of the project. The following set of reports are planned to be done as project deliverables: Periodic QA Reports Description: QA Reports will summarise the metrics collected by the SA2.3 activities and the reviews performed by the SQC task leaders. They will be done in collaboration with the SA2.5 activity. Deadlines: DSA Periodic QA Reports - End of July DSA Periodic QA Reports - End of April INFSO-RI Members of EMI collaboration PUBLIC 25 / 47

26 DSA Periodic QA Reports - End of April DSA Periodic QA Reports - End of April Author: Maria Alandes Contents: QA reports should summarise the results of the different review tasks and evaluate whether the quality factors defined by the project are respected Periodic Software Development Quality Control Description: Periodic Software Development Quality Control summarise the status and performance of the software development activity. Deadlines: DJRA1.7.1 Software Development Quality Control Report End of July 2010 DJRA Software Development Quality Control Report End of April DJRA Software Development Quality Control Report End of April DJRA Software Development Quality Control Report End of February Author: Andrea Ceccanti Contents: The Software Development Quality Control reports should summarize the results of the EMI software component reviews. It should also contain details on the availability and execution of unit, functional and compliance tests for the EMI components Periodic Software Maintenance Quality Control Description: Periodic Software Maintenance Quality Control summarise the status and performance of the software maintenance activity. Deadlines: DSA Software Maintenance Quality Control Report End of October DSA Software Maintenance Quality Control Report End of February DSA Software Maintenance Quality Control Report End of February DSA Software Maintenance Quality Control Report End of February Author: Giuseppe Fiameni Contents: The Software Maintenance and Quality Control reports should summarize the results of the Software Release Plan Review, Software Release Schedule Review and Software Maintenance and Support Plan Review. It should also contain details on availability and execution of regression tests for the supported EMI components and various metrics on released components. INFSO-RI Members of EMI collaboration PUBLIC 26 / 47

27 5. PROCEDURES This section describes the procedures to manage the software quality assurance process. This includes: - Meetings: Details about the meetings to be held for QA activity. - Reviews: Details on how the reviews will be carried out and the workflow among the different work packages and management. It also describes the change management: details on how to change the SQA documents if the results of a review are not meeting the quality expectations. - Support: What activities will be done to give support to product teams as far as QA is concerned MEETINGS The following meetings will be held to allow project members discuss and organise the different QA activities: - Weekly phone meetings will be held every Wednesday at 10h00 to coordinate the whole SA2 activity. The meeting will be chaired by the SA2 task leader. All members of the SA2 activity shall participate. SQC task leaders are also welcome to participate. The different task leaders shall report about the progress of their tasks. o o Indico Agendas: Minutes of the meetings: Meetings will be also organised as needed by the different SA2 task leaders to coordinate their tasks. These meetings will be announced and the different people who should attend the meeting will be contacted. The different SA2 members may be also appointed to take part in other activity meetings for the purpose of representing the SQA activity or to take care that SQA issues are taken into account across the project. One example is the EMT meetings where there is always a member of SA2. Mailing lists are also an important communication channel to discuss open issues, to make announcements or distribute documentation. The registration is open but the activity or tasks leaders should give their approval before. The following mailing lists are used within SA2: - emi-sa2@eu-emi.eu : general mailing list to discuss about SQA issues. - emi-sa22@eu-emi.eu : mailing list to discuss SA2.2 related issues (SQAP). - emi-sa23@eu-emi.eu : mailing list to discuss SA2.3 related issues (Metrics). - emi-sa24@eu-emi.eu : mailing list to discuss SA2.4 related issues (Tools). - emi-sa25@eu-emi.eu : mailing list to discuss SA2.5 related issues (SQA review and support). - emi-sa26@eu-emi.eu : mailing list to discuss SA2.6 related issues (Test beds) HOW TO CARRY OUT QA REVIEWS The QA Reviews should be carried out in the following way: INFSO-RI Members of EMI collaboration PUBLIC 27 / 47