A FRAMEWORK FOR THE SOFTWARE ASPECTS OF THE SAFETY CERTIFICATION OF A SPACE SYSTEM *

Transcription

1 A FRAMEWORK FOR THE SOFTWARE ASPECTS OF THE SAFETY CERTIFICATION OF A SPACE SYSTEM * GL. Cleland (1), JP. Blanquart (2), JM. Carranza (3), PKD. Froome (1), CCM. Jones (1), JF. Muller (2) (1) ADELARD ({glc, pkdf, ccmj}@adelard.com) Drysdale Building Northampton Square London EC1V 0HB United Kingdom Tel : (44 ) / Fax : (44) (2) ASTRIUM-SAS ({jean-paul.blanquart, jean-francois.muller}@astrium-space.com) 31, rue des Cosmonautes, ZI du Palays Toulouse Cedex - France Tel : (33) / Fax : (33) (3) ESA/ESTEC ([email protected]) ESTEC, European Space Agency PO Box 299, NL-2200 AG Noordwijk ZH-The Netherlands Tel : (31) / Fax : (31) ABSTRACT is a procedure by which a third-party gives written assurance that a product, process or service conforms to specified requirements. Currently there is no general policy for certification of space systems. A study funded by ESA's General Studies Program is under way, aiming at defining a framework for the software aspects of the safety certification of a space system. In particular the study will propose a set of requirements for the development of software that will be part of a system to be certified. It will also propose certification requirements for that software and a generic certification plan. Finally the study will define a set of requirements for the accreditation of certification personnel and laboratories. INTRODUCTION The role of software in space systems is drastically growing while space system criticality increases. Moreover, space systems often play a significant part in other systems where the failure or abnormal behaviour of the space system may induce catastrophic results on other systems e.g. positioning system used in the civil aviation or maritime domains, tele-medicine. In such a context, space systems are becoming more and more safety critical and decisions about their use rely on a full assessment of the risk involved. In the short term, certification will not only be required for manned space projects or systems which may directly endanger human lives, but also for the majority of space systems because of their high cost or because of the possible indirect impact on other systems which rely on services they provide. In this context, ESA granted the CARES study to a consortium lead by Astrium-SAS with Adelard, Critical Software, EADS Airbus and DNV as partners with the objective of defining a generic certification framework related to ESA space software systems. ESA has already sponsored a number of studies related to dependability, process improvement and certification, the more recent being the GNSS-2 [4] certification study related to the certification of the GNSS-2 system. Others include the PASCON [6] and SPEC [7] projects. In addition a number of standards (the ECSS series) are already in use in the space domain. In order to make best use of this existing material, and to promote an evolutionary approach (which will ease uptake of a new certification scheme), the CARES project has studied these previous projects, and will make recommendations for the reuse of the ECSS standards in the proposed new certification framework. does of course take place in other domains, and the project has also studied and consulted in a variety of these areas to identify best practice, as well as techniques and methods which are relevant to space software certification. The results of this consultation is summarised in the next section, but one consistent trend that emerged was the general tendency to move towards goal-based regulation, and away from prescriptive approaches. A regulator now tends to publish high level goals to be achieved, and suppliers, or supply chains, are required to propose appropriate approaches to demonstrate that their product meets the regulatory goals. From this work the aim is to develop a certification framework, which is built upon proven techniques and practices, although not (necessarily) to propose new methods to increase safety. The framework will provide a contribution to the overall safety assessment of the system, of which the software elements are only a part. * Published in the proceedings of the NASA/ESA Flight Conference, June 2003, Noordwijk, Holland

2 Partners in the project include ESA as sponsors; Astrium (project manager) and EADS Airbus who bring space and avionic experience; Adelard with experience of assessment and certification across other domains; Critical Software with experience in software reliability, availability and maintenance techniques and methods; and DNV who are experts in system and software certification. STATE OF THE ART As mentioned previously the project studied three previous ESA study projects related to certification. It also studied current best practice across several nonspace domains. Previous ESA Projects PASCON WO6 was concerned with the analysis, specification, and verification/validation of software product assurance processes, and product metrics for reliability and safety critical software. This project defines a credible process assessment methodology based on a realistic notion of risk class for software. This is helpful in giving guidance as to appropriate development methods, metrics, etc to be applied for a particular software development (for a specific use) and gives an assessment method to demonstrate suitability of a given process. It is relatively prescriptive and could be expensive to apply, especially if more than one assessment was necessary (i.e. if software is developed to different standards). It does not cover non processbased assurance evidence such as proven in use arguments for COTS components. One problem with the risk class approach is that it depends fundamentally on the proposed end use of the system. This may be difficult where an application for a space system is proposed after a particular system has been designed and manufactured. There are also situations where the precise nature of the system s vulnerability to the software failure modes may affect the risk class: e.g. a complete failure may be tolerable (by reverting to an alternative), but incorrect output is a critical hazard. If a system is constructed knowing which are the most critical failure modes, then design defences can be included, e.g. sanity checks or watch dogs, which reduce the reliance on the correctness of the software. SPEC proposes a certification scheme that is primarily focused on evaluating product quality through a set of metrics. There is a slight extension to cover process so that it is compatible with PASCON WO6. It is highly detailed and prescriptive in application. By analysing occurrences of quality properties across a number of standards (key ones being ISO 9126 [9], PSS [3] and NASA SATC [5]) a number of key properties were identified. An additional set of properties was developed by the SPEC project itself. A report then defines each property and associated metric and evaluation method. Five classes of software are defined: A (most critical) to E (least). It then maps properties and metrics against classes. A detailed annex provides a checklist for checking that evidence exists and is adequate. The on-going Galileo project consists of tailoring the two most relevant for software certification ECSS standards (E-40B [1] and Q-80B [2]) to add material covering things not in the standards, but deemed necessary. Added items include references to software safety cases, coding standards, tool qualification, and test coverage criteria. The underlying motive is to construct a flexible (i.e. safety case based) approach but also to provide a form filling guidance for the help of contractors. The project also defines how the safety case for Galileo might be developed giving outline claim-argumentevidence structures [14]. The project talks about negotiating the requirements in the tailored standard, but has not yet addressed the process for negotiating or assessing the safety case, or how the two aspects (standards based and safety case) will be merged. The above project reviews showed a spread of activity from defining the approach for a single project to specifying a detailed approach in a particular area. Although GNSS/Galileo has adopted some of the safety case approach, this is not seen in other ESA projects, and GNSS/Galileo is also developing project-specific standards based on the ECSS-E-40B [1] and ECSS-Q- 80B [2] standards. Consultation with regulators indicated that some felt that further development of standards per se is no longer helpful from their point of view and they prefer to issue regulations and guidance towards meeting their regulations. The regulations would typically be goal based rather than prescriptive, and standards can be employed to show that the goals have been achieved. One common thread is the use of classification schemes to categorise the criticality of different software and hardware components in order to determine the level of control and scrutiny to apply during the development process. Unfortunately the projects define different schemes, which again differ from those in commonly used external standards such as IEC [8]. This increases the burden of a certification scheme particularly for off-the-shelf components or previously assessed systems.

3 Previously developed (including COTS) software is recognised as a difficult area, but relatively little progress has been made in addressing this in the different projects. This is clearly something that the CARES project will consider in the proposed certification scheme. Prescriptive vs Goal-based Approaches Early safety standards tended to be prescriptive in their approach, taking the view that adherence to the requirements would be sufficient to ensure an adequate level of safety. Through the 1990s, however, a more goal-based approach has been gaining ground. In a goal-based approach, there is often separation of regulations and standards. Objective regulations are set and standards are then used to provide evidence that the regulations have been satisfied through the development of a safety case. In the goal-based approach, the Case is structured around a number of justification elements. The elements consist of: a claim about a property of the system or some subsystem; evidence that is used as the basis of the safety argument (which can be either facts, assumptions or sub-claims); and an argument linking the evidence to the claim. The inference rule is the means for deducing the claim from the evidence; safety claims may include functional correctness, reliability, availability, security, fail-safe response, supportability, etc. Arguments may be: deterministic or analytical: the application of predetermined rules to derive a true/false result (given some initial assumptions), e.g. formal proof, execution time analysis, exhaustive test, demonstration of the single fault criterion probabilistic: quantitative statistical reasoning to establish a numerical level of MTTF, e.g. reliability testing qualitative: the compliance with rules that have an indirect link to the desired attributes, e.g. compliance with quality and safety standards, maintenance of staff skills and experience Examples of sectors where there is a strong move to goal-based certification are civil aviation and nuclear power generation. These domains were among those studied by CARES. Techniques and Methods Relevant to Space Software The techniques and methods depend on the class of service that is needed for safety e.g. to maintain service, degraded service or no service but safe state. Different classes of services are required in various domains as shown in Table 1. Service class Nuclear Railway Medical Defence Process Auto Aero Space Maintain service Post-trip decay heat removal Route interlocking Drug infusion Pace-maker Engine control Fly by wire Exothermic reactions Steering Engine control Engine control Fly by wire Degraded service (partial or manual) Monitoring and control Radio contact to driver Anaesthetics Patient monitor Command and control Stable processes Instruments Manual command Spacecraft survival mode Table 1: Service classes for safety No service but safe state Reactor shutdown Signals red Train stopped Weapon disarmed (peacetime) In conducting the study, a range of techniques and methods were identified, along with their use across multiple domains, and with different levels of criticality. These approaches were also mapped against the purpose of the technique in reducing of managing risk. Figure 1 shows the lifecycle mapping of the broad categorisation used. analysis Fault avoidance Fault detection Requirements fault defences Design fault defences Implementation process defences There are two types of safety evidence: direct evidence-consists of quantitative evidence, e.g. statistical testing, formal proof (i.e. showing the absence of faults) and operational data; it also covers qualitative measures that it is reasonable to believe incrtease safety integrity, but in an unquantifiable way (e.g. design reviews) underpinning or backing evidence-measures that imply that the direct evidence is trustworthy, e.g. effective configuration management, or a comprehensive fault reporting system Failure Detection Monitoring and feedback Failure Containment Figure 1: Techniques Run-time defences To rectify faults

4 The techniques were initially identified from a range of sources, including PASCON, IEC [8], and the EWICS technique directory [13]. An initial set of over 100 techniques was identified were analysed for their actual use across a range of sectors including: Nuclear, Railway, Medical, Defence, Process, Automobile, Aviation, and Space. Technique Domain Nuclear Railway Defensive Programming S S S S Fail Safe Bias S A S S? Fault Tolerance (redundant channels, or A S S S S? Sy hot standby ) Graceful Degradation S S S S Sy S N-version Programming S S Manual Override S S S S S Sy A Structuring the System according to Criticality A A S S S S A Segregation/partitioning S S S S S S S S Wrapping (usually applied to COTS) Temporal redundancy, re-execution on error A: Always, S: Sometimes, Sy: At system level Table 2: Failure containment techniques current usage This analysis was furthered by considering these techniques suitability for use on systems at different levels of criticality, and whether they were specifically relevant to software and appropriate for certification activities. Fault Detection Testing: Applicability Non safety related Medical related Defence Process critical Auto Aero Relevant to software S Space S Relevant to software certification Testing coverage based Maybe Maybe Yes Yes Testing - module Maybe Yes Yes Yes Yes Testing requirements based Yes Yes Yes Yes Yes Testing operational profile Maybe Maybe Yes Yes Maybe Testing stress testing Maybe Maybe Yes Yes Yes Testing regression Maybe Yes Yes Yes Yes Test Adequacy Measures (test coverage) Other techniques: Maybe Maybe Yes Yes Inspections and Walkthroughs Yes Yes Yes Yes Yes Static analysis control and data flow Maybe Yes Yes Yes Static analysis semantic Maybe Yes Yes Simulation Maybe Maybe Yes Maybe Schedulability analysis Maybe Yes Maybe Signature Analysis, memorising executed cases Maybe Yes Maybe Table 3: Failure containment techniques usage This analysis allowed us to eliminate a number of techniques because they were not currently used in practice, or they were relevant only at a system level, and had no bearing on software elements, or because they were not relevant to certification. The remaining 43 techniques or methods are currently being documented against a template which includes information such as: Rationale Detailed description Standard or certification schemes that reference it Constraints on use Training or skill required by practitioners Cost of use Quality of evidence generated Applicability of the technique Category of software recommended for use Support tools Related metrics Complementary methods and techniques Similar method and techniques While a zero fault target should always be considered, it may only be practicable for certain systems or application where the system is simple such that its test coverage is complete, or an accurate formal model can be built to 'prove' correctness. For space system, where control and safety functions are likely to be mixed, this target is unlikely to be achievable. Instead the ALARP principle may be considered. This is a risk based approach. Figure 2 illustrates the principle. Intolerable level Risk cannot be justified on any grounds The ALARP region Risk is undertaken only if a benefit is desired Broadly No need for detailed acceptable working to demonstrate region ALARP Tolerable only if risk reduction is impracticable or if its cost is grossly disproportionate to the improvement gained Tolerable if cost of reduction would exceed the improvement gained Negligible risk Figure 2: The ALARP Principle Some risks are too low to be worth considering in detail (bottom area in Figure 2), although the claim that a risk is really this low needs careful justification. Some risks are too high to be acceptable, and must be completely eliminated (top area). Intermediate area risks are deemed acceptable, but only if they are mitigated to be as low as reasonably

5 practicable. This is usually done using a cost model, where the cost of mitigating the risk is offset against the cost (e.g. in value of lost human life or environmental damage) of the resulting accident. This is a common approach across many sectors (including transport, which is likely to be the first critical use of space systems where the public is exposed to space system failure). It therefore may be an appropriate technique for space systems. A GENERIC CERTIFICATION SCHEME Based on the analysis of the software dependability and safety related methods and techniques, the objectives of the CARES project is to elaborate a framework for certification of software for space systems. This work is taking into account the software certification framework proposed by the GNSS-2 study for the Galileo Space System [4], which includes a tailoring of the ECSS standards and the definition of additional certification related processes based on the safety case approach. The aim is to develop software safety and certification related activities by extending existing engineering and quality assurance processes as currently practised for space software. certification needs. In this respect, the safety case approach is a potentially efficient mechanism in which to gather all needed safety and certification related information, collected from the engineering and quality assurance processes, and thus providing certification evidence, but building upon existing processes. It should therefore support an evolutionary development from existing practice and standards. STAKEHOLDER ROLES AND ACCREDITATION of an emerging certification scheme will be a requirement. However the CARES project is addressing only software relevant aspects of certification. Overall certification must take place at the system level. This is considered outside the scope of the project. Instead CARES will define software certification and related accreditation requirements, which will feed forwards to the overall system certification and accreditation. Figure 4 outlines a detailed model of certification which emerged from our study. We elaborate this below with a description of (some) of the functions, along with observation and where appropriate directions for the remainder of the project This results in a proposition of adaptation of the current software space standards (ECSS), organised with the same structure. For each of the identified processes (system engineering related to software, software management, software requirements engineering, software design engineering, etc.), the existing requirements will be given, with proposed modifications when needed, as well as specific additional requirements related to the fault prevention, fault removal, fault tolerance and fault forecasting process. Finally, as shown in Figure 3, the space standards, tailored according to certification specific requirements, will be applicable to both the software engineering process and a parallel certification related process. development committees interpretation bodies Management Overall management Assessor competence evaluation Second party certification Self-certification Change mechanism standards EN45000 bodies Third party certification Supplier/operator liaison Sector regulators Regulatory objectives Independent safety assessor /assessment authority case Development process Customer Regulator Figure 4: /accreditation model standards Tailored ECSS ECSS Figure 3: generic framework This approach allows for minimum and consistent adaptations of the current software engineering and product assurance standards for space software, bringing them into compliance with requirements from This body is responsible for allowing the space system containing the software into service. It receives evidence from the supplier to support the application for approval. It also needs to monitor and influence the other elements of the scheme to ensure that the adequacy of the safety justification evidence it produces.

6 In a goal-based regime the regulator defines regulatory objectives, and suppliers/operators use appropriate standards to show conformance to the regulations. Such objective based requirements allow industry to develop whereas prescriptive requirements capture what has happened in the industry. Regulators do not approve or authorise : liability remains with the relevant operator or supplier. Instead the regulators adopt an auditing approach, typically taking a broad look at the whole system and then focusing in depth on one or two areas. requirements are seen as very important. It was clear from speaking to various regulators that space software would have to be regulated separately in all the sectors (e.g. civil aviation and railways) in which it was used. Therefore the CARES project may not wish to attempt to identify a single regulator, but instead to set up a certification framework to provide the necessary evidence that regulatory goals are met in likely regulatory regimes. Independent Assessment Another important trend discovered in consultation is towards the use of independent safety assessors (ISAs). The first sector standard to define an ISA role was the UK Defence Standard [12] on safety management. Over recent years, the ISA has become increasingly important in the UK defence and some civilian sectors In this environment, the ISA becomes a quasiregulator whose advice is relied upon by the project team. The ISA role is a mixture of audit, safety case review and independent technical analysis similar to that for the regulators described above The role of ISA is less well defined elsewhere in Europe although several companies offer similar services. This is probably because safety assessment has historically been more vertically integrated, with less subcontracting of safety assessment. However, this is beginning to change and we are aware of independent safety assessment of nuclear plants in Germany and Sweden and of some railway applications Management This body oversees and promotes general operation of the certification scheme. In some areas, there is no explicit management function but the accreditation bodies take on some of the roles, for example defining test suites, adjudicating on standards interpretation, and raising issues with standardisation bodies. This would be a body that undertakes impartial assessment of space software safety and reliability evaluation services against published standards and criteria. The existing national accreditation bodies are obvious candidates for this role, but space-specific organisations are also possible. The standards for accreditation are the EN [11] series. Very few accreditation bodies currently offer software accreditation. UKAS is the most advanced because of its involvement in the CASS scheme. Suppliers and Operators System suppliers we consulted in the automotive sector stressed the use of independent safety assessors. Although this is not mandatory in that sector, it is seen as a protection. The UK makes most use of ISAs at present but there is increasing use elsewhere in Europe. One approach used by software component supplier is to produce components that are certifiable but not actually certified, as the core components have been designed to be easily assessed in conjunction with application software. Supplying a system to customers that is specifically geared up for certification means that the total cost, including certification, for customers is much lower than for a raw system. The operators we consulted supported the separation of regulation and standards. One nuclear operator stressed the need for categorisation of safety systems. The use of a prescriptive regime for the highest category results in the regulatory requirements being met with little problem as long as the standards have been intelligently applied with the regulations in mind (although as safety can usually be separated from control in this domain the most critical category of systems are usually very simple). With lower criticality system, a less prescriptive regime allows the use of off-the-shelf software and gives freedom to derive an appropriate safety case that will satisfy the regulations. CONCLUSION The CARES project has so far achieved a good understanding of existing and emerging space software assessment practices and standards. It has also surveyed best practice and methods across other domains and consulted over emerging trends such as the move towards goal-based regulation. We are currently working towards systematically documenting techniques and methods relevant to space software certification, as well as building a catalogue of certification organisations. By the end of the project (April 2003) we will have produced a certification and

7 accreditation framework for software which will have incrementally built upon current space approaches and (ECSS) standards and including best practice from other domains. This evolutionary approach will simplify uptake amongst stakeholders and contribute to overall space system certification. REFERENCES 1 ECSS-E-40B Software Engineering-Software 2 ECSS-Q-80B Space Product Assurance: Software Product Assurance 3 ESA-PSS Guide to Software Quality Assurance for ESA Space Systems 4 GNSS Galileo Software Guideline, Study of GNSS-2/Galileo System Software, TN4, Part 2 Issue 1.1, 29/3/01 5 NSS NASA Software Standard February PASCON/WO12 TN1 RAMS related software requirements and design constraints 7 SPEC TN 4 Overview of existing software certification scheme (part1,2,3) 8 ISO/IEC Functional safety of electrical/electronic/ programmable electronic safety-related systems 9 ISO/IEC 9126 Information technology - Software product evaluation Quality characteristics and guidelines for their use. 10 ISO 9000 Series, Quality management and quality assurance 11 EN series, standards. 12 UK Ministry of Defence, Management Requirements for Defence Systems, Defence Standard 00-56/Issue 2, December Dependability of Critical Computer Applications 3: Techniques Directory, P.G. Bishop (ed), Elsevier Applied Science, ISBN , ASCAD - The Adelard Case Development Manual, ISBN