5 Certifiable safe airborne software process analyses

Certifiable safe airborne software process analyses 97 5 Certifiable safe airborne software process analyses Published as E. Kesseler, Applying theory to practise, Airworthy software measured and analysed, 16 th IFIP World computer congress, Proceedings of conference on software theory and practice, 21 26 August 2, Beijing, China. ISBN 7-553-611-4. Abstract Software which is embedded in aircraft to which people entrust their life becomes safety-critical and consequently safety is an uncompromisable requirement. Failures of such software must be virtually non-existent. Due to the high costs of aircraft, hypothetical software failures would also incur major financial losses. To guarantee aircraft safety, an independent government agency certifies the aircraft, including its software, as fit-for-use. The world-wide-standardised avionics software certification requirements deliberately allow considerable freedom for compliant software development processes. The first two versions of the discussed application have successfully passed the certification and well over 1 units are currently being flown in numerous aircraft. In fact the software product is so successful that it will be adapted to other aircraft models. Based on product and process metrics, the software development process will be analysed, applying theory to practice. Aspects covered include requirements, code size, verification, effort, and project dynamics. 5.1 Introduction To put the software process into perspective, this chapter starts with a cursory overview of the application. A succinct description of the safety requirements follows to appreciate its influence on the software development process, which is defined in the subsequent section. The main part of the paper analyses the experience gained, using software metrics where appropriate. The switch to a spiral software development model proved beneficial, however the CASE tool could not adequately support the resulting changes. A specification method more suitable for all actors involved, pilot, aircraft experts and software experts, would reduce the requirement evolution. Applying and enforcing a strict coding standard, C can be

98 Certifiable safe airborne software process analyses used for safety critical applications. The test automation paid off. The conclusions summarise these and some other findings. 5.2 Application overview The embedded application discussed combines, controls, processes and forwards the data between the aircraft sensors, all pertinent aircraft subsystems and the flight deck. Configuring the embedded application to a specific aircraft is done by configuration files. This solution reduces the certification effort for each aircraft to a validation of the data files involved. The embedded application is designed to operate in both Visual Meteorological Conditions (VMC) and Instrument Meteorological Conditions (IMC). The latter conditions allow all-weather operations of the aircraft. Under these conditions the displays of the flight deck are needed by the pilot to fly, rendering the application safety-critical. During normal operations the embedded application processes about 1 different flight parameters, originating from 1 different sensors and aircraft subsystems, some of which are duplicated. Two processors are used in each of the duplicated hardware units. The delay times within the entire embedded application should be guaranteed to be less then 3 milliseconds with a cycle time of 25 milliseconds for the main processor. Aircraft have an economic life of several decades. Consequently 5% spare processor time is reserved for the many extensions of the embedded application expected during its operational life. The I/O processor has a cycle time of 36 microseconds. As an illustration of the paramount influence of safety on the embedded application's functions, for all input data the software performs quadruple data validation: Coherency test: a check on correct length and parity of the data; Reception test: a check on the timely arrival of the data; Sensor discrepancy test: a comparison the two independent redundant sensors; Module discrepancy test: a comparison between the two parameter values produced by the same sensor; one value directly read by the system from the sensor, and one obtained from the redundant system via a cross-talk bus.

Certifiable safe airborne software process analyses 99 (Kesseler, van de Sluis 1998) contains more information on the application. 5.3 Overview safety requirements 5.3.1 Applicable software safety requirements For safety-critical software in airborne equipment (DO-178B 1992) provides guidance for both the software developers and the certification authorities. The latter, an independent governmental institution, performs the ultimate system acceptance by certifying the entire aircraft, including its embedded software, i.e. releasing it for commercial use. In an international industry DO-178B provides the important world-wide protection of the air traveller as well as a world-wide "level playing field" for the competing industries. DO-178B was the first widely used document to address safety-critical software. Based on amongst others the experience gained with this document, currently other more general-purpose standards are available, like (ISO/DIS 1526 1996) and (IEC-6158 1995). 5.3.2 Safety classification Based on the worst-case consequences of a software failure, the software is classified into 5 levels A to E. The Federal Aviation Regulation /Joint Aviation Requirements FAR/JAR-25 uses the general principle of an inverse relationship between the probability of a failure condition and the degree of hazard to the aircraft or its occupants. As DO-178B considers qualitative demonstration of software compliance to such high reliability to be beyond the current software technology, the FAR/JAR-25 failure probability in flight hours can be provided for information only. For more information on DO-178B and its classification please refer to the section 1.2.2 of the introduction.

1 Certifiable safe airborne software process analyses 5.3.3 Software life cycle In order to provide the developer with maximum flexibility, DO-178B allows the developer to choose a complying software life cycle. It enforces traceability to its general processes listed below: Software planning process which includes the production of the software development plan and the software verification plan; Software development processes split into: a software requirement process, a software design process, a software coding process and integration processes; Integral processes comprising verification, configuration management and quality assurance. The integral processes are a result of the criticality of the software. Consequently the integral processes are performed concurrently with the software development processes throughout the entire software life cycle. 5.3.4 Verification Each process within the chosen software life cycle has to be traceable, verifiable and consistent. DO-178B defines verification as "the evaluation of the results of a process to ensure correctness and consistency with respect to the inputs and standards to that process". Verification can be accomplished by any combination of: Review, which provides a qualitative assessment of correctness; Analysis, which is a repeatable process, optionally supported by tools; Test. For software classified at DO-178B level A, the mandatory 1% code coverage consists of: Statement coverage (every statement executed, called statement testing in (BS 7925-2, 1998); Decision coverage (every decision executed for pass and fail, called branch/decision testing in BS 7925-2);

Certifiable safe airborne software process analyses 11 Modified condition/ decision coverage (MC/DC, same name in BS 7925-2). MC/DC requires that for every condition in a decision, its effect on the outcome of the decision be demonstrated. Code coverage will be shown at module level testing. In an iterative software process model each delivery requires full regression testing. 5.4 Software development process The definition of the software development process has been guided by previous experience with safety-critical software for spacecraft. (Dekker, Kesseler 1996) contains more information on the spacecraft application. (Park et al., 1996) puts great emphasis on metrics definition in order to allow others to interpret and repeat the results. The following software process definition provides this context, including all items of the basic measurement set of (Gaffney 1995). The project team was set up consisting of 2 separate groups, a development group and a verification group. In compliance with DO-178B the verification team leader was independent. Furthermore the quality assurance manager was independent from both teams and not allowed to produce deliverable code or tests. In this high-tech project the quality assurance manager needed his technical background in order to arrive at its impartial judgements. The embedded application project started using: A (DOD-STD-2167A 1992) waterfall based life cycle model; Customer supplied requirement specifications in plain English; Formal reviews after each life cycle phase; Software analysis using Structured Analysis with Hatley and Pirbhai Real Time extensions (SA/RT) (Hatley, Pirbhai, 1988) supported by the Teamwork Computer Aided Software Engineering (CASE) tool; Software design using Yourdon Structured Design (SD) supported by the same CASE tool;

12 Certifiable safe airborne software process analyses The customer prescribed C language; Module and integration testing on the target system with a simulated environment; The Cantata automated test tool to aid the construction and cost effective regression testing of the functional tests and code coverage tests; A proprietary configuration management tool; Integration with the aircraft avionics suite after integration of the embedded application; All key project team members had at least 5 years experience in aerospace applications. 5.5 Experience 5.5.1 Software classification In the embedded application, software classified at DO178-B levels A, B and E has been realised. At project completion 81% of the modules were classified at level A, 8% at level B and 11 % at level E. The increasing number of data fusion requirements lead to a larger share of level A software at the expense of level B software. With the small amount of remaining level B modules it is unclear whether the advantages of less rigorous testing of level B software outweigh the more complicated software development process. When software classified at different safety levels has to run on the same processor, special partitioning software guarantees that software of one level can under no circumstance compromise the functioning of software at other levels. This partitioning software took only 1% of the total project effort. It remains necessary and cost effective for separation of level A and level E (mainly maintenance) software.

Certifiable safe airborne software process analyses 13 5.5.2 C language The C language with its numerous unspecified constructs (Hatton 1995) is considered a project risk. This risk was contained by choosing an ISO C-9 (also known as ANSI-C) compliant compiler complemented by a proprietary coding standard enhanced to define a safe subset of C. A customised commercial static analysis tool can automatically verify compliance to this project coding standard. DO-178B requires tool verification, which revealed that the version management by the tool supplier turned out to be inadequate. The tool was already marketed worldwide since 1986 to hundreds of customers. This illustrates the rigour of the applied verification processes. 5.5.3 Metrics selection The Goal Question Metric paradigm (Basili, Weis 1984) states that business goals should drive any measurement effort. Some (Solingen, Berghout 1999) take this approach so far as to abolish any notion of a standard measurement set. Others (NASA 1995), (Gaffney 1995) define a basic measurement set consisting of size, cost, schedule, quality and project dynamics. This basic measurement set is used to analyse the safety-critical software development process and compare it with some results from literature. 5.5.4 Requirements evolution Previous experience with safety-critical software suggested a waterfall software development model. The commercially defined short time-to-market induced codevelopment of the application with a customer developed display application. This co-development required an early prototype. Note that laws prohibit the use of intermediate (i.e. non-certifiable) versions in flying aircraft. The first prototype served its purpose well. Consequently the project changed to an incremental or spiral software development model. The spiral model allowed full compliance with all high-level programme milestones. Figure 5-1 depicts the resulting requirements evolution. The time axis starts at project start. Every point indicates a formal delivery to the customer. Figure 5-1 is cumulative: the number of partially implemented requirements ( partial in Figure

14 Certifiable safe airborne software process analyses 5-1) is added to the number of fully implemented requirements ( Ful ). Together with the not yet implemented requirements ( Not) this yields the total number of requirements. The number of requirement changes is superimposed. New issues of the requirement document caused the increase in the number of requirements and the reduction in the number of implemented requirements after 3 and 52 calendar days. Requirements 25 2 15 1 5 1 2 3 4 5 6 7 8 Calendar Days Full Full+Partial Full+Partial+Not Changes 5 4 3 2 1 Changes Figure 5-1. Requirements evolution. The changes are caused by (in descending order): Changes in the Human Machine Interfaces (HMI) of the aircraft. These changes originate from pilot comment and can only be obtained from demonstrating a working prototype in a realistic environment; Adding product features. Apart from marketing input, these changes also result from experience with an actual prototype; Integration of the embedded application with the displays and the aircraft. Formal interface specifications might have helped to reduce this class of changes; Ambiguities in the plain English specifications. Especially for HMI related features an unambiguous specification method which is intelligible for pilots, HMI experts and computer software experts is needed.

Certifiable safe airborne software process analyses 15 Of the 15 deliveries only the ones at 655 and 779 calendar days have been certified. 5.5.5 Design evolution The CASE tool used only allows to progress once from analysis to design and once from design to code. It provides inadequate support to incorporate model updates into the next phases. By its nature the spiral model requires frequent updates of the analyses and design models. The amount of effort needed for data input even makes updating the analysis or design model cumbersome. After day 57 it was decided to retain the analysis model but limit its depth in order to allow for its updating. The design model was abandoned, as the CASE tool data input effort became unaffordable with the requirement evolution. Instead pseudo code was added to the code. As a result design metrics spanning the entire project history have to be based on the implemented modules. Figure 5-2 shows the evolution of the number of modules (files containing C code) and external functions over time. 15 6 125 5 1 75 5 25 1 2 3 4 5 6 7 8 Modules 4 3 2 Functions 1 Calendar Days Modules External Functions Figure 5-2. Module evolution. Up until the first certified version the number of modules increased only slightly, indicating that all changes could be accommodated in the original design. Splitting some modules according to their safety classification, required for verification, caused the sharp rise in the number of modules just before the first certified version.

16 Certifiable safe airborne software process analyses This splitting was postponed until the first certified version as new data fusion requirements influenced the safety classification. Some optimisations for the second certified version resulted in a minor reduction in the number of modules. The number of external functions rose approximately continuously until the first certified version, in concert with the number of implemented requirements. The number of functions remained virtually constant for the second certified version. These two metrics reflect that the design remained relatively stable, most requirement changes could be accommodated in the existing modules. 5.5.6 Code size evolution 1 2 3 4 5 6 7 8 KLoC Calendar Days Pseudo Code Code Other Comments Figure 5-3. Code size evolution. The code size evolution is shown in Figure 5-3. This figure is cumulative, the amount of pseudo code is added to the amount of code, to which the other comments are added. Its continuous increase until the first certified version corresponds with the continuous increase in the number of implemented requirements. The subsequent slight reduction mirrors some optimisations. After day 57 the detailed design was converted to pseudo code. This pseudo code contains an abstract description of the code in about 27% of its size. Also all interface definition information was added in extensive headers per function. This extra information explains the considerable increase in the amount of comment before the first certified version.

Certifiable safe airborne software process analyses 17 The considerable requirement evolution is reflected in the changed source code metric. On average each line of executable code has been modified 13.4 times, each comment line only 4.1 times. Figure 5-3 shows that code size on its own is inadequate to measure the growing complexity and sophistication of the product as a result of the requirement evolution. 5.5.7 Code breakdown For testing purposes a division of statements had to be made into: Decisions and loops (consisting of the "switch", "if", "for" and "while" statements); Assignments; Data e.g. tables. 1 2 3 4 5 6 7 8 Statements Calendar Days Decisions Assigments Data Figure 5-4. Evolution of statement type distribution. The results are shown in Figure 5-4, again in a cumulative way, i.e. the amount of assignments is added to the amount of decisions. All statement types increase approximately continuously until the first certified version, with a slight decrease up till the second certified version. Experience gained from the various prototypes lead to more sophisticated solutions which comply with both the real-time requirements as well as with the requirements evolution. Just like the code size, the code

18 Certifiable safe airborne software process analyses breakdown does not reflect the added complexity and sophistication of the application. Similar figures displaying the evolution of complexity metrics like McCabe, Myer and Halstead also show no significant change in complexity. In order to manage this type of application, a metric is needed which does reflect the growing complexity of the product in order to relate it to the extra effort spend on it. Note that the doubling of the requirement document size does indicate the increased complexity. 5.5.8 Verification Due to the safety criticality of the product great effort is spent to ensure that no errors remain in the product. Consequently standard quality metrics like number of errors are not applicable. As an alternative the verification process metrics are complemented by some experience. Each testable requirement is identified to allow traceability from requirements through all development processes to verification. Every DO-178B compliant development phase contained full traceability of each requirement, using the requirement identification. This has greatly helped the management of the virtually continuous requirement evolution. A lesson learned is to allocate a separate identification to each verifiable part of a requirement. (Hayhurst 1998) reached this conclusion independently. The standard applied for the software requirement process has to be verified. Some simple tools can be produced to cost-effectively reduce the verification effort. The same holds for the design standard. For module tests the use of a Commercial Off-the-Shelf (COTS) test tool greatly reduced the time needed to prepare the tests and to perform the regressions tests for each delivery. The actual test code is generated from Test Case Definition (TCD) files. To satisfy the 1% statement coverage, 1% decision coverage and the MC/DC criterion (for level A software only), on average each safety-critical function is called 3.8 times during the verification tests. Due to this thorough testing the non-comment part of the test case definition files equals 2.9 times the noncomment size of the code. The size of the test descriptions necessitated adding test comment. The test comment grew to about 7% of the executable test case size. DO-178B requires data to be verified by inspection, only decisions and assignments can be verified by testing. For each testable statement 2 checks have been performed. For global data the test tool automatically checks that no global data is

Certifiable safe airborne software process analyses 19 inadvertently changed by the module, causing the large amount of checks per testable statement. Integration testing was based on the white box approach. It comprised the correct functioning of combinations of functions. During integration 184 tests have been performed. The COTS test tool did not support the multiple-module integration testing. During validation testing the requirements are verified using a black box approach. Several requirements can be verified in one test. The 132 black box tests verified 9% of the requirements. Integration tests verified 19% of the requirements. These requirements could not be verified by black box testing only. Examples of the latter are spare processor time and spare memory requirements. Analysis was used to verify 12% of the requirements. Note that some requirements can only be verified by a combination of analysis, validation testing and integration testing. Consequently the 3 percentages add up to more then 1%. Figure 5-5 shows which part of the requirements was verified by which combination of techniques. 18 16 Number of Requirements 14 12 1 8 6 4 2 Validation Validation + Integration Integration Integration + Analysis Analysis Analysis + Validation Figure 5-5. Requirement validation methods.

11 Certifiable safe airborne software process analyses 5.5.9 Documentation 1 9 8 7 Number of Pages 6 5 4 3 2 1 PSAC SDP SVP SCMP SQAP sw requirements sw design sw coding sw integration STD CAD VDD sw QA SAS Figure 5-6. Document volume. Figure 5-6 depicts the document volume in number of pages for the second certified version. The five documents of the software planning process (Plan of Software Aspects of Certification PSAC, Software Development Plan SDP, Software Verification Plan SVP, Software Configuration Management Plan SCMP, Software Quality Assurance Plan SQAP) constitute to 19% of the document volume. The four software development process documents make up 63% of the document volume. A large part of the design (47% of the project volume) is retrieved from the code, see section 5.4.5. The integral processes contribute five documents and 18% of the document volume, Software Test Description (STD), Version Description Document (VDD), software Quality Assurance (sw QA) and Software Accomplishment Summary (SAS). These figures demonstrate that the additional volume of documentation due to the certification requirements turns out to be quite modest in this case. Re-use of documents is limited to part of the software planning documents.

Certifiable safe airborne software process analyses 111 (Gaffney 1995) provides an estimate of the document volume based on the software size for a DOD-STD-2167A compliant life cycle. The actual requirement documentation and the test documentation size are less than 18% above these estimates. The actual design documentation size is more than twice this estimate. It is unclear whether this deviation for the design documentation is caused by the safety criticality of the product or the use of pseudo code in stead of a graphical design method. 5.5.1 Effort breakdown The unit of effort is 1 effective person month, i.e. one person working 3 calendardays full time. In a calendar year one person typically spends around two months on other activities like (public) holidays etc. First certified version (198 person month) Second certified version (37 person month) Plan PM QA CM Dev Ver Figure 5-7. Effort breakdown. For the first certified version the software planning process consumed 27% of the costs, 16% to produce all required planning documents (Plan in Figure 5-7) and 11% for project management (PM in Figure 5-7). This first implementation of a DO- 178B compliant process implies that all documents have been written from scratch. The switch from a waterfall process model to a spiral process model also added some effort. The software development process (Dev in Figure 5-7) took 41% of the total effort. The integral processes accounted for 32% of the effort, split in 23% for verification activities (Ver in Figure 5-7), 4% for configuration management (CM in Figure 5-7)

112 Certifiable safe airborne software process analyses and 5% for quality assurance (QA in Figure 5-7). The certification liaison effort was so small, that it is included in the quality assurance figure. The second certified version shows a dramatic shift in effort from software development to verification. In the first certified version a considerable amount of requirements were disabled using the configuration files, and hence had not been verified. Even though the test tool saved a lot of effort for the module tests, the integration and validation regression tests consumed a lot of effort. For mission-critical, embedded, real-time satellite systems (Vardanega 1999) provides effort estimates of 4%-45% for software development and 55%-6% for verification. Combining the total effort spent on both versions, 57% was spent on development and 43% on verification. This confirms that the module test automation paid of. For a wide range of aerospace applications (NASA 1996) reports 63% for software development and 37% for verification. Due to the test automation our experience comes close to the average reported for non safety-critical applications. For general software development (Gaffney 1995) reports 5% for software development, 17% for verification, 7% for quality assurance and configuration management and 26% for project management. Our 9% quality assurance and configuration management effort is only slightly higher. Surprisingly the 27% for integral processes, including the complete definition of the certifiable software development process comes close to his standard for project management. As expected the main difference is the great emphasis on verification for the safetycritical application.

Certifiable safe airborne software process analyses 113 5.5.11 Project staffing 16 14 12 Project Staff 1 8 6 4 2 1 2 3 4 5 6 7 8 9 1 11 12 13 14 15 16 17 18 19 2 21 22 23 24 25 26 Calendar Monts Out In #Persons Figure 5-8. Project staffing. Figure 5-8 illustrates that on average 11 physical people were deployed. Combining this with the total effort (Figure 5-7) shows all people to be full time allocated. However in total 36 persons contributed to the project. The average personnel turnover was 1.2 person per month with a strong peak between 18 and 22 months. A key success factor was that the design team leader and verification team leader participated from the beginning to the project completion. The DO-178B requirement for well-documented software processes eased the effects of the personnel turnover. The result of the extra capacity deployed after month 18 is reflected in the large amount of comment introduced in the code (Figure 5-3). 5.6 Conclusions For a certifiable, safety-critical product with a commercially determined time-tomarket co-development is a solution. The various prototypes provided by a spiral software development process allowed full compliance with all high-level programme milestones.

114 Certifiable safe airborne software process analyses To reduce requirement evolution there is a need for an unambiguous specification method that is intelligible for all actors involved, pilots, HMI experts, system experts and software experts. There is a need for a complexity metric based on the realised code. Code size, statement distribution and standard complexity measures do no reflect the increasing capabilities of the software. C combined with an appropriate coding standard and an automated analysis tool can be used for certifiable, safety-critical software. The additional software needed to run software of various safety levels on one processor is cost effective. The CASE tool used did not adequately support design updates rendering it incompatible with the spiral model. Automatically retrieving the required detailed design documentation from comment in the code works better. The COTS test tool significantly reduced the module testing effort. This is confirmed by comparing the measured effort distribution with data for comparable systems from literature. For integration and validation testing such support tools were not available. The additional volume of process documentation due to the certification requirements turns out to be quite modest in this case. Re-use of the software process documentation can lead to cost reduction Of the software product documentation, only the design documentation is well above the industry average for non safety-critical applications. A well-documented software process combined with the retention of the key personnel allowed to cope with significant project personnel turnover without adverse effects on product quality.