Introduction to Aviation Software Design Assurance

Transcription

1 Introduction to Aviation Software Design Assurance SYSTEMS CERTIFICATION AND INTEGRITY (SCI) DIRECTORATE OF AVIATION ENGINEERING (DAVENG) DGTA- Directorate General Technical Airworthiness

2 Introduction to Aviation Software Systems Certification and Integrity (SCI) - Directorate General Technical Airworthiness (DGTA) About this Course It s an introduction only Assumed knowledge: Little, if anything, about aviation software You know that software exists and that it is important to the This course will not make you an expert Course objective is to provide you with basic knowledge to enable you to: communicate with software engineers be a smarter customer when dealing with contractors 1

3 Agenda Start 0830 Introductions Introduction to Aviation Software Software Design Acceptance Software Development Software Assurance System Safety Break Integrating System Safety and Software Assurance Software Safety Software Requirements Lunch Lunch Reviews and Inspections Software Testing and Test Coverage Software Development CM, QA and Tools Software Compliance Findings Break In-Service Configuration Management Software Load Control In-Service Problem Reporting Mission Planning Systems Electronic Flight Bags Summary Finish 1630 Admin Mobile phones Fire Exits Toilets Course Critiques (training reform) Course Attendance List circulating Workshop and Quiz 2

4 Our Background Your Background Background Where do you work? What Software issues you currently face? What is your experience with software? What do you want from the course? Systems Certification and Integrity DD SCI Mr Mark Wade (03) SCI1 SQNLDR Vince Chong (03) Aviation Software, Systems Integration, Avionics Architecture, Electronic Flight Bags, Mission Planning Systems SCI2 Mr Stu Donaldson (03) Communications, Navigation, Electrical Systems, Electromagnetic Environmental Effects, Oxygen Systems SCI3 SQNLDR Fernando Lay (03) System Safety, Crash Protection 3

5 SCI-DGTA Roles Certification of new aircraft acquisition and Major changes to type design TAR Approval of SOR Tender Evaluation Assistance PDAS and CBD Approval Processing Issue Papers TAA Recommendations for SFP, AMTC, STC, SR Technology Specialists SCI1-Roles Identify appropriate airworthiness standards Interpret and apply airworthiness standards Assess the competence of a software development organisation Review Specs, SOWs, DIDs Tender Evaluations Assist with the Design Acceptance process Particularly compliance findings 4

6 SCI1-Roles Sponsor Training Intro to Aviation Software Aviation System Safety (5 days) FAA Avionics Certification Workshop (3 days) Software Reviews and Inspections (1 day) Software Testing (1 day) IEEE (3 days) Software Design Assurance via DO-178B (3 days) SCI1-Roles See the DGTA website for schedules

7 References AAP Section 2 Chapter 1 System Safety Chapter 7 Aviation Software Chapter 17 In-Service Management of Aviation Software Chapter 22 Electronic Flight Bags Chapter 24 Mission Planning Systems Standards DO-178B Software Considerations in Airborne Systems and Equipment Certification MIL-STD-882C System Safety Program Requirements SAE ARP 4754/761 Certification Considerations for Highly-Integrated or Complex Aircraft Systems MIL-STD-498 Software Development and Documentation IEEE/EIA Software Life Cycle Processes IEEE/EIA 1228 Standard for Software Safety Plans DEF STAN Requirements for Safety Related Software in Defence Equipment Guidance Material DO-248B Final Report for Clarification of DO-178B FAA Order Software Approval Guidelines CAST Position Papers Joint Software System Safety Committee Software System Safety Handbook 6

8 Introduction to Aviation Software Computer System Controls Sensors CPU Effectors Displays Memory Computer Program 1

9 Source Code Language Compiler or Assembler Compiled Modules Object Code Linker Computer Program Advantages Flexibility Durability Weight Software 2

10 Source Lines of Code (SLOC) int main () { int i; cout << "Please enter an integer value: "; cin >> i; cout << "The value you entered is " << i; cout << " and its double is " << i*2 << ".\n"; return 0; } 3

11 4

12 5

13 11,000 pg C MLOC 4ft 6

14 20,000 pg 11,000 pg C MLOC 4ft F-18E/F 1.1 MLOC 7ft 7

15 40,000 pg 20,000 pg 11,000 pg C MLOC 4ft F-18E/F 1.1 MLOC 7ft F MLOC 13ft 8

16 144,000 pg 40,000 pg 20,000 pg 11,000 pg 9

17 Millions of Lines of Code C-130J 99 F-18E/F 99 F B F

18 Aviation Software On aircraft software, Off-aircraft software with aircraft interface, Off-aircraft software with no interface but which has airworthiness or safety implications Is this Aviation Software? Picture of in flight entertainment 11

19 Is this Aviation Software? Picture of Mission Planning System Is this Aviation Software? Picture of EFB 12

20 Is this Aviation Software? Picture of ADASS Is this Aviation Software? Picture of ATC 13

21 Don t forget Aviation Software also includes technologies that are developed using software development paradigms Firmware: The combination of a hardware device and computer instructions or computer data that reside as read-only software on the hardware device [12207 & 498] 14

22 15

23 Why is software difficult? Flexibility?!? Continuity Testabiliity Complexity Flexibility Defects CHANGE 1 CHANGE 2 CHANGE 3 CHANGE 6 CHANGE CHANGE 5 4 Software (in practice) Software (in theory) Time 16

24 Complexity Continuity Output Structural Strength Electrical Power Test Predict Test Predict Predict Software Test Input 17

25 Testability Begin A B C D E F Loop 20x End click 18

26 Software related accidents happen 1 Wrong requirements 2 Confusing reliability and safety 3 4 Relying on redundancy Re-using software Wrong Requirements A320 Accident in Warsaw 2 dead, 54 injured, aircraft lost Braking Logic Reverse Thrusters: only when weight on both wheels (>12T) Spoilers: only when wheel speed >72 knots 19

27 Reliability and Safety Software Re-use Arianne-5 reused an Inertial Reference System (IRS) from Arianne-4. 5 Both at 33s Altitude 4 Horizontal Distance 20

28 Over-reliance on redundancy The Ariane 5 accident report notes that according to the culture of the Ariane program, only random failures are addressed and they are primarily handled with redundancy. Software Silver Bullets Formal Methods CMMI Development Standards Reviews and audits Metrics Extensive Testing CASE tools 21

29 The Four Software Airworthiness Requirements Software Dependability Cube Software Development 22

30 Software Assurance 1 Requirements 2 Design 3 Code 4 Test Systems Safety 23

31 Software Safety Operator Input Authority Limiter DFCS Sanity check outputs, don t send commands that are obviously wrong (Software Safety) Actuators Real World Sensors? f(t)? Real World Sanity check inputs, don t act on bad data (Software Safety) Independent System Make sure the software does what it is supposed to do (Software Assurance) 24

32 Software Assurance Software Safety System Safety Operating Procedures Error High Integrity if not detected Fault Tolerant if exposed Fail Safe Fault Failure Hazard Accident if dangerous external factors Minimise the number of faults in a system Implement functions to tolerate faults that might exist Detect failures and revert to a safe state Communicate limitations to operators The Four Software Airworthiness Requirements Software Dependability Cube 25

33 Software Design Acceptance SCI1 DGTA- What is Design Acceptance? A determination of technical acceptability by the Commonwealth. The cornerstone of technical airworthiness and fundamental reason for the DAR. Provides confidence that the product is safe and fit for purpose. By critical examination of design evidence. A process, not an act! SCI1 DGTA-

34 Design Acceptance Competence Specification Verification Certification SCI1 DGTA- A determination that: a competent organisation produced evidence to demonstrate that an aircraft design complies with an approved specification and is willing to stand behind the design. Design Acceptance Design Acceptance is a Commonwealth function that requires Commonwealth involvement in all four pillars. Cannot rely solely on competence. Cannot rely solely on certification. SCI1 DGTA-

35 Software Design Acceptance with sufficient OPPD to develop aviation software AEO 4 Software Airworthiness Requirements Functional Behaviours Assessment of Evidence (Software Compliance Finding) Evidence of Satisfaction of Airworthiness Requirements Evidence of Functional Behaviours Design Approval Competence Specification Verification Certification SCI1 DGTA- Pillar: Competence Aviation software that is part of a Major change to type design must be developed by an AEO. Or an exemption should be sought (note that such an exemption would be an exemption against AEO compliance, not an exemption against the requirement to establish the competence of the organisation). Aviation software that is part of a Minor change to type design should be developed by an AEO. Noting Regulation c. AEO requirements alone are not sufficient to demonstrate that an organisation is competent to develop aviation software. Need to look deeper at the organisation to determine whether plans and procedures define a development process that is likely to produce airworthy software. SCI1 DGTA-

36 Pillar: Specification Need to specify software functional requirements. i.e. what does fit for purpose mean? Also need to specify airworthiness requirements. i.e. what does safe mean? There are four airworthiness requirements that must be specified for any change to aviation software SCI1 DGTA- Pillar: Evidence The evidence pillar requires the design agency to produce evidence of requirements satisfaction. It is not sufficient just to say that a requirement has been satisfied, it must be proved. The Design Acceptance process requires that the Commonwealth review evidence of requirements satisfaction. Not a design review, a check to determine whether the evidence produced supports the claims of requirements satisfaction. To do this, the Commonwealth conducts compliance findings. SCI1 DGTA-

37 Pillar: Evidence A compliance finding is: an engineering decision based on relevant evidence that an aircraft design satisfies one or more airworthiness requirements. SCI1 DGTA- Pillar: Certification The software development organisation must issue a Design Approval certificate in accordance with Regulation The DAR should check for: the scope of use for which the design has been approved e.g. flight test or fleet release? any limitations, caveats or conditions SCI1 DGTA-

38 Recognition of Prior Acceptance SCI1 DGTA- To rely on informed RPA, must demonstrate the following: an aircraft design has been certified by a competent airworthiness authority, that aircraft design has some degree of configuration consistency with the design to be acquired by the, the role and operating environment in which the will use the design has some degree of consistency with the role and operating environment which the other airworthiness authority used as the basis for certification, and the other airworthiness authority did not retain any risks that would be intolerable to the. Software and RPA Identify evidence of certification Flight Clearance, Airworthiness Release, TSOA, TC, etc. Must be issued by a competent airworthiness authority. Demonstrate configuration consistency Compare Software Configuration Index or Version Description Document. May be differences at file level, may be differences in configuration data, may be differences at run time. Demonstrate role and operating environment consistency Look for differences in how the aircraft is flown. Generally the same process as for other technologies. Identify and incorporate procedures and workarounds Look at flight manual, problem reports. Identify retained risks Look at hazard log, problem reports, airworthiness instruments. SCI1 DGTA-

39 Common RPA Pitfalls Recurring Issue: RPA is planned, but then falls through. Contract does not enable four pillars (no access to data, no AEO requirements, no airworthiness requirements). No basis for Design Acceptance. SCI1 DGTA- RPA Trap Unique Functions Interference Interference USN Certified Software Interference Interference Functionality removed or altered by the USN. Unique CRE Interference Light Blue (Inner Circle) RPA Everything Else Oversight SCI1 DGTA-

40 Common RPA Issues Although the item is in service, it was never assessed by a competent airworthiness authority, only by a contractual authority. Slight CRE differences can have big effects: e.g. one pilot operation vs. two pilot operation e.g. different usage spectrum unique functionality. An airworthiness authority may have approved a design as suitable for flight test. The can t rely on this to approve the design for service release. Loss of control of schedule: can t release product until other airworthiness authority has provided certification SCI1 DGTA- Design Acceptance Strategy There are 3 alternatives to completing the Design Acceptance Process: satisfy the four pillars, Informed RPA, or Combination of both SCI1 DGTA-

41 Design Acceptance Strategy RPA to the maximum extent possible: where the configuration is consistent where the role and operating environment is consistent where retained risks are tolerable oversight (four pillars) for the remainder unique configuration unique functionality unique role and operating environment treatment of retained risks SCI1 DGTA- Commercial Contract For software, the contract must: impose the four airworthiness requirements, impose competence requirements (AEO plus delivery of software specific plans), require delivery of or access to sufficient data to support the compliance finding process, and provide access to software personnel when required. SCI1 DGTA-

42 FMS Case For software, the LOA should: require delivery of airworthiness artefacts, provide for support from airworthiness personnel, and provide access to or delivery of documents required to support the airworthiness process. SCI1 DGTA- NAA Oversight SCI1 DGTA-

43 NAA Oversight variations within each service) professional judgement used to determine whether software development and verification activities are sufficient may or may not align with DO-178B heavy reliance on structured flight test program (dedicated squadrons conducting structured flight test programs for 3, 6 or 12 months) SCI1 DGTA- NAA Oversight Similar to /FAA/EASA: DEF-STAN requires developer to propose suitable processes, DEF- STAN (now cancelled) used as a benchmark (requirement for formal methods at highest levels). No RPA. SCI1 DGTA-

44 Oversight Consistent with FAA/EASA: DO-178B or equivalent. The approach is reflective of our relative size SCI1 DGTA- NAA Oversight Difficult to resolve the above differences: If the focus is restricted to development and lab verification, there are software items that could be accepted under RPA but would be rejected if oversight was applied. Need to look at the bigger picture: USAF/USN do not rely as heavily on software assurance as the does. The three approaches are different, but not incorrect. There are many approaches to assuring software integrity. SCI1 DGTA-

45 Summary Design Acceptance is a determination of technical acceptability Software Design Acceptance Design Acceptance Strategies usually involve a combination of Informed RPA and oversight. Informed RPA may have limited value for software if the configuration is unique. The Software Design Acceptance Strategy and Compliance Finding approach must be determined pre-contract: SCI1 DGTA- SCI1 DGTA-

46 Software Development Software Engineering Concept System of Requirements Operations Validate Operational Use Software System Requirements Verify Software/ System System Test Test Subsystem Software Design Software Engineering Software Programming Verify Code System and Construction Compile Integration/ Subsystem Unit Test Test Software engineering is the set of controls placed around software programming to assure the right product is produced. 1

47 Software Life Cycle Software Development Capability For major projects with substantial aviation software development, an assessment of the Contractor s aviation software development competence should be conducted prior to contract or LOA signature 2

48 Software Development Standards MIL-STD Software Development and Documentation IEEE/EIA Industry Implementation of ISO/IEC 12207:1995 Would you like to know more? Introduction to IEEE12207 course 3

49 Key Point A comprehensive software development program conducted by a competent software development agency is essential to the successful development of safe aviation software 4

50 Software Assurance Definition The planned and systematic actions necessary to provide confidence and evidence that a product or process satisfies given requirements 1

51 The Idea The more important it is that the software should work, the more rigorous the development must be to make sure it does work. Development Vs Assurance 1 Requirements 2 Design 3 Code 4 Test 2

52 Principles of Software Assurance Requirements Satisfaction Have the required behaviours been translated into the product? Requirements Validity Does the software have the right behaviours? Requirements Traceability Are the behaviours of the software accounted for? Non-Interference Do other functions interfere with critical functions? Configuration Consistency Does the verification evidence have traceability to the product? Design Integrity Were good design practices adhered to? True or False Software Assurance makes software safe Software Assurance guarantees the absence of errors Software Assurance guarantees good requirements 3

53 Software Assurance Standards DO-178B DO-178C STANAG 4404 DEF STAN DO-278A for ATM systems DO-178B DO-178B - Software Considerations in Airborne Systems and Equipment Certification FAA recognised, preferred. Risk based approach. Known as ED-12B in Europe. DO-248B and FAA Order provide further guidance. 4

54 DO-178B DO-178B is not prescriptive. specifies outcomes, not tasks product, not process focus vendors are allowed to decide how objectives are satisfied Sets objectives for planning, development (requirements, design, code, test) integral processes (CM and QA). Objectives vary depending upon how software failures can affect system safety. DO-178B Software Level Level A Level B Level C Level D Level E 5

55 DO-178B Increasing Assurance Higher software assurance levels are achieved by increasing the degrees of rigour in: verification of requirements verification of software architecture test coverage configuration control independence of software verification process overlapping software verification process activities robustness testing verification of compliance with standards traceability Effort CM & QA Functional Testing Statement Coverage Traceability to Source Code Architectural Coverage Design Based Testing Software Design CM & QA Functional Testing Decision Coverage Traceability to Source Code Architectural Coverage Design Based Testing Software Design CM & QA Functional Testing MC/DC Coverage Traceability to Object Code Architectural Coverage Design Based Testing Software Design CM & QA Functional Testing Functional Requirements D Functional Requirements C Functional Requirements B Functional Requirements A 6

56 DO-178B example table Table A.5 Verification of Outputs of Software Coding and Integration Process DO-178B Chapter 6 DO-178B Chapter 11 7

57 Myth: DO-178B costs D C B A 1 Level A $$$$$$$ 2 3 Level B $$$ Level C $$ Cost 4 Level D $ Truth: DO-178B is expensive Windows 7 16GB of Disk Space 10 5 :1 Triplex Autoland System 150KB of ROM 1:10 4 $300 $10,000,000 Market Forces, Litigation How much dependability is enough? Safety Requirements, Government Regulations Not much Rigour? A lot! 8

58 Does it help? Things we know: Amount of software in aircraft has increased Criticality of software has increased e.g. FADECs, Digital Flight Controls. Accident rate has remained roughly constant (slight decrease) Key Point Development of safety-related aviation software should satisfy the objectives of either a TAR recognised software assurance standard or a software assurance matrix approved by the TAR 9

59 10

60 System Safety What is System Safety? It is a planned, disciplined, and systematic approach to identifying, analysing, eliminating and controlling hazards by analysis, design and management procedures throughout a system's life cycle 1

61 Basic Concepts Emphasises building in safety, not adding it on to a completed design Deals with systems as a whole rather than with subsystems or components System safety takes a larger view of hazards than just failures Basic Concepts Emphasises analysis rather than past experience Emphasises qualitative rather than quantitative approaches 2

62 Identify Hazards Analyse Hazards Continuous Hazard Management 3 Treat Hazards Civil System Safety 3

63 Concept Development Aircraft Functional Hazard Assessment Function Phase Failure Condition Failure Effect Class. Decelerate Aircraft on Ground Landing RTO Loss of Deceleration Capability on the Ground Crew is unable to stop aircraft on runway Cat. Decelerate Aircraft on Ground Landing Unannunciated Loss of All Automatic Stopping Functions Crew must use manual procedures to stop aircraft Maj. Aircraft Fault Tree Analysis Loss of Deceleration Capability on the Ground Loss of Thrust Reversers Loss of Effective Wheel Braking Loss of all Speedbrakes on a Contaminated Runway Loss of all Wheel Braking Preliminary Design Speedbrake System Thrust Reverser System System Functional Hazard Assessment - Brake System Function Phase Failure Condition Failure Effect Class. Wheel Braking Landing RTO Loss of all Wheel Braking Crew s ability to stop the aircraft on runway significantly reduced Haz. Auto Braking RTO Landing Unannunciated Loss of Autobraking Crew must use manual procedures to stop aircraft Maj. Preliminary System Safety Assessment FTAs Loss of all Wheel Braking Loss of Normal Braking Loss of Alternate Braking Loss of Reverse Braking 4

64 Loss of all Wheel Braking Loss of Normal Braking Loss of Alternate Braking Loss of Reverse Braking Hazard Software Hardware Human Likelihood of occurrence based upon software faults/errors? x10e-? Likelihood of occurrence based upon component failures? x10e-? Likelihood of occurrence based upon trainee individuals? x10e-? Hazard Severity Regardless of the hazard causal factors (hardware, software, human error), the severity of the hazard usually remains constant in most instances 5

65 Hazard Probability System Inputs Operating Time Hardware Systematic Failures Random Failures System Inputs Software Systematic Failures Conceptual Design Aircraft FHA Preliminary Design System FHAs Detailed Design Function Decelerate Aircraft on Ground Decelerate Aircraft on Ground Phase Landing RTO Landing Aircraft FTAs Failure Condition Loss of Deceleration Capability on the Ground Unannunciated Loss of All Automatic Stopping Functions Loss of Deceleration Capability on the Ground Failure Effect Crew is unable to stop aircraft on runway Crew must use manual procedures to stop aircraft Class. Cat. Maj. Function Wheel Braking Auto Braking Phase Landing RTO RTO Landing Speedbrake System Thrust Reverser System Brake System Failure Condition Loss of all Wheel Braking Unannunciate d Loss of Autobraking Failure Effect Crew s ability to stop the aircraft on runway significantly reduced Crew must use manual procedures to stop aircraft Class. Haz. Maj. Item FMEAs System FMEAs Electrical System Hydraulic System Braking System Loss of Thrust Reversers Loss of Effective Wheel Braking PSSA FTAs Loss of all Wheel Braking SSA FTAs Loss of all Wheel Braking Loss of all Speedbrakes on a Contaminated Runway Loss of all Wheel Braking Loss of Normal Braking Loss of Alternate Braking Loss of Reverse Braking Loss of Alternate Braking Loss of Manual Braking Loss of Reverse Braking 6

66 What is wrong with this? Hazard Consequence Probability level Incorrect GPS coordinates provided by the software The consequence of the mode of failure for this hazard is CRITICAL due to the potential significant degradation of mission capability. The probability of this mode of failure being experienced is IMPROBABLE as the software is certified by the United States National Geospatial Agency. Additionally, the software is currently in-service on the AF- CTS and with US Forces with no errors known. MEDIUM Key Point Do not assign likelihoods to software failures! 7

67 Integrating Systems Safety and Software Assurance Risk based approach The more important it is that the software should work, the more rigorous the development needs to be to make sure it does work. 1

68 Airworthiness Requirements S/W Assurance Paradigms in the System Safety Software Assurance ARP 4761/4754 DO-178B System Safety Software Assurance MIL-STD-882 Software Assurance Matrix 2

69 ARP 4754/4761 and DO-178B ARP 4754 System Safety Approach 3

70 Preliminary Design Speedbrake System Thrust Reverser System System Functional Hazard Assessment - Brake System Function Phase Failure Condition Failure Effect Class. Wheel Braking Landing RTO Loss of all Wheel Braking Crew s ability to stop the aircraft on runway significantly reduced Haz. Auto Braking RTO Landing Unannunciated Loss of Autobraking Crew must use manual procedures to stop aircraft Maj. Preliminary System Safety Assessment FTAs Loss of all Wheel Braking Loss of Normal Braking Loss of Alternate Braking Loss of Reverse Braking ARP 4754 to DO-178B ARP Detailed DAL assignment guidelines Failure Condition Classification Catastrophic Hazardous/Severe Major Major Minor No Safety Effect System Development Assurance Level A B C D E 4

71 Loss of all Wheel Braking Loss of Normal Braking Loss of Alternate Braking Loss of Reverse Braking Is it really that easy? Hazardous/Severe Major Major Failure conditions that could result in a large reduction in safety margins. Failure conditions that could result in a significant reduction in safety margins. 5

72 How does SCI-DGTA handle this? Look to FAA policy: ACs, TSOs, etc Best Reference: AC D Appendix 1 No published list for Part 25 (high value IP) MIL-STD-882C and DO-178B 6

73 MIL-STD-882C Task Title System Safety Program System Safety Program Plan Integration/Management of Associate Contractors etc System Safety Program Review/Audits SSG/SSWG Support Hazard Tracking and Risk Resolution System Safety Progress Summary Preliminary Hazard List Preliminary Hazard Analysis Safety Requirements/Criteria Analysis Subsystem Hazard Analysis System Hazard Analysis Operating and Support Hazard Analysis Task Type MGT MGT MGT MGT MGT MGT MGT ENG ENG ENG ENG ENG ENG Note that there are no software specific tasks! MIL-STD-882C - Hazard Severity Categories Description Catastrophic Critical Cat I II Definition Death, system loss or severe environmental damage. Severe injury, severe occupational illness, major system or environmental damage. Marginal III Minor injury, minor occupational illness or minor system or environmental damage. Negligible IV Less than minor injury, occupational illness or less than minor system or environmental damage. 7

74 Hazard Probability Levels (MIL-STD-882C) Description Cat Specific Individual Item Fleet or Inventory Frequent A Likely to occur frequently Continuously experienced Probable B Will occur several times in the life of an item Will occur frequently Occasional C Likely to occur some time in the life of an item Will occur several times Remote D Unlikely but possible to occur in the life of an item Unlikely but can reasonably be expected to occur Improbable E So unlikely it can be assumed occurrence may not be experienced Unlikely to occur, but possible MIL-STD-882C - Hazard Risk Index Matrix CAT CRIT MAR NEG Frequent Probable Occasional Remote Improbable Colours link to risk acceptance authority (e.g. AA, OAA, OAAR, DAR). Likelihoods cannot be correctly assigned to software causal factors: how then does software fit into the above matrix? 8

75 Software Control Categories Control Category Ia Ib IIa IIb IIIa IIIb IV Direct control. Description Displays information that is directly and constantly relied upon. Direct control but time for intervention by independent safety system. Displays information that if incorrect, may lead to operator action/inaction that progressively degrades safety margins. Control over hazardous systems, but operator action required to complete the task. Displays information about discrete events that require an operator to respond to prevent a hazard occurring. No control of safety related functions. Summary of Definitions from Section 2 Chapter 7. MIL-STD-882C to DO-178B CAT CRIT MAR NEG I II III IV Aircraft hazards will mostly fall into here SHRI DO-178B Level A B C D E Satisfaction of assurance level is the measure of hazard acceptability. 9

76 SHRI vs. HRI CAT CRIT MAR NEG Frequent Probable Occasional Remote Improbable MIL-STD-882C and MIL-STD

77 MIL-STD-882C to MIL-STD-498 Hazard Category CAT CRIT MAR NEG Control I II III IV SHRI MIL-STD-498 Level?????????? MIL-STD-498 does not define varying levels of development and verification rigour. Software Assurance Matrix 11

78 Software Assurance Matrix AAP Section 2 Chapter 7 Key point Software assurance relies on the identification of software s contribution to system level failure 12

79 Software Requirements SCI1 DGTA- What are requirements Specification of what should be implemented How the system should behave Application Domain Information Constraints on the System s Operation Specification of a system property or attribute Constraints on the development process SCI1 DGTA- 1

80 Requirements Engineering The activities involved in discovering, documenting, maintaining a set of requirements for a computer based system Systematic and repeatable techniques SCI1 DGTA- Inputs and Outputs Mission Needs Domain Information Regulation Previous Systems Information Requirements Engineering Process Agreed Requirements System Specification System Models SCI1 DGTA- 2

81 Requirements: The right way There is no magical answer or standard Some software development standards Requires a systematic approach to ensure goodness and quality SCI1 DGTA- Requirement Qualities Understandable Singular (not redundant) Complete Unambiguous Consistent Traceable Logically Grouped Conform to Standards Testable SCI1 DGTA- 3

82 Where do requirements hide? Requirements Evidence System Specification Aircraft Performance Specification Functional and Performance Specification Software Requirements Specification System/Sub-System Design Document Interface Requirements Specification Operational Functional Documents Prime Item Development Specification Contractor Item Development Specification DOORS Requirements Traceability Matrix SCI1 DGTA- Requirement Traceability System Specification High Level Requirements Low Level Requirements Code 5 Test Points REQ1 REQ2 REQ3.. REQ1:1 REQ1:2 REQ1:3.. REQ1:1.1 REQ1:1.2 REQ1:1.3.. #REQ1:1.1 Code DO. LOOP WHILE (x=y) TEST 1.1 TEST 1.2 TEST 1.3 TEST 1.4. SCI1 DGTA- 4

83 What to assess Traceability (more is required with increased software assurance levels) Requirements are accurate and consistent, verifiable, confirm to standards Algorithms are accurate SCI1 DGTA- Theoretical Case Study Radio Frequency Selection Mission Need We can t change the frequency on the VHF1 radio. SCI1 DGTA- 5

84 Sequence Diagram Radio Frequency Selection Sequence Diagram COMMS Panel 1 Mission Computer VHF Radio COMMS Status New Frequency selected Send new frequency Confirm new frequency set Update frequency information New Frequency selected Send new frequency Display error Invalid frequency rejected SCI1 DGTA- Use Case Diagrams Radio Frequency Selection Use Case Pilot Input VHF1 Frequency Cancel Selection COMMS Panel 1 Confirm Selection Display VHF1 Frequency On Status Page Send new Frequency to VHF1 <<uses>> Update VHF1 Frequency Data COMMS Panel 2 Input VHF1 Frequency Co-Pilot Cancel Selection Confirm Selection Tune VHF1 to new Frequency VHF1 SCI1 DGTA- 6

85 Software Requirements Specification Radio Frequency Selection Requirements High Level Requirements [COMM-0100] COMMS Panel 1 shall allow the operator to select a new frequency for the VHF1 Radio. [COMM-0200] COMMS Panel 1 shall allow the operator to input up to four digits with one decimal place using the keypad, with digits entered displayed on the scratchpad. Low Level Requirements [COMM-0101] COMMS Panel 1 shall display a confirmation request at SOFTKEY8 for selection of a new frequency if the VHF1 button SOFTKEY1 is depressed and the operator has input a frequency into the scratchpad. [COMM-0102] Following Operator confirmation of a new frequency for VHF1 by depressing SOFTKEY8 where confirmation is active, COMMS Panel 1 shall send scratchpad input as a new VHF1 Frequency to the Mission Computer (TO ADDR: 62FE, FROM ADDR: 013E) as an 8-bit unsigned integer. [COMM-0201] COMMS Panel 1 shall delete scratchpad input if CANCEL hardkey button is depressed. [COMM-0202] COMMS Panel 1 shall ignore keypad input if numeric keys are depressed if four digits are already displayed on the scratch pad. SCI1 DGTA- Workshop Ground Proximity Warning System Mission Need Provide us with an alert when we are rapidly approaching the ground!! Cocking Warning Indicator Radar Altimeter Mission Computer Inter-communications System SCI1 DGTA- 7

86 Sequence Diagram Ground Proximity Warning System Radar Altimeter Mission Computer INTERCOM WARNING Lamp SCI1 DGTA- 8

87 Software Safety 11 Directorate General Technical Airworthiness (DGTA-) Complementary Approaches Perform functional analysis Does the software satisfy the requirements? (Did we build the product right?) Assign software assurance levels Allocate software assurance objectives Software Assurance System and Software Safety Design, code and test functions to meet objectives Verify satisfaction of software safety requirements Safer System Define software safety requirements to treat hazards Identify software contribution to hazards Identify mishaps, hazards and failure modes Are the requirements appropriate? (Did we build the right product?) 12 Directorate General Technical Airworthiness (DGTA-)

88 Software Safety Requirements Safety-related software faults arise most often from: Discrepancies between documented requirements specifications and the requirements needed for correct functioning of the system. Misunderstandings about the software s interface with the rest of the system. Software-related accidents have occurred when the software satisfied its specification and when the operational reliability of the software was very high. Requirements specify behaviour that is not safe from a system perspective. Requirements do not specify some particular safety behaviour. The software has unintended (and unsafe) behaviour beyond what is specified in the requirements. i.e.: The software satisfies the requirements, but the requirements were wrong or incomplete. 13 Directorate General Technical Airworthiness (DGTA-) Software Safety Requirements Therefore: The identification and allocation of software safety requirements in the context of the system are key to the realisation of acceptably safe software intensive systems. That s all very well, but how does one go about it? Is there a formal approach? How do we understand software in the system context? I know how to write software that meets the specification, doesn t that mean it s safe? What other requirements do I need? 14 Directorate General Technical Airworthiness (DGTA-)

89 What is software safety? Software safety is the analysis of a software driven system that is conducted in order to identify safety requirements. The software shall (do something safe). The software shall not (do something dangerous). Software safety is not the assignment of software assurance levels. How does software safety differ from system safety? The focus: software safety considers each software item, system safety considers an entire system. Otherwise, no real difference. Similar techniques, similar goals. 15 Directorate General Technical Airworthiness (DGTA-) Identifying Software Safety Requirements Analyse the system and software in order to identify: requirements that detect and handle erroneous inputs to the software system e.g. sanity checks on input data requirements that detect and handle software faults or erroneous outputs from the software system e.g. output voting schemes requirements to detect and handle common software failures e.g. watchdog timer on processor activity 16 Directorate General Technical Airworthiness (DGTA-)

90 An Example We ll look at a rudimentary flight control system. Process: Identify system functions. Identify system level functional failure modes and associated severities. Identify potential software failure modes. Define software safety requirements to detect and handle or prevent software failure modes. Define generic software safety requirements. Verify that software satisfies allocated requirements. 17 Directorate General Technical Airworthiness (DGTA-) Identify Flight Control System Functions and Failure Modes 18 Directorate General Technical Airworthiness (DGTA-)

91 Diagram of Automatically Controlled Aircraft External Disturbances Actuator Dynamics Aircraft Dynamics Sensor Dynamics Discrete effects, computational delay AFCS Control Laws Mode Controller Reference Signals Pilot Interface 19 Directorate General Technical Airworthiness (DGTA-) Associated Aircraft Functions Control of Pitch, Roll and Yaw Control Lift and Drag Control Flap/Slat Spoiler Control Gear Position Automatic Control of Flight Stability Augmentation 20 Directorate General Technical Airworthiness (DGTA-)

92 How to identify failure modes? Consider: Loss of What happens if the system stops working and I detect that it has stopped working? Malfunction without warning What happens if the system continues to work, but does the wrong thing and I can t detect it? 21 Directorate General Technical Airworthiness (DGTA-) Failure Conditions Loss of Control of Pitch, Roll and Yaw Catastrophic. Hazardous for loss of one lateral axis. Control Lift and Drag Control Flap/Slat Major or Hazardous Spoiler Control Major or Hazardous Gear Position Major Automatic Control of Flight Minor (with warning) Major (without warning) Stability Augmentation Variable depends on flight characteristics of aircraft 22 Directorate General Technical Airworthiness (DGTA-)

93 Failure Conditions Malfunction without Warning Control of Pitch, Roll and Yaw Catastrophic. Control Lift and Drag Control Flap/Slat Hazardous or Catastrophic Spoiler Control Hazardous or Catastrophic Gear Position Major Automatic Control of Flight Major (single axis limited authority), Hazardous (multiaxis limited authority), Catastrophic (unlimited authority). Stability Augmentation Variable depends on flight characteristics of aircraft 23 Directorate General Technical Airworthiness (DGTA-) Likely Safety Critical Software Functions Any function which controls or directly influences the pre-arming, arming, enabling, release, launch, firing, or detonation of a weapon system, including target identification, selection and designation. Any function that determines, controls, or directly influences the flight path of a weapon system. Any function that controls or directly influences the movement of gun mounts, launchers, and other equipment, especially with respect to the pointing and firing safety of that equipment. Any function which controls or directly influences the movement of munitions and/or hazardous materials. Any function which monitors the state of the system for purposes of ensuring its safety. Any function that senses hazards and/or displays information concerning the protection of the system. Any function that controls or regulates energy sources in the system. Source: JSSSC SSSH 24 Directorate General Technical Airworthiness (DGTA-)

94 Likely Safety Critical Software Functions Fault detection priority. The priority structure of fault detection and restoration of safety or correcting logic shall be considered safety-critical. Software units or modules handling or responding to these faults. Interrupt processing software. Interrupt processing software, interrupt priority schemes and routines that disable or enable interrupts. Autonomous control. Software components that have autonomous control over safety critical hardware. Software controlled movement. Software that generates signals which have been shown through analysis to directly influence or control the movement of potentially hazardous hardware components or initiate safetycritical actions. Safety-critical displays. Software that generates outputs that displays the status of safety critical hardware systems. Where possible, these outputs shall be duplicated by non-software generated output. Critical data computation. Software used to compute safety-critical data. This includes applications software that may not be connected to or directly control a safety-critical hardware system (e.g., stress analysis programs). Source: JSSSC SSSH 25 Directorate General Technical Airworthiness (DGTA-) Identify Software Failure Modes and Software Safety Requirements 26 Directorate General Technical Airworthiness (DGTA-)

95 Aims of Software Hazard Analysis Identify software contribution to system level hazards. Identify software failure modes that are hazardous in the system context. May identify hazards that weren t identified at the system level. Define software safety requirements to prevent, or detect and handle, software failure modes. Produce evidence to support the system safety case. 27 Directorate General Technical Airworthiness (DGTA-) AFCS Guidance Functions Reference Signals + - Error Signals Inner-loop Controller Command Signals Aircraft Motion Variables Outer-loop Controller Flight Related Parameters 28 Directorate General Technical Airworthiness (DGTA-)

96 AFCS Software Functions Sensors (Control Column, Rudder Pedals) AFCS Control Laws Aileron Actuators Flight Management System (including Mode Controller) Altitude and Heading Reference System (Angular Rate or Yaw, Pitch Angle, Roll Angle, Yaw Angle, Heading, etc) Air Data Computer (Altitude, Airspeed, etc) Flight-Path Related Parameters Motion Variables Flight-Path Related Parameters Motion Variables AFCS Guidance Function 1 Outer-loop Controller Outer-loop Controller + Ref Signals + Inner-loop Controller AFCS Guidance Function n Ref Signals Error Signals -... Error Signals - Inner-loop Controller Command Signals Command Signals Elevator Actuators Rudder Actuators Flap Actuators Spoiler Actuators 29 Directorate General Technical Airworthiness (DGTA-) Software Safety Analysis Techniques Software Functional Failure Analysis Software Fault Tree Analysis Software FMEA (FMECA) Software HAZOP (DEF STAN Computer HAZOP) Software Hazard Analysis and Resolution in Design (SHARD) refinement of software HAZOP Markov Analysis and Data Flow Diagrams Petri Net Analysis Software Sneak Analysis and many more 30 Directorate General Technical Airworthiness (DGTA-)

97 Analysis Approach One technique in isolation will not be sufficient. There is no technique that is suitable for all situations. A combination of techniques will need to be selected for a particular application. There is no predefined or regulated solution. Intelligent thought is required. We ll have a look at SHARD. 31 Directorate General Technical Airworthiness (DGTA-) Software Hazard Analysis and Resolution in Design SHARD employs a series of guidewords to classify how the information flows and associated communication events (and associated services) might deviate from their intended forms. The guidewords are: Omission service not delivered. Commission service delivered when not required. Early service delivered, but early. Late service delivered, but late. Value service delivered, but with incorrect value. 32 Directorate General Technical Airworthiness (DGTA-)

98 SHARD SHARD requires that the system be analysed backwards from the outputs (i.e. identify the system level effects first) back towards the inputs. The internal and input deviations are expressed in terms of how they cause or contribute to deviations in downstream items already investigated. Additional causes Failure of the information flow itself Failure of the flow source Failure of the flow initiator 33 Directorate General Technical Airworthiness (DGTA-) SHARD Process 34 Directorate General Technical Airworthiness (DGTA-)

99 AFCS Software Functions Sensors (Control Column, Rudder Pedals) AFCS Control Laws Aileron Actuators Flight Management System (including Mode Controller) Altitude and Heading Reference System (Angular Rate or Yaw, Pitch Angle, Roll Angle, Yaw Angle, Heading, etc) Air Data Computer (Altitude, Airspeed, etc) Flight-Path Related Parameters Motion Variables Flight-Path Related Parameters Motion Variables AFCS Guidance Function 1 Outer-loop Controller Outer-loop Controller + Ref Signals + Inner-loop Controller AFCS Guidance Function n Ref Signals Error Signals -... Error Signals - Inner-loop Controller Command Signals Command Signals Elevator Actuators Rudder Actuators Flap Actuators Spoiler Actuators 35 Directorate General Technical Airworthiness (DGTA-) Flight Control System SHARD Command Signals Guide Word Deviation Cause Effect Detection/Protection Omission Command signals not applied to actuators when required. Programming error within Inner Loop Controller. Inner Loop Controller is not called. Aircraft actuator states remain unaltered. Aircraft may fail to respond to pilot or FMS direction. Inner Loop Controller to provide explicit output for each AFCS cycle. All exit points from Inner Loop Controller require an explicit update of command signals. AFCS cycles shall be explicitly synchronised to hard real time clock. Monitor to ensure Inner Loop Controller completely executes for each AFCS cycle. Commission Command signals applied to actuators when not required. Programming error within the Inner Loop Controller. Invalid process writes command signals. Inner Loop Controller is called when not required. Aircraft actuators move in unintended fashion. Aircraft responds incorrectly to pilot or FMS direction. Possible hardover or loss of control. Inner Loop Controller shall be the only process to write to command signals. Inner Loop Controller shall only be permitted to write control signals once per AFCS cycle. Early Command signals applied to actuators earlier than required. Programming error within the Inner Loop Controller. Inner Loop Controller is called earlier than required. Aircraft actuator movement leads intended output, perhaps resulting in flight control oscillations (pilot or system induced). Inner Loop Controller to provide explicit output for each AFCS cycle. AFCS cycles shall be explicitly synchronised to hard real time clock. Late Command signals applied to actuators later than required. Programming error within the Inner Loop Controller. Inner Loop Controller is called when not required. Aircraft actuator movement lags intended output, perhaps resulting in flight control oscillations (pilot or system induced). Inner Loop Controller to provide explicit output for each AFCS cycle. AFCS cycles shall be explicitly synchronised to hard real time clock. Value Incorrect command signals applied to actuators. Programming error within the Inner Loop Controller. Incorrect error signals passed to Inner Loop Controller. Aircraft actuators may move in unintended fashion. Aircraft responds incorrectly to pilot or FMS direction. Possible hardover or loss of control. Limit and reasonableness checks shall be performed on error signals passed to Inner Loop Controller. Limit and reasonableness checks to be performed on Inner Loop Controller outputs prior to writing command signals. 36 Directorate General Technical Airworthiness (DGTA-)

100 Software Hazard Analysis Software Requirements Apply SHARD to other data flows working backwards towards the input. Remember, each failure condition will now be expressed in terms of how they cause or contribute to deviations in downstream items already investigated. Continue to identify software safety requirements relating to detection and protection criteria. Repetition of detection and protection criteria (and necessary software safety requirements) is OK it is just establishing the importance of particular criteria. Present list of software safety requirements to System Safety Working Group to determine whether they are appropriate in the full system context. 37 Directorate General Technical Airworthiness (DGTA-) Identify Generic Software Safety Requirements 38 Directorate General Technical Airworthiness (DGTA-)

101 Generic Software Safety Requirements Generic software safety requirements serve a number of purposes: Implement Lessons Learnt Generic software safety requirements often evolve from accidents Treat Common Issues Many software systems share the same susceptibilities Not Constrained by Functionality Many software faults have no direct relationship to functionality, so a function based assessment might not identify them Check on Completeness of Software Hazard Analysis Helps determine whether software hazard analysis has identified enough software safety requirements 39 Directorate General Technical Airworthiness (DGTA-) Generic Software Safety Requirements System Design Power-Up System Initialisation Computer System Environment Self Check Safety Critical Computing System Function Protection Interface Design Requirements Human Interface Critical Timing and Interrupt Functions Software Design and Development Software Maintenance Software Analysis and Test Source: JSSSC SSSH 40 Directorate General Technical Airworthiness (DGTA-)

102 Generic Software Safety Requirements Safety Kernel ROM Safety kernels should be resident in non-volatile ROM or in protected memory that cannot be overwritten by the computing system. Inadvertent Jumps The system shall detect inadvertent jumps within or into safety critical functions; return the system to a safe state, and, if practical, perform diagnostics and fault isolation to determine the cause of the inadvertent jump. Decision Statements Decision statements in safety-critical computing system functions shall not rely on inputs of all ones or all zeroes, particularly when this information is obtained from external sensors. Source: JSSSC SSSH 41 Directorate General Technical Airworthiness (DGTA-) Leveson s Safeware Generic Software Safety Requirements State Completeness Input and Output Variable Completeness Trigger Event Completeness Robustness Criteria Non-Determinism Value and Timing Assumptions Output Specification Completeness Environmental Capacity Considerations Data Age Latency Output to Trigger Event Relationships Specification of Transitions Between States Reachability Recurrent Behaviour Reversibility Pre-emption Path Robustness Constraint Analysis 42 Directorate General Technical Airworthiness (DGTA-)

103 Leveson s Safeware Generic Software Safety Requirements The system and software must start in a safe system state. Interlocks should be initialised or checked to be operational at system start-up, including start-up after temporarily overriding interlocks. All incoming values should be checked and a response specified in the event of an out of range or unexpected value. Safety critical outputs should be checked for reasonableness and for hazardous values and timing. Reachable hazardous states should be eliminated or, if that is not possible, their frequency and duration reduced. 43 Directorate General Technical Airworthiness (DGTA-) Review Relevant Lists of Software Safety Requirements Review all relevant lists of software safety requirements and identify those that might be relevant to the system. This is as easy as stepping through the list with a highlighter in hand. Alternatively use a working group/team to conduct the activity. For example, for a flight control system as described in this presentation, a review of those generic requirements identified approximately 70 potentially relevant high level software safety requirements. 44 Directorate General Technical Airworthiness (DGTA-)

104 Verify that Software Satisfies Software Safety Requirements 45 Directorate General Technical Airworthiness (DGTA-) Software Verification Once the software safety requirements have been identified it must also be shown that the software satisfies those requirements. Verification techniques can be difficult, expensive or time consuming to apply. Therefore it is important to understand what techniques are most appropriate to what aspects of safety critical systems. Techniques include: Formal Methods Mandated by DEF STAN SIL 4 Model Checkers Static Analysis Dynamic Testing 46 Directorate General Technical Airworthiness (DGTA-)

105 Complementary Approaches to Verifying Safety Critical Systems Correctness by Construction To control the design and construction of a product to assure that requirements are satisfied. Static analysis, formal methods, etc. Correctness Through Test To prove that a developed product satisfies requirements by dynamically exercising functionality. Black and White Box Testing 47 Directorate General Technical Airworthiness (DGTA-) Static Analysis Analysis of source code before it is executed. Techniques include: Data Flow Analysis, Control Flow Analysis Symbolic Execution, Check of Source Against a Formal Mathematical Specification Checking against a Language Subset/Coding Standards Advantage if a property is shown to hold, it will hold for all scenarios. Disadvantage some software coding architectures and techniques do not easily lend themselves to static analysis. Recursion, pointers, etc May be difficult to employ retrospectively Static analysis is heavily reliant on software tools. 48 Directorate General Technical Airworthiness (DGTA-)

106 Dynamic Testing Execution of the code against numerous test cases at varying levels of integration. Because software is complex, it is not possible to execute test cases for all possible inputs and outputs. Techniques to reduce number of test cases include: Equivalence Classes Boundary Value Analysis Structural Testing (Statement Coverage, Decision Coverage, Modified Condition/Decision Coverage) Static Analysis and Dynamic Testing used effectively to complement each other provide deep insight into the behaviour of a software program. 49 Directorate General Technical Airworthiness (DGTA-) Software Aspects of Safety Case Implementation of System Safety Requirements Requirements to detect and handle other system failures or treat system level hazards. Detect and Handle Input Failure Conditions Software can detect and handle bad inputs. Detect and Handle Internal Failure Conditions Software can detect and handle internal failures without propagating to a system level failure. Integrity Where a software function is relied upon above, sufficient proof exists that the software satisfies the requirements. Completeness The system and software safety processes have completely considered the interaction between the software and the system. The most difficult part: necessitates a structured approach. 50 Directorate General Technical Airworthiness (DGTA-)

107 A Real Life Case Study 51 Directorate General Technical Airworthiness (DGTA-) AFTI F-16 Advanced Fighter Technology Integration (AFTI) F-16 triple redundant digital flight control system (DFCS) analogue backup DFCS was an asynchronous design channels run fairly independent of each other each computer samples sensors independently evaluates control laws independently, and sends its actuator commands to an averaging component that drives the actuator concerned 52 Directorate General Technical Airworthiness (DGTA-)

108 A Serious Shortcoming What happens when sensor noise and sampling skew cause independent channels to take different execution paths at decision points resulting in the production of widely divergent outputs? Occurred on Flight 44 of the AFTI F-16 Each channel declared the others failed Analogue backup was not selected because the simultaneous failure of two channels had not been anticipated Aircraft was flown home on a single digital channel Note now that all protective redundancy had been lost yet no hardware failure had occurred Numerous other mishaps subsequently traced to the same issue 53 Directorate General Technical Airworthiness (DGTA-) The Assessment The initial fix introduce voting (as opposed to averaging), extensive simulation and testing performed. On the next flight the problem was still there The final fix: The asynchronous design, while introducing some independence between channels, exposed the software to common sensor failure modes. Introduction of a requirement to synchronise the channels at decision points in the control laws. Required a complete system redesign. First discovered on Flight 44 a long way into the design process. Would have been cheaper to fix if detected earlier. 54 Directorate General Technical Airworthiness (DGTA-)

109 The Problem (Exaggerated) Sensor Output Channel 1 Sample Command Pitch Down Channel 2 Sample Command No Change Time Sensor Noise (e.g. air pocket, bird strike, etc) Channel 3 Sample Command Pitch Up 55 Directorate General Technical Airworthiness (DGTA-) Conclusion and Summary System and Software Safety provide assurance that the following error and accident causes are eliminated: Discrepancies between documented requirements specifications and the requirements needed for the correct functioning of the system. Misunderstandings about the software s interface with the rest of the system. Requirements specify behaviour that is not safe from the system perspective. Requirements do not specify some particular safety behaviour. The software has unintended (and unsafe) behaviour beyond what is specified in the requirements. Remember: The identification and allocation of software safety requirements are key to the realisation of acceptably safe software intensive systems. Verification activities cannot contribute to safety if the requirements being verified do not specify safe behaviours. 56 Directorate General Technical Airworthiness (DGTA-)

110 Questions 57 Directorate General Technical Airworthiness (DGTA-)

111 Software Reviews Definition A review of the work product, by one or more colleagues of the author, to evaluate the technical content and/or quality of the work 1

112 Review Spectrum Inspection Walkthrough Desk Check Ad Hoc Ad Hoc Informal reviews may often be unnecessarily expensive (because of time-wasting through lack of focus), and frequently provide a sense of security which is quite unjustified by the relatively small number of real defects found and repaired. 2

113 Desk Check Only one peer besides the author examines the work product. This is the cheapest review approach, but its usefulness is reliant upon the skills, knowledge and self discipline of the reviewers. Walkthrough The author leads members of the development team through a software product and the participants ask questions and make comments about defects. 3

114 Inspection Very formal type of peer review where the reviewers are following a well-defined process to find defects. 4

115 What to look at? Test Evidence Source Code Software Requirements Software Design Software Architecture Object Code What to look for Problems!! Traceability. Does each higher level element trace to at least one lower level element and does each lower level element trace to at least one higher level element? Compliance. Do the requirements/design/code satisfy the required system functional, performance and safety requirements? Accuracy/Consistency. Are the requirements/design elements accurate or is there ambiguity? Is there any conflict between requirements or design elements? Conformance to Standards. Has the defined requirements or design language been used? Have dangerous design and coding practices been avoided? Does the code comply with the approved language subset? Compatible. Are there any conflicts between the software requirements or design and the target hardware (e.g. 16-bit vs. 32-bit). Verifiable. Is it possible to prove through test that a requirement has been satisfied or design element correctly implemented? 5

116 Evidence of Reviews The following is a pretty common review sheet, what does it tell the certifying authority? Review Sheet Item Under Review: Module X Version: Reviewers: Findings: A. Smith M. Jones J. Smith Date: 01 Feb 10 No issues found. Evidence of Reviews A procedure that outlines the criteria that must be reviewed for. Positive evidence from the review that the reviewed item satisfies each criteria. Review Sheet Item Under Review: Module X Version: Reviewers: A. Smith M. Jones Date: 01 Feb 10 Checklist: The reviewed source code correctly implements the relevant design element The reviewed source code conforms to the coding standard. Findings: No issues found. 6

117 Summary Software Reviews are an effective tool for identifying errors as early in the life cycle as possible are required in order to comply with software assurance standards must produce positive evidence of item compliance Peer Reviews Course (1 day course) Would you like to know more? Peer Reviews Course (1 day course) 7

118 8

119 Software Testing SCI1 DGTA- Definition Software Testing: The evaluation of software by observing its execution Testing is about choosing elements from the input space of the software being tested Testing can be implemented at any time in the development process. SCI1 DGTA- 1

120 Software Verification Correctness through construction To control the design and construction of a product to assure that requirements are satisfied. Correctness through test To prove that a developed product satisfies requirements by dynamically exercising functionality. SCI1 DGTA- Software Testing Attitudes 1 2 Testing shows that the software works 3 Testing shows that the software doesn t work 4 Testing reduces the risk of using the software 5 Testing is a mental discipline that helps everyone develop higher quality software No difference between testing and debugging SCI1 DGTA- 2

121 Best Practices Define a test strategy Carefully design tests for your system Build a test environment Track defects to ensure their resolution Adopt tools to automate testing Manage Testing Resources SCI1 DGTA- Fundamental Testing Problem Begin A B C D E F Loop 20x End SCI1 DGTA- 3

122 Universal Truth Testing is used to show a system has zero defects Testing shows the presence, not the absence of bugs SCI1 DGTA- Testing Completeness Measures 1 Requirements Coverage 2 Code (structural) Coverage 3 Architectural Coverage SCI1 DGTA- 4

123 Requirements Coverage Normal Range Conditions coverage of equivalence classes and boundary values for variables (real, integer, boolean, etc) multiple iterations of time-related functions (e.g. filters, integrators, delays, etc) coverage of valid state transitions Robustness Conditions coverage of equivalence classes and boundary values for invalid conditions system initialisation during abnormal conditions failure modes of incoming data out of range loop count values protection mechanisms for exceeded timeframes arithmetic overflow coverage of invalid state transitions SCI1 DGTA- Eqivilance Classes Consider a FADEC for a jet engine with afterburner. Expected throttle setting is Above 75, afterburner is engaged. 8-Bit Two s Compliment Integer What tests are required to provide coverage of equivalence classes (normal and robustness) and boundary conditions? Equivalence Class Normal Boundary Normal Equivalence Class Robustness Boundary Robustness SCI1 DGTA- 5

124 Testing Completeness Measures 1 Requirements Coverage 2 Code (structural) Coverage 3 Architectural Coverage SCI1 DGTA- Structural (Code) Coverage Statement coverage Decision (branch) coverage Condition coverage Multiple Condition coverage Modified Condition/Decision Coverage Path Coverage SCI1 DGTA- 6

125 Definitions Conditions Decision A if (weight_on_wheels) & (air_speed < limit) B rev_thrust = REVERSE_THRUST_ENABLED; else C rev_thrust = REVERSE_THRUST_DISABLED; Statement A B C SCI1 DGTA- Statement Coverage Every line of code exercised at least once during requirements based testing. if (C OR D) AND (F OR G) Proc1(); else Proc2(); SCI1 DGTA- 7

126 Decision Coverage Every decision point has taken all possible outcomes if (C OR D) AND (F OR G) Proc1(); else Proc2(); SCI1 DGTA- Example What is the minimum number of tests to meet the structural coverage criteria for exhaustive coverage? if (C OR D) AND (F OR G) Proc1(); else Proc2(); SCI1 DGTA- 8

127 Modified Condition/Decision Coverage Every decision point has taken all possible outcomes during requirements based testing. Every condition has taken all possible values during requirements based testing. To show that the response to all possible conditions is appropriate. Every condition has been shown to be able to independently affect the outcome of the decision during requirements based testing. To show that the software only makes decisions based on relevant SCI1 DGTA- MC/DC Test Cases if (C OR D) AND (F OR G) Choosing where to start is the hardest part. Test C D F G Result 1 F T F T T 2 F T F F F 3 F T T F T 4 F F T F F 5 T F T F T Start Change G Change F Change D Change C Result has been both T and F (Decision Coverage) Each condition has been both T and F (Condition Coverage) Each test has changed one variable and the result has changed. Some effort is required to determine the above tests. Would it be easier to just do exhaustive testing? SCI1 DGTA- 9

128 Measuring Structural Coverage Measuring structural coverage can be a time consuming process Can be done by hand (e.g. Classic Hornet), but not recommended Highly recommended to use a software tool SCI1 DGTA- Testing Completeness Metrics 1 Requirements Coverage 2 Code (structural) Coverage 3 Architectural Coverage SCI1 DGTA- 10

129 Architectural Coverage Assure that the real interfaces are sufficiently exercised during testing. To do this, need to exercise each of the following at least once using actual software modules: Data Paths (passing of data from one software module to another). Control Links (i.e. one software modules control over another). Control Paths (passing of control from one to module to another). SCI1 DGTA- Testing Paradigms Requirements Driven Testing Risk Driven Testing Unit testing and Integration testing Regression Testing Security Testing Scalability Testing SCI1 DGTA- 11

130 DO-178B Testing SCI1 DGTA- A Check List for DO-178B Test Completion Test Procedures (including expected outcomes) Defined? Compatibility with Target Hardware confirmed? Test Environments Qualified? Requirements Coverage Achieved? Normal and Robustness Testing of Software (high level) Requirements (Level D and above) Normal and Robustness Testing of Software Design (low level) (Level C and above) Code (Structural) Coverage Achieved? Statement Coverage (Level C) Decision Coverage (Level B) Modified Condition/Decision Coverage (Level A) Architectural Coverage Achieved? Each actual control and data link exercised (Level C and above) Testing sufficient to validate hardware assumptions (Level D and above) SCI1 DGTA- 12

131 Test Procedures Test Procedures should be defined and approved prior to the commencement of testing. Test Procedures should identify the expected outcome of the test. It is possible that there are errors in the test procedures: Can be redlined during testing if a simple process error. Otherwise require raising of problem report. SCI1 DGTA- A Check List for DO-178B Test Completion Test Procedures (including expected outcomes) Defined? Compatibility with Target Hardware confirmed? Test Environments Qualified? Requirements Coverage Achieved? Normal and Robustness Testing of Software (high level) Requirements (Level D and above) Normal and Robustness Testing of Software Design (low level) (Level C and above) Code (Structural) Coverage Achieved? Statement Coverage (Level C) Decision Coverage (Level B) Modified Condition/Decision Coverage (Level A) Architectural Coverage Achieved? Each actual control and data link exercised (Level C and above) Testing sufficient to validate hardware assumptions (Level D and above) SCI1 DGTA- 13

132 Hardware Compatibility Also need to test software for compatibility with target hardware. Most testing will be conducted in a simulated environment. Need to check that the actual target hardware satisfies any assumptions inherent in the design process. Particular concerns: Hardware Transients and Failures Built In Test Validation Interface Errors Control Loops Resource Management (time, interrupts, memory) Field Loading Operations Protection Boundaries SCI1 DGTA- A Check List for DO-178B Test Completion Test Procedures (including expected outcomes) Defined? Compatibility with Target Hardware confirmed? Test Environments Qualified? Requirements Coverage Achieved? Normal and Robustness Testing of Software (high level) Requirements (Level D and above) Normal and Robustness Testing of Software Design (low level) (Level C and above) Code (Structural) Coverage Achieved? Statement Coverage (Level C) Decision Coverage (Level B) Modified Condition/Decision Coverage (Level A) Architectural Coverage Achieved? Each actual control and data link exercised (Level C and above) Testing sufficient to validate hardware assumptions (Level D and above) SCI1 DGTA- 14

133 Test environment Credit can only be taken for tests conducted on the target hardware or in a qualified test environment. It must be shown that the test environment is sufficiently representative of the target environment for the test case to be valid. Requires test environment qualification procedures (i.e. tests to verify correct operation) and configuration management of the test environment. SCI1 DGTA- A Check List for DO-178B Test Completion Test Procedures (including expected outcomes) Defined? Compatibility with Target Hardware confirmed? Test Environments Qualified? Requirements Coverage Achieved? Normal and Robustness Testing of Software (high level) Requirements (Level D and above) Normal and Robustness Testing of Software Design (low level) (Level C and above) Code (Structural) Coverage Achieved? Statement Coverage (Level C) Decision Coverage (Level B) Modified Condition/Decision Coverage (Level A) Architectural Coverage Achieved? Each actual control and data link exercised (Level C and above) Testing sufficient to validate hardware assumptions (Level D and above) SCI1 DGTA- 15

134 Structural Coverage What happens if the requirements based tests do not satisfy structural coverage objectives? All tests must be requirements based. i.e. can t just write tests to cover gaps in structural coverage Options: Write new requirements based test cases. Write new requirements. The code is dead (i.e. not used, not needed), remove it. Deactivate the code: take measures to prevent execution and verify those measures. Used when one software item can be used on multiple installations. SCI1 DGTA- A Check List for DO-178B Test Completion Test Procedures (including expected outcomes) Defined? Compatibility with Target Hardware confirmed? Test Environments Qualified? Requirements Coverage Achieved? Normal and Robustness Testing of Software (high level) Requirements (Level D and above) Normal and Robustness Testing of Software Design (low level) (Level C and above) Code (Structural) Coverage Achieved? Statement Coverage (Level C) Decision Coverage (Level B) Modified Condition/Decision Coverage (Level A) Architectural Coverage Achieved? Each actual control and data link exercised (Level C and above) Testing sufficient to validate hardware assumptions (Level D and above) SCI1 DGTA- 16

135 Development Configuration Management, Quality Assurance and Software Tools SCI1 DGTA- Development Software Configuration Management SCI1 DGTA-

136 CM - Purpose Establish and maintain the integrity of software development life cycle data through the software life cycle. SCI1 DGTA- CM Goals Planned and repeatable process Software work products are identified and controlled Changes to identified software work products are controlled Development and verification is only performed using relevant software baselines SCI1 DGTA-

137 CM Benefits Links between object code and the development and verification evidence that proves its acceptability. Assures that the software that is loaded to the target environment is the same as the software that was verified. Assures that developers will work from the correct version of data. Prevents unintended changes to software or data. Tracks problems through to resolution. SCI1 DGTA- CM Key Tasks Establish a process (software configuration management plan) Establish a library for repository for software baselines. Work products to be placed under CM should be identified. Change requests and problem reports are initiated, recorded, reviewed, approved and tracked in accordance with a documented and repeatable procedure. Changes to baselines are controlled in accordance with a documented and repeatable procedure. SCI1 DGTA-

138 CM Key Tasks (cont d) Release of software products is controlled in accordance with a documented procedure. Status of CIs is recorded IAW a documented procedure. Keep records/evidence documenting SCM activities and baselines. Software baseline audits are conducted according to a documented procedure. Software life cycle environment configuration is controlled. SCI1 DGTA- Challenges for CM Large development teams Parallel or concurrent development Geographically or time dispersed teams Multiple versions for different targets/platforms Level of integration between software components Level of control Level of automation When should SCM begin? SCI1 DGTA-

139 IBM Clear Case (example) SCI1 DGTA- Problem Reporting Needed to identify, analyse, and resolve anomalous behaviour identified at any point in the development life-cycle Often combined with Change Management / Change Requests SCI1 DGTA-

140 Development Software Quality Assurance SCI1 DGTA- Software Quality Assurance SCI1 DGTA- An independent check that the developer has done what they said they would do Types of Involvement Check that particular product complies with requirements. Check that transition criteria have been satisfied. Software Conformity Review. Periodic Audits. QA Personnel must be independent enough to be effective (e.g. answer to t)

141 Final Thoughts on QA SCI1 DGTA- QA rarely staffed with experience software developers/coders, but are generally excellent at record keeping QA often not capable of negotiating with developers QA and development need to be in a partnership Project and senior management often back the developers and not the QA team QA people must be proactive, independent and drivers Address QA at the project reviews Development Software Tool Qualification SCI1 DGTA-

142 Software Tools Use to Develop, Test, Analyse, Produce and Modify Software Reduce workload, but are a potential source of error Two Types of Tools Development Tools Verification Tools Why qualify tools? SCI1 DGTA- Development Tools Tools whose output is part of the airborne software Modifies or creates requirements, design or code Can inject errors into development products Requires software assurance Software Development Tool Baseline Product + Errors New Product + Errors SCI1 DGTA-

143 Verification Tools Unable to create or modify development products Requirements, Design or Code May be able to create or modify verification products Test Cases, Test Results, Code Inspections, etc Cannot introduce errors into the object code, but can fail to detect them Verify that software tool satisfies requirements Software Verification Tool Unknown Errors Unknown Errors Errors SCI1 DGTA- Examples of Tool Classifications Development Autocode Generators Compilers Software Libraries Verification Simulators Emulators Test Case Generators Test Tools Coverage Analysers Static Analysers Proof Checkers Model Checkers SCI1 DGTA-

144 Auto-coding example SCI1 DGTA- AMC/Metrowerks CodeTEST SCI1 DGTA-

145 What is tool qualification? Process to ensure that a tool provides confidence at least equivalent to the processes that are eliminated, reduced or automated. See DO-178B Section SCI1 DGTA- Tool Qualification Criteria Not all tools require qualification Only need to qualify a software tool if the answers to the following three questions are all yes: Can the tool insert an error or allow an existing error to remain undetected? Are software assurance processes eliminated, reduced or automated? Will the tool s output not be verified per section 6 of DO-178B? SCI1 DGTA-

146 Special Tool Topics Compilers Development Tool? Yes Qualification Required? No, as other software assurance objectives assure verification of processes handled by compilers. Configuration Management Tools Development Tool? Yes (but ) Most certification authorities treat CM tools as Verification Tools for the purposes of qualification Pragmatic Decision based on usage SCI1 DGTA- Summary Software Tool Qualification SCI1 DGTA- If software tools are to be relied upon, they should be qualified. Should have the same level of confidence in the tool as provided by the assurance processes reduced, eliminated or automated. Three questions to determine qualification: Software assurance process reduced, eliminated or automated? Can introduce errors into software or fail to detect and error? Output not independently verified by other software assurance processes? Verification Tools Can t introduce an error, but can fail to detect Relatively easy to qualify, in common use Development Tools C i t d

147 Questions SCI1 DGTA- Tool Qualification For all tools Tool operational requirements should be defined (as per DO-178B section ) SCI1 DGTA- Verification Tools Verification that tool meets operational requirements PSAC, SAS Development Tools Software level assigned to tool commensurate with software level assigned to software being developed. Complete DO-178B data package for assigned software level Tool Qualification Plan, Tool Accomplishment Summary

148 Tool Operational Requirements Development Tool Functionality Operational Environment Installation or Operational Information Development Process Performed Expected Response under Normal and Abnormal Conditions Verification Tool Functionality Operational Environment Installation or Operational Information SCI1 DGTA-

149 Software Compliance Finding SCI1 DGTA- Compliance Findings A compliance finding is: an engineering decision based on relevant evidence that an aircraft design satisfies one or more airworthiness requirements. Compliance findings can only be conducted by Commonwealth personnel, or someone acting directly on behalf of the Commonwealth. Can t contract out compliance findings. But can rely on external expertise. SCI1 DGTA-

150 Example Software Benchmarks Civil Aircraft System Safety and Software Safety: FAR 2x.1309 and SAE ARP 4754/4761 Software Assurance: DO-178B Software Development: IEEE/EIA Military Aircraft System Safety: MIL-STD-882C Software Safety: MIL-STD-882C and IEEE STD 1228 Software Assurance: Software Assurance Matrix Software Development: MIL-STD-498 SCI1 DGTA- Software and Verification The verification pillar requires the design agency to produce evidence of requirements satisfaction. The Design Acceptance process requires that the Commonwealth review evidence of requirements satisfaction. Not a design review A check to determine whether the evidence produced supports the claims of requirements satisfaction. To do this, the Commonwealth conducts compliance findings. SCI1 DGTA-

151 Compliance Finding Plans Software assurance benchmark (from System Safety Activities) Activities the project office intends to do to demonstrate that the benchmark has been achieved Evidence the project office intends to use during those activities Who will be undertaking compliance finding and why they are competent The combination of the above should logically lead to the capability for the compliance finding agency to establish of the level of compliance against the identified benchmark More Info: AAP S2C7 SCI1 DGTA- The Software Compliance Finding The software compliance finding is similar to other compliance findings. It is an engineering decision based on relevant evidence that an aviation software item satisfies an airworthiness requirement. How many software compliance findings are required? Usually just one: against software assurance. Software in System Safety is covered by the System Safety Compliance finding. Software Safety is usually (but not always) embedded within System Safety. Consideration of software assurance objectives is generally sufficient consideration of compliance with a software development standard. SCI1 DGTA-

152 Software Compliance Finding Challenges The software compliance finding is heavily dependent on access to data and personnel. Need to have a good plan prior to contract signature to ensure the contract supports the plan. It can be an onerous activity. A lot of compliance findings are done at the end of a program. The software compliance finding is a continuous process that should start the day after contract signature. Trying to catch up at the end is a dangerous approach: Might not get it all done. Shortfalls discovered late in the process can be very expensive to fix. SCI1 DGTA- Software Compliance Finding - Cover: Satisfaction of software assurance objectives Non-Interference Retained Risk The mechanics: Review sample design and verification evidence against assurance objectives for unique software Review sample evidence of regression analysis and re-verification activities to confirm noninterference SCI1 DGTA-

153 Reviewing Evidence Against Assurance Objectives SCI1 DGTA- Goal: verify complete and correct satisfaction of each objective. Four Step Process: Verify that the software development organisation has claimed complete and correct satisfaction of the objective. Verify that the analysis method used by the software development organisation to determine that the objective has been completely and correctly satisfied is appropriate. Verify that the objective has been correctly satisfied for a sample set of evidence. Verify that the process employed to satisfy the objective is sufficiently robust to assure that similar findings could be made across the entire set of evidence. Example Reviewing Traceability Objective: Source code is traceable to low-level requirements. Four Steps: Verify that the software development agency has determined that all low-level requirements trace to one or more source code elements and that all source code elements trace to one or more low-level requirements. Verify that the analysis method was appropriate and that relevant data was relied upon. Select a number of low-level requirements and attempt to trace to the source code that implements them using only documented data. Attempt to do the same from source code to low-level requirements. Review relevant processes to ensure that traceability data would be consistently established, recorded and reviewed. SCI1 DGTA-

154 Software Compliance Finding How much is enough? SCI1 DGTA- A compliance finding is not a design review. You should not review every piece of evidence. But you do have to review evidence against every objective. You should review enough evidence to obtain confidence that your finding (e.g. compliant or noncompliant) is correct. But remember, you may have to convince somebody else (the DAR, DGTA) that your findings are correct. How much is enough? depends on a number of factors: the criticality of the software the developmental history of the software the competence of the software development organisation Software Compliance Finding Competencies Software compliance findings can be difficult tasks. Particularly at higher assurance levels. The core training and experience profile for FLTLT ELECTRs (or equivalent) generally does not provide the competencies required to complete the more challenging software compliance findings. That is, not all FLTLT ELECTRs (or equivalent) have the required competencies, but some do. The DAR or PEM should be aware of this and, where possible, seek to have the competencies obtained. SCI1 maintains expertise in software and can assist or conduct the more complex compliance findings. Specific guidance on required competencies is available in the paper Compliance Finding Agency Competencies available through the DGTA website. SCI1 DGTA-

155 Software Compliance Finding Competencies In addition to general competencies. e.g. ability to determine situations beyond their competence Low Level of Competence Demonstrated understanding of.054 S2 C7 and C17. Completion of this course. Completion of the software testing course. Medium Level of Competence Completion of the DO-178B, MIL-STD-498 and IEEE/EIA courses as appropriate. Previous experience in the development or acquisition of airborne software systems. High Level of Competence Masters of Software Engineering (or equivalent) Previous experience conducting compliance findings against the relevant standard. SCI1 DGTA- Key Issue Contractual Support The contract or LOA must support the Compliance Finding strategy. Therefore, the Design Acceptance strategy must be defined prior to contract or LOA signature. For software, the contract must: impose the four airworthiness requirements, impose competence requirements (AEO plus delivery of software specific plans), require delivery of or access to sufficient data to support the compliance finding process, and provide access to software personnel when required. For software, the LOA should: require delivery or disclosure of airworthiness design and verification artefacts, provide for support from airworthiness personnel, and provide access to or delivery of documents required to support the airworthiness process. SCI1 DGTA-

156 Aligning irpa and Oversight SCI1 DGTA- There are differences between the software assurance approaches of the three major military airworthiness authorities. Consistent with FAA/EASA: DO-178B or equivalent. USAF/USN (some variations within each service) professional judgement used to determine whether software development and verification activities are sufficient may or may not align with DO-178B heavy reliance on structured flight test program (dedicated squadrons conducting structured flight test programs for 3, 6 or 12 months) UK MoD Similar to /FAA/EASA: DEF-STAN requires developer to propose suitable processes, DEF-STAN (now cancelled) used as a benchmark (requirement for formal methods at highest levels). No RPA. Aligning irpa and Oversight Difficult to resolve the above differences: If the focus is restricted to development and lab verification, there are software items that could be accepted under RPA but would be rejected if oversight was applied. Need to look at the bigger picture: USAF/USN do not rely as heavily on software assurance as the does. The approaches are different, but not incorrect or incompatible. The approach is reflective of our relative size: We generally don t design our own aircraft, so we need to rely on aircraft designs that are acceptable to other forces. We need to establish that the relative level of risk retained by an authorised National Airworthiness Authorities could be sensibly retained by the SCI1 DGTA-

157 irpa Compliance Finding Establish the impact of difference in Configuration (substantially similar is not the same as identical) Establish the impact of differences for the s intended use (roles) and operating environment Establish risk that has been retained by the National Airworthiness Authority and assess whether those risks can be sensibly retained by the SCI1 DGTA- NAA Oversight of Unique Equivalent Rigour Design Equivalent Rigour Safety NAA Oversight Requirements Validity Unique Requirements Derived Requirements Satisfaction SCI1 DGTA-

158 Summary Plan for Software Compliance Finding Benchmark, Activities, Evidence, Agencies Provide for Compliance Finding within contracts Compliance Finding Execute Plan Examine Objective Evidence SCI1 DGTA- Questions SCI1 DGTA-

159 In-Service Configuration Management SCI1 DGTA- Key Goals Software Configuration Index / Software List Problem Reporting/Change Requests Software Load Control Compatible with aircraft Change Management System SCI1 DGTA-

160 Key Goals - Software List Identify the authorised software configuration for your aircraft (source for continuing airworthiness products) Identify software components on your aircraft (regardless of whether they are field loadable) SCI1 DGTA- Software List (Basic Elements) CSCI Name CSCI Identifier / Part Number CSCI Version Parent CSCI Worst case failure mode Software assurance level Brief Description SCI1 DGTA-

161 Problem Reporting TAREG e. (1) Software problem reporting Problem assessment Tracking of problem reports and corrective actions Ensure safety related errors, faults and failures are identified and resolved in a timely manner SCI1 DGTA- Will aircrew fill this out? SCI1 DGTA- System Change Request/Problem Anomaly Report and Resolution Record Project Name: Originator: Problem Number: Problem Name: Software element: Origination Date: Category: Priority: Tail No: Date of occurrence: Time of occurrence: Related ASOR: Related Defect Report: Problem Description: CSCIs affected: Version of CSCIs affected: Procedures to produce anomaly: Proposed solution: Required implementation time: Funding source for analysis / resolution: Section Commander Approval: SO Capability Development Approval: SPO Approval: Recommended resolution action:

162 What about this? What happened? When did it happen? What were you doing? Has this happened before? SCI1 DGTA- What about this? What needs to happen? When should it happen? SCI1 DGTA-

163 DAR and TAR notification TAREG e. (2) Notify the DAR and TAR of errors, faults or failures of aviation software which potentially result in a large reduction in safety margins SCI1 DGTA- Software Load Control SCI1 DGTA- TAREG d. (5) Procedures that assure that the software that is loaded to the target environment is the same as the software that was verified (and therefore form part of the authorised configuration) Smart aircraft may have on-board interrogation capability allowing maintainers to establish current aircraft software configuration Steam-driven aircraft may need to be verified via other means such as an OEM report/attestation

164 Questions SCI1 DGTA-

165 Mission Planning Systems SCI1 DGTA- Overview What is a mission planning system? Technical Airworthiness Aspects of Mission Planning Systems Integrity Requirements for Mission Planning Systems SCI1 DGTA-

166 What is a mission planning system? SCI1 DGTA- A suite of software applications and associated hardware that allow maps, charts, weather, intelligence and aircraft performance data to be used in developing navigation solutions, communication settings, flight/mission calculations, etc. Examples: Portable Flight Planning System Joint Mission Planning System Ground Mission Management System Functions performed (generally): Aircrew enter flight details (waypoints, aircraft loading, etc). MPS performs calculations to validate mission plan. e.g. Is weight and balance within limits? MPS produces data to be loaded to the aircraft. e.g. sequence of waypoints, TOLD, digital maps PFPS/JMPS SCI1 DGTA-

167 Is this a good idea? Mission Planning and Google Earth? SCI1 DGTA- EK407 departing MEL 20 Mar 09 SCI1 DGTA-

168 Technical Airworthiness Aspects of MPS There are hazards associated with MPS that could propogate to the aircraft: Databases may have errors. Calculations can be incorrect. Data could be corrupted during transfer. MPS must be assured commensurate with the severity of associated hazards. No different to any other system with safety implications. Knowledge of operational processes is required to determine how severe the hazards are. policy is based on FAA policy, with some changes to account for use of MPS. SCI1 DGTA- Principles for Design Acceptance Establish the context and intended use of data and products that are output by the Aviation Support System at an aircraft level. Establish your customer's safety requirements. Conduct an aircraft-level hazard assessment based on the context and intended customer use of output data / products. Identify appropriate treatments for identified hazards. Consult with SCI for assistance with establishing appropriate safety benchmarks and treatment options for identified hazards. Implement treatments that are within boundary of the design of the Aviation Support System. Communicate the level of residual risk inherent in data and products product by the Aviation Support System to users in conjunction with product delivery. Communicate recommended treatments that are beyond the boundary of the Aviation Support System to users in conjunction with product delivery. SCI1 DGTA-

169 Common MPS Hazards SCI1 DGTA- Erroneous Weight and Balance Calculations Erroneous Take Off and Landing Data Calculations Erroneous Fuel Usage Calculations Incorrect Translation of Waypoint Map Coordinates Erroneous Data Packages for Guided Weapons Introduction of Errors into Aeronautical Data Transfer, Formatting or Translation Digital Maps Navigation Databases Airfield Databases Digital Terrain Elevation Data Aeronautical Data Processing Database Reformat e.g. DAFIF, DTED, Jeppesen Real World Data Reversal Check Checksum Assurance W&B Calculations (Generate) Assurance Takeoff Data (Generate) Checksum Airfield Database (Transfer) Mission Planning System For each link, assure that: errors are not introduced, or errors will be detected. Checksum SCI1 DGTA-

170 Aeronautical Data Processing For each step of the aeronautical data processing chain, must assure that either: errors are not introduced, or introduced errors are detected and handled. A Mission Planning System is a step in the aeronautical data processing chain. To achieve the above, must either: assure the correct function of the MPS, or implement (and assure) detection and handling mechanisms. SCI1 DGTA- The Process Identify the data processed by the MPS. Identify the data criticality and data assurance level. Identify each MPS component that processes each data element. Identify what function the each MPS component performs for each data element. Transfer, Format, Manipulation, Generation Define and Implement Treatments to assure the detection and handling or absence of errors. Reference: RTCA/DO-200A, Section 2 Chap 24 SCI1 DGTA-

171 Criticality Severity DAL Data Criticality Definition Data Assurance Level Catastrophic Hazardous Major Minor No Safety Effect A B C D E Critical Critical Essential Essential Routine The data, if erroneous, would prevent continued safe flight and landing or would reduce the ability to cope with adverse operating conditions to the extent that there is a large reduction in safety margins or functional capabilities. There is a high probability when trying to use corrupted critical data that an aircraft would be placed in a life threatening situation. The data, if erroneous, would reduce the ability to cope with adverse operating conditions to the extent that there is a significant reduction in safety margins. There is a low probability when trying to use corrupted essential data that an aircraft would be placed in a life threatening situation. The data, if erroneous, would not significantly reduce aircraft safety. There is a very low probability when trying to use corrupted routine data that an aircraft would be placed in a life threatening situation SCI1 DGTA- Criticality Considerations Care should be taken to clearly identify the hazard associated with MPS use. This may be different to the aircraft level hazard. Example: Weight and Balance Attempting to fly an overweight aircraft, or with an out of range CoG, could be Catastrophic. Does this mean that the severity of weight and balance calculation errors is Catastrophic? Possibly, but not necessarily. What if the variables to the calculation can only have a limited effect on W&B? Is an obscure load required to produce the worst effects? How is the data used? To validate the flight plan? Or as in input to TOLD calculations? SCI1 DGTA-

172 Treatment Options Function Transfer Format Manipulat e/ Generate Data Detection and Handling Assurance Level Critical Digital Error Detection OR Essential Feedback / Read Back Verify Routine No requirements. Critical Feedback / Reversibility Check OR Essential Independent Redundancy Routine No requirements. Critical Feedback / Reversibility Check OR Essential Independent Redundancy Level C or D AND Logical Consistency Checks OR Semantic Consistency Checks Routine No requirements. See notes in Table 3 AAP S2 C24 (DGTA NPRM 03-09) for additional information on assurance levels. OR OR OR OR OR OR OR Absence Level A or B Level C or D Level A or B Level C or D Level A or B SCI1 DGTA- Definitions SCI1 DGTA- Logical Consistency Logical consistency validates by comparing the relationship between two different data sets. Example: Translate two maps: same location, different scale. Landmark appears in the same place on both maps. Semantic Consistency Semantic consistency validates by comparing data to an expected value or range of values. Example: Calculated aircraft weight should be less than max weight. Neither technique is perfect.

173 Example Weight and Balance Calculations Data: aircraft weight, centre of gravity. Criticality: depends on aircraft type, but usually Essential. MPS Components: W&B Calculation Component: Generation Data Transfer Module: Transfer Treatments: W&B Calculation: Function assured to Level C or D and semantic consistency check (within expected range) Data Transfer Device: Digital Error Detection assured to Level C or D SCI1 DGTA- Example DTED Transfer Data: Digital Terrain Elevation Data Criticality: Critical (used in TAWS) MPS Components: Import DTED: Transfer Export DTED to Data Transfer Device: Transfer Treatment Options Digital Error Detection assured to Level A or B for each MPS component, or calculated at source, checked on aircraft. Note: other treatments would be required if the MPS manipulates the database (i.e. where digital error detection could not be applied). SCI1 DGTA-

174 Example Digital Map Formatting Data: digital maps Criticality: Essential or Critical if used in a moving map display Essential if static display MPS Components Import Function: Transfer Formatting Function: Format Data Transfer Device: Transfer Treatments Input Device: Digital Error Detection assured to appropriate level Formatting Function: Reversibility check assured to appropriate level Data Transfer Device: Digital Error Detection assured to appropriate level SCI1 DGTA- Example - Waypoints Data: Flight Path Waypoints, may include: Mission Flight Path, Weapon Release Points, Terminal Approach Data, or Flight Path through Controlled Air Space Criticality: depends on type of waypoint Mission Flight Path: Routine Others: Essential or Critical MPS Components User Interface: Generation Data Transfer Device: Transfer Treatments: User Interface: assured to appropriate level (depends on type) and semantic consistency checks (in range, distance from departure, etc) Data Transfer Device: Digital Error Detection assured to appropriate level. SCI1 DGTA-

175 Example - Summary Level of Assurance depends on criticality of the data. Scope of assurance depends on function performed: Generation: assure entire function and some fault tolerance Transfer: generally only need to assure digital error detection Formatting: generally only need to assure readback/verify Many common requirements will emerge: e.g. data transfer device almost always needs digital error detection SCI1 DGTA- Other Factors SCI1 DGTA- Above analysis only considers the introduction of errors or erroneous calculations. There are other, important characteristics of aeronautical data that the MPS may negatively affect: accuracy resolution timeliness completeness format These considerations may result in additional requirements being placed on the MPS. e.g. When translating the navigation database, the input resolution shall be i t i d

176 Technical Airworthiness Policy for MPS Technical Airworthiness Design Requirements for MPS incorporates two considerations: Data produced or calculations made by an MPS are correct. Aeronautical data processing does not introduce errors. ADRM S2C24 provides guidance for certification of an MPS Assuring the correctness of data produced or calculations made by an MPS is a readily apparent consideration. No difference to the assurance of other safety related systems. If a system is a part of the Aeronautical data chain it can impact airworthiness and therefore hazards must be identified and subjected to treatment. SCI1 DGTA- What does DGTA expect to see? A List of Functions What is the MPS used for? Assignment of Severity How critical is the data? Allocation of Treatment Assurance of absence Detection and Handling Evidence of Implementation Standard software assurance artefacts. SCI1 DGTA-

177 A Good Format Data Element A B etc... Criticality Critical Essential MPS Components and Functions Component Function X Transfer Y Z P Q Format Manipulation Transfer Manipulation Treatment CRC Check assured to Level A Reversibility Assured Check to Level B and Semantic Consistency None = Risk Independent Redundancy with Component R and Logical Consistency See Table 4 of AAP S2 C24 for more details. SCI1 DGTA- Summary SCI1 DGTA- MPS may contain safety-related software. Erroneous calculation of safety-related data. Introducing errors into safety-related databases. Like any safety-related item, an MPS needs to be appropriately assured. Errors are not introduced, or Errors are detected and handled. An assessment should be conducted to determine: which components of the MPS perform safetyrelated functions, and how critical each function is.

178 Further Reading AAP Section 2 Chapter 24 Mission Planning Systems AAP Section 2 Chapter 22 Electronic Flight Bags Useful for determining data criticality RTCA DO-200A Standards for Processing Aeronautical Data RTCA DO-201A Standards for Aeronautical Information SCI1 DGTA- Questions SCI1 DGTA-

179 Background FAA Policy MPS are not directly considered by FAA policy. FAA look at: Electronic Flight Bags (EFBs) Aeronautical Data Processing use of MPS incorporates both EFB and aeronautical data considerations from the FAA system. has a system focus (i.e. we look at Design Acceptance of an MPS). FAA has a data focus, approve aeronautical data for use. This approval considers all systems that interact with the aeronautical data. SCI1 DGTA-

180 Electronic Flight Bags SCI1 DGTA- Introduction Software Type A Static (Publications, Manuals) Type B Interactive (Performance Calculations, precomposed or interactive maps) Type C Primary Flight Displays, Navigation Displays, Moving Maps Hardware Portable (FAA Class 1 and 2) Integrated (FAA Class 3) SCI1 DGTA- 1

181 Type A Software Applications SCI1 DGTA- Pre-composed manuals, procedures, publications, references, application software Anticipated to be COTS Software assurance S2C7 key issues don t specifically need to be addressed Focus on best practice: Reduce/eliminate misleading data Readable Robust Accurate, available and timely provision of data Compatible with OS Type B Software Applications Performance Calculations, Chart Viewers, Aeronautical chart viewers, Mission Planning, and Flight Planning Software assurance S2C7 key issues should be addressed Conduct Safety assessment required to assurance required SCI1 DGTA- 2

182 Type C Software Applications Primary Flight Displays, Navigation Displays, Moving Map Displays Software assurance S2C7 key issues should be addressed Conduct Safety assessment required to assurance required SCI1 DGTA- Design Considerations Use of aircraft power Batteries Environmental Considerations Mounting Devices Human Machine Interface COTS Operating System Aeronautical Information Databases Source Documents Security/Information Assurance Configuration Control Sustainment SCI1 DGTA- 3

183 Operational Considerations EFBs aren t just a technical airworthiness issue Procedures for all phases of flight Paper Backups during transition Software Load Control Operational Evaluation at least six months recommended by FAA SCI1 DGTA- Certification Principles Portable Type A, Type B SCI1 DGTA- Use supported by Safety Case Software assured to a level commensurate with the worst case failure-mode Software robust Robust backups or procedures in the event of failure Hardware compatible with the target environment Operational procedures are adequate Use supported by operational evaluation/workload assessment 4

184 Questions SCI1 DGTA- 5

185 Course Summary Aviation Software >>A challenging technology that requires careful management. >>Critical to aviation capability >>Critical to the safety of aircraft. 1

186 System Safety >>Software s contribution to aircraft level hazards should be identified and analysed. >>It is not enough for the software to be built well, it must also do the right things. >>Structured techniques are necessary to prove that safety analyses are sufficiently complete. Software Assurance The more critical the software, the more effort should be expended in proving its correctness. A clear standard of proof (e.g. software assurance standard) needs to be defined. Software assurance concepts can also be used to prove capability integrity. 2

187 Reviews and Inspections >>Reviews and Inspections can be the most effective and efficient means for detecting errors. >>Reviews and Inspections must produce evidence of positive as well as negative findings. Software Testing >>There are objective measures for measuring the completeness of software testing. >>Software testing needs to be relevant: always requirements based. 3

188 CM, QA and Tools >>Software Configuration Management provides the links between the object code and the proof that the object code is acceptable. >>Software Quality Assurance is an independent check that the developer did what they said they would do. >>If you are going to use software tools, you should have at least as much confidence in the tool as you do in the process being reduced, eliminated or automated. Mission Planning Systems >> Mission Planning Systems can handle safety related data and perform safety related calculations. >> It must be assured that the Mission Planning System does not introduce errors (absence) or that errors will be detected and handled. >> The level of development and verification rigour for a Mission Planning System varies with the criticality of the action being performed (just like on-aircraft systems). 4

189 Design Acceptance >>Software Design Acceptance is a determination of the technical acceptability of an aviation software item with respect to a set of functional and airworthiness requirements. >>A software compliance finding is an engineering decision based on relevant evidence that a software item satisfies the four software airworthiness requirements. >>Software Design Acceptance is a process that must be started prior to contract signature and continues throughout the project. Final Thoughts Most software projects that struggle do so because: They underestimate the effort or competence required to effectively manage aviation software. They look for a single solution (CMMI, RPA, COTS, etc). They leave it all to the last minute. 5