Anwendung von Polyspace im Software Entwicklungsprozess nach IEC 60880 München, 19.05.2011, Dr.-Ing. Jörg Barrho
Agenda 01 Tognum and MTU Friedrichshafen 02 Background and project 03 Overview IEC 60880 and application for tool validation 04 Validation of PolySpace 05 Result Page 2 Anwendung von Polyspace im Software Entwicklungsprozess nach IEC 60880 / TKE-R, Barrho 19.05.2011
01 MTU s Place in the Tognum Group Powerful Brands under One Umbrella Business Units Engines Onsite Energy Components Brands Products Complete Drive and Propulsion Systems Gas Engine Systems Diesel Engine Systems Injection Systems Page 3 Anwendung von Polyspace im Software Entwicklungsprozess nach IEC 60880 / TKE-R, Barrho 19.05.2011
01 The Tognum Group A Wide-Ranging Product Portfolio Business Unit Engines Business Unit Onsite Energy & Components Marine Industrial Defense Oil & Gas Diesel Systems Gas Systems Injection Systems Yachts Commercial Naval Rail C&I Agric. Mining Light and Medium Vehicles Heavy Vehicles Onshore Offshore Emergency Power Prime Power Continuous Power Continuous Power Highspeed Engines Medium Speed Engines Page 4 Anwendung von Polyspace im Software Entwicklungsprozess nach IEC 60880 / TKE-R, Barrho 19.05.2011
01 MTU Key Technologies for Diesel Engines Fuel injection Exhaust Turbocharging Electronics Exhaust Emission Minimizing Analytics Injection pressure Injection process Combustion process Efficiency Pressure ratio Turbocharger map Engine management Map control System Control Exhaust gas recirculation Miller Cycle Aftertreatment Strength Bearing Load LCF-behavior Fuel consumption Exhaust emission Noise Emission Fuel consumption Exhaust emission Power-to-weight-to size ratio Fuel Consumption Exhaust Emission Transient behavior Lowest Emission with optimized LCC Fuel consumption Durability Weight Acoustics Page 5 Anwendung von Polyspace im Software Entwicklungsprozess nach IEC 60880 / TKE-R, Barrho 19.05.2011
01 MTU Product Program LM 2500/ LM 6000 TF 40/50 Gas turbine Vericor Gas turbine GE 8000 956/1163 4000 396 2000 1600 PP1800 900/ 460/500 S60 PDU 900/460/500/S60 Power [kw] 106/199/837/ 870/880/890 0 1.000 2.000 3.000 4.000 5.000 6.000 7.000 8.000 9.000 25.000 45.000 Page 6 Anwendung von Polyspace im Software Entwicklungsprozess nach IEC 60880 / TKE-R, Barrho 19.05.2011
Application: Emergency Diesel Generators Application of MTU diesel engines in nuclear power plants as emergency power generators Required standards KTA 3702, IEC 60880 and specific national safety guidelines Qualification of software and existing hardware Page 7 Anwendung von Polyspace im Software Entwicklungsprozess nach IEC 60880 / TKE-R, Barrho 19.05.2011
History and background of project First approach Test of the existing software safety functions are not impacted by none safety functions, no run time errors, check of complexity etc. PolySpace Result: ca. 50 000 loc, 52 red/ 12279 orange/ 50143 green/ 1692 grey) Result: not sufficient for qualification Second approach Entire new development process according IEC 60880 New developed software according the standard Page 8 Anwendung von Polyspace im Software Entwicklungsprozess nach IEC 60880 / TKE-R, Barrho 19.05.2011
Common aspects to the IEC 60880 (2006) Official title: Nuclear power plants - Instrumentation and control systems important to safety Software aspects for computer-based systems performing category A functions Scope: This international standard provides requirements for the software of computer-based I&C systems of nuclear power plants performing functions of safety category A as defined by IEC 61226. Note: The nuclear standard IEC 60880 could be regarded as a specific derivation of the IEC 61508. Additional topics are e.g. configuration management, Common Cause failure (CCF), tool qualification. The IEC 60880 does not provide methods like the IEC 61508 does. Page 9 Anwendung von Polyspace im Software Entwicklungsprozess nach IEC 60880 / TKE-R, Barrho 19.05.2011
Overview IEC 60880 Chapters of the IEC 60880 5 General requirements for software projects: e.g. project management, configuration management 6 Software requirements: e.g. self-supervision, periodic testing 7 Design and implementation: e.g. language and associated translators and tools 8 Software Verification: e.g. process, activities 9 Software aspects of system integration: e.g. software aspects of system integration plan 10 Software aspects of system validation: e.g. software aspects of system validation plan 11 Software modification: e.g. procedure, modification after delivery 12 Software aspects of installation and operation: e.g. on-site installation, security, operator training 13 Defenses against common cause failure due to software: e.g. implementation of diversity 14 Software tools for the development of software: e.g. selection of tools, requirements for tools 15 Qualification of pre-developed software: e.g. evaluation and assessment process Page 10 Anwendung von Polyspace im Software Entwicklungsprozess nach IEC 60880 / TKE-R, Barrho 19.05.2011
Specific Requirements to Tool Qualification Tool in scope: software tools used in the development of software for computers in safety systems of nuclear power (chapter 14.1) Appropriate software tools can increase the integrity of the software development process, and hence the software product reliability (chapter 14.1) can reduce the risk of introducing faults in the process (chapter 14.1) Support the software engineering process (chapter 14.2.1) have a defined limit of applicability (chapter 14.3.1) Why tool qualification? The tools used [ ] shall be verified and assessed to a level consistent with the tool reliability requirements the type of tool potential of tool to introduce faults (chapter 14.2.2) The tools used shall have sufficient reliability to ensure that they do not jeopardise the reliability of the end product. (chapter 14.2.2) Page 11 Anwendung von Polyspace im Software Entwicklungsprozess nach IEC 60880 / TKE-R, Barrho 19.05.2011
MTUs Validation Approach I To gain confidence in the suitability (appropriateness) and the correct function of the tool have to be ensured, MTU has developed a validation approach. According to this approach, a tool is marked as critical when the possibility is high - that the tool may introduce faults in the end product (e.g. compiler) - or that a fault in the end product may not be found due to a defective tool (e.g. verification tool). Other tools (e.g. word processing) are identified as non-critical. The project tool chain includes 15 non-critical and 8 critical tools. For every tool, a tool qualification document is produced (common tool data, tool identification ) For critical tools, a tool validation is performed according to a strategy defined in a validation plan. The results are documented in a validation report. Page 12 Anwendung von Polyspace im Software Entwicklungsprozess nach IEC 60880 / TKE-R, Barrho 19.05.2011
MTUs Validation Approach II The validation plan includes: implementation of validation strategy all validation measures verification criteria and method evaluation and acceptances criteria and is agreed upon with tool vendor. In the project lifecycle, the tool is 1. planning 2. selected 3. validated 4. used Measures from every category shall be selected. Those which can be performed by MTU are preferred. Confidence in appropriateness and function (A) Correct Function (B) Structured Development (C) Experience From Use Page 13 Anwendung von Polyspace im Software Entwicklungsprozess nach IEC 60880 / TKE-R, Barrho 19.05.2011
Application of Polyspace in the Project PolySpace is applied in the implementation and integration phases to fulfill the following topics: static analysis of the implementation (absence of run time errors, data flow) part of non-formal developer tests integration tests nightly build Criteria for evaluation of PolySpace test results according implementation plan: passed: justification for oranges, no reds passed with justification: justification for oranges and reds failed: no justification for oranges and/or reds Page 14 Anwendung von Polyspace im Software Entwicklungsprozess nach IEC 60880 / TKE-R, Barrho 19.05.2011
Pre-validation activities The following activities have to be done: - Collecting information about the tool: Supplier specification, legal aspects, tool specification - Selecting tool (criteria of the ISO 9126-1): Functionality, Security, Adherence to standards, reliability, usability, efficiency, - Definition of settings/configuration - Document interoperability with other tools - Incoming good inspection - How to use documents and installation document - Training - Definition of product release strategy, update strategy, replacement strategy - All information are documented in the Tool Qualification Document Page 15 Anwendung von Polyspace im Software Entwicklungsprozess nach IEC 60880 / TKE-R, Barrho 19.05.2011
PolySpace Validation PolySpace is identified as critical tool, due to potentially not detecting critical errors in the product. PolySpace is identified as a testing tool according to chapter 14.2.3 2). Specific requirements for this tool type are given in chapter 14.3.6 Automation of testing. Polyspace is marked as criticial: Tool validation must be carried out. Tool Validation Plan includes measures for: - correct tool function - structured development - proven in use Confidence in appropriateness and function (A) Correct Function (B) Structered Development (C) Experience From Use Page 16 Anwendung von Polyspace im Software Entwicklungsprozess nach IEC 60880 / TKE-R, Barrho 19.05.2011
Tool Validation - Correct Tool function Possible measures A.1 Test against specifications (e.g. tool manual, tool supplier s internal requirements) at MTU A.2 Providing results and documents about the development (by tool supplier) A.3 Black-box tests with defined input and expected output A.4 Carrying out tests according MTU demands (by tool supplier) Chosen measures and implementation Measure 1 (A.2): Provide DO178B Kit incl. test cases, procedures and expected results Measure 2 (A.1): Plan Software Test based on DO178B kit input Measure 3 (A.3): Perform Test execution Measure 4 (A.3): Provide Software Test Report Confidence in appropriateness and function (A) Correct Function (B) Structered Development (C) Experience From Use Page 17 Anwendung von Polyspace im Software Entwicklungsprozess nach IEC 60880 / TKE-R, Barrho 19.05.2011
Tool validation - Structured Development Possible measures B.1 Usually MTU has no access to internal development documents, so this measure can only be provided by the tool supplier B.2 Providing plans and reports about tool development or tool life cycle (by tool supplier) B.3 Providing certificates according international standards (by tool supplier) B.4 Audit of development, which is carried out by MTU Chosen measures and implementation Measure 5 (B.2): Provide process description Measure 6 (B.4): Perform process evaluation Confidence in appropriateness and function (A) Correct Function (B) Structered Development (C) Experience From Use Page 18 Anwendung von Polyspace im Software Entwicklungsprozess nach IEC 60880 / TKE-R, Barrho 19.05.2011
Tool validation Proven in use Possible measures C.1 Collection internal experience and information about the tool C.2 Collection external experience and information about the tool (by tool supplier) Chosen measures and implementation Measure 7 (C.2): Provide Maintenance Process Description Measure 8 (C.2): Perform impact analysis of known bugs Measure 9 (C.2): Provide Safety Manual Measure 10 (C.2): Provide Mathworks experience Measure 11 (C.1): Provide MTU experience Confidence in appropriateness and function (A) Correct Function (B) Structered Development (C) Experience From Use Page 19 Anwendung von Polyspace im Software Entwicklungsprozess nach IEC 60880 / TKE-R, Barrho 19.05.2011
Validation result (category A) Measure 1 (A.2): Provide DO178B Kit incl. test cases, procedures and expected results Mathworks provided the DO178B kit containing a tool qualification plan, test cases, test procedures, expected test results and the tools operational requirements. As the kit contained everything requested by MTU and fits to the used PolySpace version, this measure is rated: Ok, passed without deviations. Measure 2 (A.1): Plan Software Test based on DO178B input MTU derived a Software Validation Plan from the DO178B kit obtained during execution of measure 1 and has been reviewed by Dr. Barrho, fulfilling independence criteria Department. Measure is rated: Ok, passed without deviations. Measure 3 (A.3): Perform Test execution The tests have, after overcoming some obstacles regarding the adaption of the out-of-the-box test to the MTU configuration, been successfully executed and documented (see Measure 4). Measure is rated: Ok, passed without deviations. Measure 4 (A.3): Provide Software Test Report The test report contains all necessary information regarding used configuration, settings, test cases references, annotations to the test cases and the pass/failed result per test case. All 252 of the planned 252 test cases have passed the test run. Measure is rated: Ok, passed without deviations. Page 20 Anwendung von Polyspace im Software Entwicklungsprozess nach IEC 60880 / TKE-R, Barrho 19.05.2011
Validation result (category B) Measure 5 (B.2): Provide process definition Mathworks did not provide process definitions in terms of data that have been transferred and are available for storage. A WebEx-session was hold instead, whose details and evaluation can be found in. The free of deviations impression that was gained during the session is backed by the certificates from measure 6 and therefore measure 5 rated: Ok, passed without deviations. Measure 6 (B.4): Perform process evaluation The process evaluation was NOT performed as originally planned by performing an audit. This deviation was caused by the fact, that Mathworks could provide valid certificates that testimony that the development and the operational use of PolySpace matches the demands from ISO 26262, IEC 61508 and EN50128. See for details. This measure s outcome, in terms of the intended gain in knowledge and trust, is rated: Ok, passed without deviations. Page 21 Anwendung von Polyspace im Software Entwicklungsprozess nach IEC 60880 / TKE-R, Barrho 19.05.2011
Validation result (category C) I Measure 7 (C.2): Provide Maintenance Process Description The maintenance process is organized analog to the standard development process, expect that it is primary managed by bug reports and the integration if fixes is not bound to a specific new major tool version release schedule. Compare. The measure is closed with status: Ok, passed without deviations. Measure 8 (C.2): Perform impact analysis on known bugs Mathworks publishes a list of bugs for PolySpace on a monthly basis (for critical bugs weekly), including impact on the tools use, affected tool versions and unique ID to allow tracing of the bug. Measure achieves status: Ok, passed without deviations. Measure 9 (C.2): Provide Safety Manual MTU created a manual how to safely and correctly install, configure and use PolySpace within the project. The references and evaluation of this manual can be found in. Additional comments can be found within. This measure is finished with status: Ok, passed without deviations. Page 22 Anwendung von Polyspace im Software Entwicklungsprozess nach IEC 60880 / TKE-R, Barrho 19.05.2011
Validation result (category C) II Measure 10 (C.2): Provide Mathworks experience To prove the PolySpace in use experience, 3 customer case studies from project scenarios alike to this project were delivered by Mathworks. The information was evaluated in and achieved the status: Ok, passed without deviations. Measure 11 (C.1): Provide MTU experience The MTU experience is described and evaluated. The experiences meet the criteria defined in for the measure and therefore it is closed with status: Ok, passed without deviations. Page 23 Anwendung von Polyspace im Software Entwicklungsprozess nach IEC 60880 / TKE-R, Barrho 19.05.2011
Result and Conclusion 11 out of 11 measures have been performed as initially defined. The overall result is: Number of Measures for PolySpace 7.2 qualification: 11 Passed without deviations: 11 Passed with deviations: 0 Failed: 0 Successful measures coverage: 100 % Final status of Tool Validation: OK, passed without deviation - PolySpace: Confidence in appropriateness and tool function is demonstrated successfully - Entire tool strategy and tool processes are communicated to customer, final confirmation is outstanding Page 24 Anwendung von Polyspace im Software Entwicklungsprozess nach IEC 60880 / TKE-R, Barrho 19.05.2011
Lessons learned - Software cannot be tested until a defined quality level is achieved. Software has to be developed in an appropriated way. - A formal and structured development supports finding functional specification errors in a very early development stage. - Create a plan, execute it, document it helps quit a lot. Page 25 Anwendung von Polyspace im Software Entwicklungsprozess nach IEC 60880 / TKE-R, Barrho 19.05.2011
Thank you very much for your attention.