The Role of CM in Agile Development of Safety-Critical Software

Transcription

1 The Role of CM in Agile Development of Safety-Critical Software Tor Stålhane1, Thor Myklebust 2 1 Norwegian University of Science and Technology, N-7491, Trondheim, Norway 2 SINTEF ICT, Strindveien 2, N-7491 Trondheim, Norway stalhane@idi.ntnu.no, thor.myklebust@sintef.no Abstract. Agile development is getting more and more used, also in the development of safety-critical software. For the sake of certification, it is necessary to comply with relevant standards in this case IEC and EN In this paper we focus on two aspects of the need for configuration management and SafeScrum. First and foremost we need to adapt SafeScrum to the standards needs for configuration management. We show that this can be achieved by relative simple amendments to SafeScrum. In addition in order to keep up with a rapidly changing set of development paradigms it is necessary to move the standards requirement in a goal based direction more focus on what and not so much focus on how. Keywords: Safety critical systems, agile software development, configuration management, IEC61508, EN Introduction It is always a challenge to change software safety critical or not. In this paper we will discuss challenges related to changes and change processes when using agile development. We will do this from two perspectives IEC 61508: 2010 (a generic standard often used in industrial automation) and EN 50128: 2011 (railway signalling systems). Change is always tightly connected to configuration management (CM) which is a well-known process. The challenge is more important for agile development that in any other development paradigm since agile development promises to embrace change. The challenges related to CM will, however, increase when we use agile development since changes will be more frequent possibly several changes included in each sprint. Changes during agile development come from several sources, e.g.: New requirements added after the development process has started Changes to existing requirements due to new knowledge or new customer needs New risk and hazards due to changes in the operating environment Refactoring tidy up the code, which is important in agile development Not-accepted user story implementation from a sprint adfa, p. 1, Springer-Verlag Berlin Heidelberg 2011

2 All changes, irrespective of source, represent challenges for both the developers and for the system s integrity, e.g.: Testing. Which tests need to be re-run after the changes the need for regression testing has to be evaluated Change impact analysis. How will the change affect system Complexity both IEC and EN require that the system complexity shall be controlled Safety which safety and hazard analyses should be checked or repeated The CM process is well known and there is a plethora of tools available to support it. However, none of the proposed methods handle the problems pertaining to change impact analysis. Traditionally, the processes have been heavy on management and on documentation. None of these concepts fit well with agile development. 2 Related Works First and foremost, there exist a well-known standard for CM IEEE Std. 828 [10] which should be used as a reference document. When searching for existing work related to agile configuration management, we note that there exist few academic articles and that the few that exist mostly can be summarized in one sentence: CM is important, also for agile development. If we look for papers on how to do CM during agile development, we find that the majority of relevant articles are published on blogs e.g., [1, 2], although there are exceptions, e.g., [3, 4]. It is important to note the conclusion from [1]: CM can be adapted to an agile iterative process without any problems and the summary from [4]: Modern CM tools can handle any number of configuration items without any problems and, therefore, controlling everything should be the tendency in agile methods. In addition, there exist some books on the topic, e.g., [5-7]. The current conventional wisdom is best summed up in [1] edited by the authors: Add support for parallel multiple development branches Avoid centralized Change Control Boards (CCB) that controls all changes. Control non-strategic decisions on changes in the distributed organization of Scrums. Reserve centralized decisions for changes that impact the whole program or organization. Let the agile team assumes the CM role instead of a dedicated Configuration Manager. Use automated tools Automated continuous integration helps to reduce the integration and testing delays and allow quick feedback about the product quality. Continuously observe and adapt the CM tools and process.

3 3 Agile Development and SafeScrum SafeScrum [8] is an adaptation of Scrum to make the development process comply with IEC 61508, EN and IEC , ed. 2. The main adaptations are shown in Fig 1. Fig. 1: The SafeScrum development process. A complete description of Scrum and its development into SafeScrum can be found in [9]. The other important adaptation is the introduction of separation of concerns see Fig 2. The main message here is that SafeScrum cannot handle everything we have decided to only handle software development, which is thus separated from the rest of the process described in IEC and EN The term separation of concerns stems from software design and we found this to be a useful and descriptive term when we designed the SafeScrum process. Fig. 2: Separation of concerns

4 SafeScrum gets its requirements from the system s SSRS and incorporate methods and techniques described in annex A, part 3 of the standard as required by the designated SIL. Each increment from SafeScrum goes through a RAMS validation and the result is either that it is OK, that it needs to be modified in another sprint or that it requires a new hazard analysis. 4 CM in Two Standards 4.1 CM in IEC The following sections of IEC are related to CM: Part 1 sections and These two sections states that the project shall have a procedure for CM and the project needs configuration identification of items under test, the procedure applied for testing and the test environment Part 2 sections and D.2.1. These two sections are concerned with the CM of the complete E/E/PE system hardware and software Part 3 sections 6.2.3, , and Section is the most important one since this section defines the requirements for the CM process to be used. We will look at this process in some more details below also because part 3 is the part of the standard that is concerned with software development. The other sections are concerned with CM history and CM for tools of categories T2 and T3 Part defines the term configuration data and also defines configuration baseline as the information that allows the software release to be recreated in an auditable and systematic way Parts 5 and 6 no relevant sections on CM Part 7 appendix C.4.3 and C The first of these two appendices state that the CM tools shall be certified and the second one describes the goals of CM Part 1, part 2, part 4 or part 7 will have an impact on the chosen development method while part 3 is important. According to Software configuration management shall: a) apply administrative and technical controls throughout the software safety lifecycle, in order to manage software changes and thus ensure that the specified requirements for safety-related software continue to be satisfied b) guarantee that all necessary operations have been carried out to demonstrate that the required software systematic capability has been achieved c) maintain accurately and with unique identification all configuration items which are necessary to meet the safety integrity requirements of the E/E/PE safetyrelated system. Configuration items include at least the following: safety analysis and requirements; software specification and design documents; software source code modules; test plans and results; verification documents; pre-existing software elements and packages which are to be incorporated into the E/E/PE safetyrelated system; all tools and development environments which are used to create

5 or test, or carry out any action on, the software of the E/E/PE safety-related system d) apply change-control procedures: to prevent unauthorized modifications; to document modification requests to analyse the impact of a proposed modification, and to approve or reject the request to document the details of, and the authorisation for, all approved modifications; to establish configuration baseline at appropriate points in the software de- velopment, and to document the (partial) integration testing of the baseline to guarantee the composition of, and the building of, all software baselines (including the rebuilding of earlier baselines). e) ensure that appropriate methods are implemented to load valid software elements and data correctly into the run-time system f) document the following information to permit a subsequent functional safety audit: configuration status, release status, the justification (taking account of the impact analysis) for and approval of all modifications, and the details of the modification g) formally document the release of safety-related software. Master copies of the software and all associated documentation and version of data in service shall be kept to permit maintenance and modification throughout the operational lifetime of the released software. The majority of these requirements will not be influenced by a choice of using an agile approach in this case SafeScrum. Even most of the requirements in part 3, section 6.2.3will not be influenced by agile development. The mail challenges are point c and f. There is nothing there that cannot be fulfilled when using agile development but the resources needed may be large, depending on the tools used. It is thus important to agree on when we define a new configuration. This decision should be reached with the assessor and may be with the customer before development starts. According to requirement c, we need CM for the following documents: safety analysis and requirements, software specification and design documents, software source code modules, test plans and results, verification documents. We could for instance define configurations to be the result of: Each sprint. Use CM to recreate the status of the system after any chosen sprint. Separate CM-item sprints. The system s state can be recreated only to the states at these points The complete system. Only the final, delivered system can be recreated 4.2 CM in EN Even though both EN and EN also are relevant for railway applications, we have decided to only look at EN since we focus on software. The following sections in EN50128 are related to CM:

6 Section 4 requirement 4.1, which specifies that development environment shall contain a system for CM Section 5 requirement , which specifies that the project shall have a CM plan, drawn up from the start of the project Section 6 requirement and are related to tests. Requirements and handles quality assurance requirements, while requirements and handles change control. Each project needs a CM plan. All software documents and deliverables shall be placed under CM control and changes to these documents shall be authorized and recorded. The CM system shall cover the software development environment used during a full lifecycle. Each test specification shall document test environment, tools, configuration and programs, requirement and that we need to identify the configuration of all items involved in a test report. All changes shall be done according to the CM plan and deliver a new CM record. Section 7 requirements , , , and Requirement requires that each component shall have a CM history attached Requirement requires that each component shall be under CM before we start documented testing Requirements , and require that the software and hardware integrations shall identify the configurations of the elements involved and the validation report Section 8 has configuration requirements for configuration data of the system. In our opinion, this is outside the scope of this paper except for requirements and which requires that the configuration data shall be placed under configuration control Section 9 contains requirements for deployment and maintenance, which both are topics outside the scope of this paper. It is, however, important to note that requirement opens up for incremental deployment Note that EN does not mention regression testing. This should, however, be included in the next version of the standard 4.3 Prescriptive vs. Goal Based Standards The most convenient way to start this discussion is to use a definition of CM. In order not to favour any of the two standards that we discussed here, we will use the definition given by IEEE [12] which states that configuration management is the process of Identifying and defining the items in the system Controlling the change of these items throughout their lifecycle Recording and reporting the status of items and change requests Verifying the completeness and correctness of items

7 In order to make standards more robust when techniques and methods change, we work to make as many as possible of the standards goal-based. I.e., instead of saying what you shall do, we want to focus on what you shall achieve. To show what we mean, we will show a possible goal-based approach for some of the CM requirements for the two standards discussed above. The main goal for both standards is to be able to recreate a consistent set of documentation, code, test data and tools that is maintained throughout the development project. This goal alone covers IEC 61508, part 1, part 2 and part 7 plus EN 50128, section 4, section 5 and section 6. For IEC 61508, only part 3 is important, since it describes a set of concrete requirements to CM while part 4 only contains relevant definitions and parts 5 and 6 have no relevant sections pertaining to CM. A closer look at part 3, section shows that it only says that you should be able to recreate the complete project state at any defined time. For EN 50128, sections 4 and 5 just say that a project shall have a plan and a system for CM. Section 6 adds that the CM shall cover the software development over a full lifecycle. We have earlier stated that all the development requirements stated in annex A and B of IEC only contains sound software engineering practices. It would not be unreasonable to claim the same for the two standards requirements for CM. 5 The SafeScrum Approach to CM 5.1 Some general considerations First and foremost: CM is more important when using an agile development approach then when using a waterfall process. There are several reasons for this: Builds baselines more frequent Have more frequent deliveries/releases Have more and more frequent changes If we look at the table supplied by [4], we see that software CM is not a part of Scrum and thus needs to be added also to SafeScrum. Table 1: CM in agile development methods Method Adaptive Software Development Crystal family of methodologies Dynamic System Development Method Extreme Programming Feature Driven development Software configuration management approach SCM not explicitly considered Tool viewpoint on SCM - SCM explicitly considered SCM partially considered SCM explicitly considered - Practices related to SCM All changes during the development must be reversible Collective ownership, small releases and continuous integration Configuration management, regular builds

8 Pragmatic Programming Tool viewpoint on SCM Source code control Scrum SCM not explicitly considered - Note that [1] advices us to avoid a centralized Change Control Boards and to reserve centralized decisions for changes that impacts the whole program or organization. The rest should be handled by the SafeScrum team. The term component is often used both in IEC and in EN From the definition given in EN well-defined interfaces and behaviour with respect to the software architecture it is reasonable to interpret component as a functional unit see IEC , Functional unit: entity of hardware or software, or both, capable of accomplishing a specified purpose. 5.2 Regression Testing One important issue not explicitly related to CM but important for each change which is the reason why we need CM is regression testing. When we do test-first, e.g. TDD, we make a set of tests based on the current set of requirements (user stories) and develop software with the goal that the piece of code currently developed shall pass the tests. However, we will also have a lot of tests developed for previous requirements. In addition, the tests developed for a user story will in most cases depend on a set of stubs, fakes or mocks. It is not obvious that these tests can be run on any later version of the system without much ado. We see two practical ways out of this: Organize the user stories in such a sequence that we avoid or at least minimize the need for stubs, fakes and mocks. See for instance [11] Have two sets of tests one for the total system and one for each increment. The first will be a system test that is increased for each sprint while the other one is a set of tests only relevant for the designated sprint see Fig 1. The system test could be maintained and run by the same persons who do the RAMS validation in the current SafeScrum model while the other tests could by the responsibility of the development team see Fig 2 Another important consideration is the need to retest only what was affected by the last sprint. To achieve this we will use two important mechanisms (1) connecting tests to user stories and (2) using the trace information. We need traces from user stories to code and from user stories to tests. This will give us information about which tests are related to which code units. We only need to retest components that are changed or receive input (directly or indirectly) from changed components. By having efficient tools for automation, it is possible to enable regression testing of relevant parts of the system, with increased frequency. When developing safety-critical systems, changes may have effects that are outside the changed modules or components. This challenge is handled by change impact analysis. Even though this is important it is not part of CM. We have, however, discussed this problem and the suggested SafeScrum solutions extensively in [9]. The interested read should consult this paper and its references.

9 5.3 SafeScrum CM in IEC and EN The most important statement related to CM is that the Software Quality Assurance Plan, Software Verification Plan, Software Validation Plan and Software Configuration Management Plan shall be drawn up at the start of the project (i.e., outside SafeScrum) and be maintained throughout the software development life cycle. For IEC 61508, this holds also for hardware and for tools of category T2 and T3 plus the documents listed in part 3, section The important thing for SafeScrum is to have a procedure at the start of each sprint where all plans are updated when necessary. This can be done either by the SafeScrum team itself as part of the sprint planning process or by the people who developed the plans, using information from the SafeScrum team. EN also requires that all information related to testing e.g., environment, tools and software shall be included in CM. Note also that the standard requires all components to be under CM before we start documented testing. Testing done during development using e.g., TDD does not need to be included. In most projects, documented testing only includes integration testing and system testing. The only part that we need to go through in some detail is IEC 61508, part 3, section This section specifies that we shall Have administrative and technical control throughout the lifecycles Apply the correct change control procedures and document all relevant information for later safety audits i.e., that the CM job is done properly Have control over all identified configuration items Formally document the releases of safety-related software An important challenge to the SafeScrum process is the first statement: administrative control throughout the lifecycles. For the other CM requirements, the challenge for SafeScrum is not to fulfil the requirements but to decide how often and under what circumstances. Most of the information needed for efficient CM is created automatically by tools. We suggest the following approach: Management decides at which milestones a new configuration should be defined. This is done before the project starts and is mentioned in the CM plan. The responsibility for managing the CM is normally assigned to the quality assurance department (QA). All code and data are tagged during check-in. The tags are administrated by the QA but used by the SafeScrum team. The QA and the SafeScrum team have regular meetings that focus on CM and other QA-related issues. EN adds some requirements all components shall have a CM history attached to it and that the validation report and configuration data shall be included in the documents under CM. In addition, the integrations shall identify the hardware and software components used. It is also worth mentioning that EN requires that each component shall be under CM before we start documented testing. Thus, the components need not be under CM during test-driven development.

10 To sum up; we have considered all requirements to CM in IEC and EN There are no practises in SafeScrum that prevent or hinder the use of standard CM methods and procedures. SafeScrum needs two add-ons (1) tagging of code and data at check-in and (2) regular meetings between the SafeScrum tam and the company s QA-department. 5.4 Threats to Validity As always with this type of discussions, there are two threats to the validity of our discussions and conclusion: (1) have we understood the standards and (2) have we understood CM. We claim that we have understood The two standards, based on the practical experiences the two authors have with the standards one is a certified assessor for EN while the other author has worked with IEC in industrial settings. CM, based on the practical experience of one of the authors with software development 6 Summary and Conclusions We have had a detailed walk-through of the two standards IEC and EN with focus on CM and change management when we use an agile development process in this case SafeScrum. The challenges related to CM increase when we use agile development since changes will be more frequent. There are just a few requirements in either of the standards that prevent the use of SafeScrum as is. The following changes (additions) are needed: A new process at the start of each sprint to do necessary updates to the CM plan when needed. The SafeScrum team should cooperate with QA in this process. A separation of testing into development tests e.g., TDD which is the responsibility of the SafeScrum team and system and integration tests, which are the responsibility of the RAMS process All tools used, all documents generated and all plans should be under CM. An efficient tracing tool is needed, e.g., to keep track of the relationships between user stories, test cases and code In addition we suggest changes to the two standards under discussion in order to make the requirements goal-based in order to be able to keep up with new ideas and concepts in software engineering.

11 Acknowledgements This work was partially funded by the Norwegian Research Council under grant # (the SUSS project). References 1. Rey-Mermet, B.: Agile Configuration Management-overview, 2013, May Norin, J.: Lean Configuration Management. Evolving the CM Discipline Through the Agile Paradigm Shift Lindroth-Olson, Prentare, O.: What, When, Why and How. Introducing Software Configuration Management in Agile Projects. February 28, Koskela, J.: Software configuration management in agile methods. VTT-publications 514, Espoo Moreira, ME.: Adapting Configuration Management for Agile Teams: Balancing Sustainability and Speed, Wiley ISBN: October Jonassen Hass, AM.: Configuration Management Principles and Practice Addison-Wesley Professional 7. Black, R.: Managing the Testing Process: Practical Tools and Techniques for Managing Hardware and Software Testing, John Wiley and Sons, 2002, ISBN , Stålhane, T, Myklebust, T. and Hanssen, GK.: The application of Safe Scrum to IEC certifiable software. ESREL, Helsinki Stålhane, T., Hanssen, GK., Myklebust, T. and Haugset, B.: Agile Change Impact Analysis of Safety Critical Software, SASSUR, 2014, Florence, Italy 10. IEEE: Standard Glossary of Software Terminology. IEEE Std Bjerke-Gulstuen, K., Wiik Larsen, E., Stålhane, T., Dingsøyr, T.: High level test driven development Shift Left: How a large-scale agile development project organized testing, XP2015, Helsinki, Finland 12. IEEE: Standard for Configuration Management in Systems and Software Engineering, IEEE Std