Integrating model-checking architectural analysis and validation in a real software life-cycle

Integrating model-checking architectural analysis and validation in a real software life-cycle Daniele Compare 1, Paola Inverardi 2, Patrizio Pelliccione 2, and Alessandra Sebastiani 1 1 Lab NMS C2, Marconi Mobile, L Aquila, Italy, {danielecompare,alessandrasebastiani}@marconicom 2 University of L Aquila, Computer Science Department Via Vetoio 1, 67010 L Aquila, Italy {inverard,pellicci}@diunivaqit Abstract In this paper we describe the use of a model-checking based tool, Charmy, in an industrial software development environment The goal of the project is to model and analyze the software architecture of a software system currently under development We have defined and formally checked its overall architecture By zooming into relevant subsystems we have been able to identify a set of uncorrect behaviors We have used an iterative process, where both the architectural sub-systems models and the properties to be checked were defined, checked and revised several times The results of the experience allow an evaluation of the effort to use the analysis framework in the industrial software development environment and an assessment of the efficacy and the role of the architectural analysis in the software development process The main relevance of this experience is in the effort to smoothly integrate the use of model-checking techniques in a standard software life-cycle, in particular concerning the discovery and definition of architectural properties 1 Introduction In the last years there has been a growing interest for the architectural level description of software systems New tools and architectural languages have been proposed to specify the Software Architecture (SA) of large scale systems SA description represents the first, in the development life-cycle, complete system description It provides at the high abstraction level of components and connectors both a description of the static structure of the system and a model of its dynamic behavior The aim of SA descriptions is twofold: on one side they force the designer to separate architectural concerns from other design ones On the other, they allow for analysis and validation of architectural choices, both behavioral and quantitative in order to obtain better software quality in an increasingly shorter time-to-market development scenario Despite the high level of abstraction very often SA can still be too complex to be managed A way to tackle system complexity consists of representing the

system through several view points [18, 16, 9]; as a direct consequence, different models are used to represent the different views The use of views allows to handle different aspects of the system separately and to manage the description of real systems Several slightly different views (logical, module, deployment and so on) are defined in different approaches [16, 9, 4]; different notations, generally graphical, have been introduced for their representation All the approaches agree that dynamic aspects are orthogonal to all the views and they allow for analysis and validation of architectural choices Several works use these dynamic descriptions for testing [5, 21], analysis and model checking [3, 7, 8, 15] Many specification languages [14, 1, 3, 17] have been introduced to specify and model system behavior at the architectural level but there is not a standardized way to represent SA dynamics In the current industrial practice state-based machines and scenarios are the most common used tools to model behavioral aspects: state diagrams describe components behavior while scenarios (message sequence charts or sequence diagrams) identify how they interact Although very expressive, this approach has two drawbacks with respect to analysis and validation The first one deals with the model incompleteness of the system specification: notably in industrial contexts, these models are not completely specified Thus we can in general rely only on a finite set of scenarios that complement (sub-) systems state models The second one is a problem of view consistency: the use of several views facilitates the growing of inconsistencies between views that represent different aspects of the same system [19, 20] Indeed, state diagrams and scenarios provide alternative views of the system: these views are not independent and can erroneously specify contradictory or inconsistent behaviors The approach proposed in Charmy (CHecking Architectural Model consistency) [12, 13, 11] has been thought to deal with these problems Charmy allows the use of the most common notations employed in the current industrial practice to model behavioral aspects: ie, state and sequence diagrams (scenarios) used to describe a system software architecture through two different views Charmy, starting from these (incomplete) dynamic views, synthesizes, through a suitable translation into Promela language (the specification language of the SPIN model checker), an actual SA complete model that can then be executed and verified in the SPIN model checker [24] The resulting model can then be validated with respect to a set of selected scenarios which are suitably translated in Linear Temporal Logic (LTL) formulae The validated model can be used as a first prototype to control and eliminate problems of the software architecture model such as deadlock, correctness of properties, starvation, etc In industrial contexts like the one participating in this experience, (Unified modelling Language) UML [2] is commonly used for the system architectural description The model checker SPIN, is a widely distributed software package that supports the formal verification of concurrent systems allowing to analyze their logical consistency by on-the-fly checks, ie without the need of constructing a global state graph, thus reducing the complexity of the check It is the core engine of Charmy and it is not directly accessible by a Charmy user

In this paper we describe the use of Charmy in an industrial software development environment The goal of the project is to model and analyze an industrial software system currently under development We have defined and formally checked its overall architecture By zooming into relevant sub-systems we have been able to identify a set of uncorrect behaviors We have used an iterative process, where both the architectural sub-systems models and the properties to be checked were defined, checked and revised several times The results of the experience allow an evaluation of the effort to use the analysis framework in the Marconi software development environment and an assessment of the efficacy and the role of the architectural analysis in the software development process The main relevance of this experience is in the effort to smoothly integrate the use of model-checking techniques in a standard software life-cycle, in particular concerning the architectural properties discovery and definition In the literature much work has been directed to improve the model generation phase, little attention has been put in the properties to be proved In general these are assumed to exist as part of the problem specification Research efforts in system properties and in the corresponding (temporal) formula mainly concern the ability to informally describe a property and then translate it in a formal formula [23, 10] Little or no attention is devoted to the problem of properties elicitation, that is the ability to determine from the software artifacts at hand which are the relevant properties to check for We can certainly assume that at the requirements level, explicit stated behavioral requirements, like my system has always to perform two b actions whenever it gets in input an a action exist These are global system requirements, that is they characterize the input-output behavior of the system Once a software architecture is specified, it is still possible to check if it guarantees the global functional requirements, but if this is not the case we need to be able to refine the global property into sub-properties that focus on relevant portions of the system software architecture In Section 2 we summarize the framework and the way it works Sections 3, 4, 5 describe the framework applied on the case study Section 3 describes the system specification of the case study and its main properties Section 4 shows the modelling in the framework defining the components and its interaction Subsection 41 is a modelling example while 42 describe two sub-systems representing most critical parts of the system Section 5 shows the analysis process applied to the NICE case study, showing the detected problems and the proposed solutions Section 6 summarizes our experience and presents conclusions and future work 2 The Framework The framework we used in the project tries to integrate several tools in order to correctly drive the software architect during the system modelling design phase The framework involves the UML notation, and the Charmy tool

UML is the design support used in the industrial context we operate It is used for all software development phases, in particular as Architectural Description Language (ADL) Charmy is a tool that helps the software architect to draw architectural state diagrams and scenarios and automatically translates these models in Promela code and in LTL formulae expressing suitable system behavioral properties Charmy performs the model verification using the model checker SPIN Details about Charmy s related algorithms are given in [12] In the following we summarize how we used the proposed framework starting from the system specification described in UML (use cases, activity diagrams, etc) and requirement documents in natural language: 1 identify the system components and describe their architectural behavioral aspects through state diagrams; 2 define the significant dynamic interactions among components through sequence diagrams; 3 use the two above set of views to produce a SA model in Promela code The sequence diagrams are also separately translated in LTL formulae and used to test the correctness of the SA Promela model; 4 prove the global behavioral properties, stated in the requirements, on this first model of the system; 5 identify critical sub-systems and their relevant properties in order to guarantee the global properties; 6 refine the sub-systems architecture and iterate the process; The most critical part of this approach is the selection of the LTL formulae representing the properties that must be satisfied by the system In a simplistic approach we can think that the set of the LTL formulae can be extracted from the system specification This is only true for the global behavioral properties The actual formulae that we prove at the architectural level must be more refined in order to point out critical architectural interactions There is no automatic way to do this refinement step, which is connected with the identification of the most critical sub-systems that need a deeper modelling and verification In the following we describe the process followed in the project, trying to highlight the motivations that allowed us to focus on significant sub-systems and to identify relevant properties Our approach is based on the construction of an executable prototype of the system software architecture in order to understand its behavior and its features It is clear that we are not able to detect all potential erroneous behaviors in one verification step For this reason we iterate the process; at each iteration we focus on a system part refining its specification and identifying other potentially critical sub-parts To some extent, our approach is similar to extracting test cases In eliciting test cases a tester focusses on the critical parts of the system in order to verify his intuitions In the same way we elicit the properties that represent potential flows and check them on the system model by using model checking techniques

3 The NICE case study The Naval Integrated Communication Environment (NICE) operates in a naval communication environment The purpose of the system is to fulfill the following main functionalities: provide voice, data and video communication modes; enable system aided message preparation, processing, storage, retrieval distribution and purging; implement radio frequency transmission and reception, variable power control and modulation; provision of remote control and monitoring including detection of equipment failures in the transmission/reception radio chain and the management system elements; provision of a data distribution service; implement communications security techniques to the required level of evaluation, accreditation and certification WS NICE MS C2 CTS PROXYAGENT C1 PROXY AGENT CTS Radio, Transmitting Antenna EQUIPMENT VUU Fig 1 NICE static software architecture The NICE Management Sub-system (NICE MS) is the more critical component It controls both internal and external communications, supporting the following functionalities: fault and damage management; system configuration; security management; traffic accounting;

performance management The NICE MS complexity and heterogeneity (different hardware and operating systems and relative applications) together with its real time context needs the definition of a precise software architecture to express its coordination structure The system involves several operational consoles that manage the heterogeneous system equipment including the ATM based Communication Transfer System(CTS) through blind Proxy computers For this reason the high level design is based on a manager-agent architecture that is summarized in Figure 1, where the Workstation (WS) component represents the management entity while the Proxy Agent and the CTS Proxy Agent component represent the interface to control the managed equipment and the CTS respectively Subsystems are connected by connectors (C1 and C2) that allow the communication between them 4 Modelling NICE in the framework The first step in modelling the NICE system is to define its static SA, as shown in Figure 1 The second step is to define the state diagrams for the components and the scenarios defining components interaction This second step is part of the usual Marconi software development process, therefore the effort to provide input for Charmy has been minimal In the following subsection we present some state diagrams and scenarios related to the modelled system 41 An excerpt of the system behaviors In this Subsection we present only a small example of the modelled system, just to give an idea of the modelling technique A1!fail1VUU_CTS!Recovery1VUU_CTS (a) A2 B1!Recovery2VUU_CTS B3?fail1VUU_CTS?Recovery1VUU_CTS (b) B2!fail2VUU_CTS B4 C1?fail2VUU_CTS C2?Recovery2VUU_CTS (c) Fig 2 Components state diagrams Interested readers can see [22] for a full treatment of the system Starting from the NICE global model we consider the situation that occurs when the

CTS Proxy Agent detects a CTS or VUU (Voice User Unit) failure together with the relative consequent actions Note that a VUU is an ISDN voice terminal equipment to access the ship s operational internal and external communications, including the ship s telephone system In Figures 2a, 2b and 2c are shown the CTS, the CTS Proxy Agent and the WS state diagrams, respectively To help diagrams comprehension, note that the symbol? denotes a message reception and the symbol! denotes a message sending In Figure 3 it is represented how these components interact each other through a sequence diagram Since at present Charmy does not directly interface with UML, architectural state charts and sequence diagrams had to be directly provided However, this was not a big effort due to the closeness among the UML and the Charmy notations The state diagrams in Figure 2 and the scenario in Figure 3, obtained with the Charmy editors, are then automatically translated into Promela code and LTL formulae to perform the third step of the process described in Section2 For further details on the complete system verification the interested reader can refer to [22] CTS CTS Proxy Agent WS fail1vuu_cts fail2vuu_cts Recovery2VUU_CTS Recovery1VUU_CTS Fig 3 Components interaction sequence diagram Because of the large system context (26 components state diagram, 85 exchanged messages), in the following we present the analysis of only two subsystems These sub-systems have been identified as highly critical, and therefore eligible for deeper modelling and analysis because of the high risk of mismatch and interoperability issues due to the different nature of their interacting components (hardware, software, different operating systems) 42 NICE Management Sub-system The NICE Management Sub-system, in Figure 1 surrounded by dashed line, is composed of three interacting components as described at the end of Section 3

Due to the nature of the system communications, it was decided to separately model and analyze the communication between WS and CTS Proxy Agent and the communication between WS and Proxy Agent The size of the two subsystems is limited to 3 components state diagram and 26 messages for the first one and 3 components state diagram and 22 messages for the second one As we will see later on, the analysis of these models allowed the identification of several potential system communication problems In the following we briefly describe the two sub-systems and their interactions in order to better understand the achieved analysis results WS-CTS Proxy Agent communication is based on the SNMPv2 protocol [6] where WS is the manager entity while CTS Proxy Agent is the managed agent one Messages exchanged are the following: Set PDU: the manager asks the agent to change a specified variable of agent MIB; Get PDU: the manager asks the agent to report value of a specified variable of agent MIB; Notify PDU: the agent acknowledges a received Set/Get PDU to the manager; Trap PDU: the agent warns the manager about its own MIB changing WS-Proxy Agent communication is based on a proprietary protocol over LAN The Proxy Agent is directly connected to the managed equipment and simply converts messages coming from WS in a format that the equipment is able to understand and vice-versa In doing such activity, it also monitors the equipment state through a timer-based polling In order to avoid useless network overload, it forwards to the WS only the variations of the equipment state 5 The analysis process Let us shortly recall how the analysis process described in Section 2 is carried on First we analyze the overall NICE system in order to understand its main dynamic features The analysis results, that we not report here, show the existence of potential dangerous behaviors Therefore we identify and analyze in detail two sub-systems These two sub-systems realize the communications into the NICE MS as defined in Subsection 42 In particular the first one focus on the possible loss of messages during the communications between the WS and Proxy Agent (Subsection 51) The other one allows monitoring the communication between the WS and the CTS Proxy Agent (Subsection 52) The interest on the latter model is driven by the fact that the interacting components are heterogeneous in terms of operating system platforms and are developed by different project partners Note that the two sub-systems models are still the product of an abstraction process Some components were leaved out, focusing only on interesting ones and refining the communication among them For further details the interested reader can see [22] In the following subsections we explain by means of sequence diagrams the results obtained by using the model checker SPIN We decide to translate SPIN traces in sequence diagrams to make the analysis output comprehensible to system designers

51 Results for WS-Proxy Agent communication In this sub-model several anomalous scenarios were verified using a Pentium III 833 MHz personal computer with 256 MB of ram and Windows NT 40 The scenario expressed by the sequence diagram of Figure 4a represents the result analysis for the LTL formula in Figure 5 It outlines how the loss of a message ( Data msg2 ) does not allow Proxy Agent to update WS about the state of the managed equipment until a new state change applies on the Equipment itself In Figure 4 the [State X] labels on each lifeline component represent the local Equipment state view Referring to the scenario Figure 4a, an state inconsistency occurs between WS and the managed Equipment After the reception of the set2 message the Equipment changes its state to [STATE B]; after it applies the request, changing its state to [STATE C] while WS still knows [STATE B] It is lost [State A] WS set1 ack1 Data msg2 Proxy [State A] set2 ack2 Polling Data msg1 Polling Equipment [State A] Time to apply the set request WS [State A] set1 It is lost Not responding1 It is lost Data msg2 Proxy [State A] set2 ack2 Timeout Polling Data msg1 Polling Equipment [State A] Time to apply the set request It does not warn the WS because the state is not changed ack ack (a) (b) Fig 4 WS PA anomalous scenarios In the sequence diagrams the lost messages are highlighted with a note In Figure 5 we present a high level description of the LTL formula which describes the scenario in Figure 4a; we are omitting details about the actual formula syntax because a complete description requires the knowledge of the data structures in the Promela code The check of the LTL formula shows after a few minutes of processing the existence of a path that fakes the formula The highlighted problem is the loss

set1 happens in the time t_set1 set2 happens in the time t_set2=t_set1+1 ack2 happens in the time t_ack2=t_set_2+1 ack1 happens in the time t_ack1=t_ack2+1 polling happens in the time t_polling=t_ack1+1 datamsg1 happens in the time t_datamsg1=t_polling+1 datamsg2 happens in the time datamsg2=t_datamsg1+1 polling happens in the time t_polling= t_datamsg2+1 ack happens in the time t_ack=t_polling+1 Fig 5 LTL formula: an example of datamsg2 message and it suggests to introduce a timer which allows the Proxy Agent to resend periodically the equipment state The sequence diagram in Figure 4b shows the system behavior caused by the expiration of the Proxy Agent s timer due to the loss of the equipment ack2 message This implies a not responding1 message sent from the Proxy Agent to WS which signals an equipment malfunctioning Note that such timer is set by the Proxy Agent every time that it sends a message to the equipment The loss of the message sent by the Proxy Agent and containing the new equipment state induces inconsistency In fact WS continues to believe that the managed Equipment is in a not responding state ([STATE B]), even if the managed Equipment is correctly working This situation persists until a new equipment state change occurs The LTL formula resulting from this scenario was also validated thus indicating a system behavior that has to be differently managed In an industrial context it is unfeasible to write by hand the LTL formulae, which are complex ones even in the above abstract version On the contrary, the sequence diagrams are more intuitive and highly familiar Charmy offers to the user an interface to specify scenarios and a set of options on how these scenarios can be interpreted as behavioral properties For example, among these options, a user can say that a scenario has to be verified for all possible system behaviors, or that it represents just one possible behavior Other options allow for more sophisticated verifications 52 Results for WS-CTS Proxy Agent communication In this sub-model the anomalous scenario of Figure 6 was verified using the same hardware configuration described in the previous subsection Note that the communication between the two entities WS and CTS Proxy Agent is based on a retry mechanism for sending at least N times (eg N=3) the same message when no acknowledge or Msg1 corrupt is received Only after the failure of all the N retries the communication can not occur and a Nack Msg1 message is sent to the NICE MS Manager In the above scenario after N attempts, WS signals a failed sending since an acknowledge for the sent message is generated

NICE MS Manager Msg1 request WS CTS Proxy Agent Msg1 timer expired Msg1 Msg1 corrupt Msg1 timer expired Retry = N Msg1 timer expired Nack Msg1 Ack Fig 6 WS CTS PA anomalous scenarios later by the CTS Proxy Agent The analysis of this communication protocol allowed us to identify a mismatch: WS believes its request was not accepted while CTS Proxy Agent accepted it This showed the critical importance of a right timer setting choice The resulting LTL formula was validated after less than ten minutes of processing 6 Conclusions and Future Work In this paper we presented our experience in integrating Charmy in the Marconi software development environment Only few results about the NICE MS system are reported in this paper: more details on the complete system model can be found in [22] The goal of the experiment was twofold: 1 to evaluate the effort to use the analysis framework proposed; 2 to evaluate the efficacy and the role of the analysis itself Since state and sequence diagrams are commonly used in industry software development practice, it is easy and quick to use the framework, given that Charmy s inputs subsumes UML diagrams In order to follow this process no

particular skills is required since the framework feedback is easily comprehensible for a software architect The time required for the verification is strongly dependent on the size of the model Considering the abstraction used in the case study, the verification time was always acceptable As far as the efficacy and the role of the proposed analysis is concerned, it must be noticed that the framework actually helps the software architect in reasoning about the system The framework easies the definition of a right architecture and provides an executable prototype to perform analysis As we discussed in the Introduction, it is not easy to define the properties to be investigated on a given model In this experience, as one might expect, we saw that the process of properties elicitation and subsystems identification proceeds incrementally and it is driven by the confidence and system knowledge that the software architect acquires along the modelling process Properties were always described as set of sequence diagrams, characterizing either wanted or un-wanted behaviors and then automatically translated in LTL formula In the same way, output traces were offered to the software architect always as sequence diagrams This allowed a rather straightforward integration with the standard software development process and allowed early identification of critical sub-systems behaviors whose design and implementation should be carefully carried out Summarizing the study carried out clearly indicated future work directions: 1 the CHARMY tool should be integrated with existing case environment based on UML to eliminate input costs; 2 scenarios are a good way to express properties at the architectural level A comprehensive set of interpretation options has to be provided and their translation to LTL formula has to be completely hidden from the user Acknowledgments The authors would like to acknowledge the Italian MIUR national project SAHARA that partly supported this work References 1 Darwin, an Architectural Description Language In Web site: <http://wwwdsedocicacuk/research/darwin/ darwinhtml> 2 Rational Corporation UML Resource Center UML documentation, version13 On-line at: <http://www rationalcom/uml/indexjtmpl> 3 R Allen and D Garlan A Formal Basis for Architectural Connection In ACM Transactions on Software Engineering and Methodology, pages 6(3):213 249, July 1997 4 L Bass, P Clements, and R Kazman Software Architecture in Practice SEI Series, Addison-Wesley 1998 5 A Bertolino, P Inverardi, and H Muccini An Explorative Journey from Architectural Tests Definition downto Code Tests Execution In ACM Proceedings, ICSE2001, June 2001

6 R community RFC On-line at: <http://communityroxencom/developers/ idocs/rfc/rfc1901html> 7 D Compare, P Inverardi, and A Wolf Uncovering Architectural Mismatch in Dynamic Behavior In Science of Computer Programming, pages 33(2):101 131, February 1999 8 D Giannakopoulou, J Kramer, and S Cheung Analysing the behaviour of distributed systems using tracta In Automated Software Engineering, special issue on Automated Analysis of Software, pages 6(1):7 35, January 1999 9 C Hofmeister, R Nord, and D Soni Applied Software Architecture Addison Wesley, 1999 10 J G Holzmann The logic of bugs In Proc Foundations of Software Engineering (SIGSOFT 2002/FSE-10), 2002 11 P Inverardi, F Mancinelli, H Muccini, and P Pelliccione An Experience in Architectural Extensions: Active Objects in J2EE In Proc FIDJI 2002 International Workshop on scientific engineering of Distributed Java applications To appear on LNCS, November 28-29, 2002 12 P Inverardi, H Muccini, and P Pelliccione Automated Check of Architectural Models Consistency using SPIN In the Automated Software Engineering Conference Proceedings (ASE 2001) San Diego, California, November 2001 13 P Inverardi, H Muccini, and P Pelliccione Checking Consistency Between Architectural Models Using SPIN In Proc the First Int Workshop From Software Requirements to Architectures (STRAW 01), year 2001 14 P Inverardi and A Wolf Formal Specifications and Analysis of Software Architectures Using the Chemical Abstract Machine Model In IEEE Transactions on Software Engineering, pages 21(4):100 114, April 1995 15 N Kaveh and W Emmerich Deadlock Detection in Distributed Object Systems In Proc of the Joint 7th European Software Engineering Conference and 9th ACM SIGSOFT Int Symposium on Foundations of Software Engineering, Vienna, Austria LNCS Springer Verlag, 2001 16 P Kruchten The 4+1 View Model of Architecture pages pp42 50 IEEE Software, 12(6) November 1995 17 J Magee and J Kramer Concurrency: State models & java programs Wiley publisher, April 1999 18 J Magee, J Kramer, and D Giannakopoulou Behaviour analysis of software architectures In Proc First Working IFIP Conference on Software Architecture (WICSA1), San Antonio, Texas, 1999 19 C Pons, R Giandini, and G Baum Dependency Relations Between Models in the Unified Process In Proc IWSSD 2000, November 2000 20 G Reggio, M Cerioli, and E Astesiano Towards a Rigorous Semantics of UML Supporting its Multiview Approach In Proc FASE 2001, LNCS n 2029 Berlin, Springer Verlag, 2001 21 D Richardson and A Wolf Software Testing at the Architectural Level In Second International Software Architecture Workshop (ISAW-2), October 1996 22 A Sebastiani Specifica e verifica formale dell architettura di gestione di un sistema integrato reale per le telecomunicazioni Tesi di Laurea in Informatica, Universitá di L Aquila, Facoltá di Scienze Matematiche Fisiche e Naturali, October 2002 23 R L Smith, G S Avrunin, L A Clarke, and L J Osterweil PROPEL: An Approach Supporting Property Elucidation In Proc 24th International Conference on Software Engineering (ICSE2002), pages pp11 21, May 19-25 2002 24 SPIN Home page on line at: <http://cmbell-labscom/cm/cs/what/spin/ indexhtml>