SMock A Test Platform for the Evaluation of Monitoring Tools

SMock A Test Platform for the Evaluation of Monitoring Tools User Manual Ruth Mizzi Faculty of ICT University of Malta June 20, 2013

Contents 1 Introduction 3 1.1 The Architecture and Design of SMock................ 4 2 Preparing an SMock Specification 6 2.1 A domain specific language in Python................. 7 2.1.1 Action.............................. 7 2.1.2 Action Configuration...................... 7 2.1.3 Anonymous Action....................... 8 2.1.4 Sequential Composition.................... 9 2.1.5 Parallel Composition...................... 10 2.1.6 Choice.............................. 10 2.1.7 Sequential ForEach....................... 11 2.1.8 Parallel ForEach........................ 11 2.1.9 The Document Management System Specification...... 12 2.2 Further Example scenarios....................... 13 2.2.1 Image Processing Example................... 13 3 Using SMock 17 2

Chapter 1 Introduction Runtime verification is a technique which introduces added assurance to software systems by analysing the generated events and checking them against formally specified behaviour. Most times this process involves the execution of verification code on the same resources being used by the monitored system, while the system itself is running. In order to ensure widespread adoption of runtime verification, a challenge that needs to be addressed by most runtime verification tools is therefore that of ensuring that the overheads they impose on the execution of the system being analysed are kept to a minimum. SMock aims to provide a flexible and customisable testing framework, for runtime verification tools, that is capable of providing quantitative measures on the overall effect of the runtime monitoring process on an executing system. SMock is a configurable framework which may be used to mock transaction systems under different loads and usage patterns. The key feature of the framework is the ease with which one can define different types of behaviour and farm out concurrent instances of such behaviours using user-specified distributions. As an illustrative example, consider the simulation of a document management system. In order to mock the typical architecture and usage of such a system, the following scenario could be specified: Example 1 A document management system commonly allows users to log into the system and start their own session, upload documents, edit these documents and so forth. In addition, such systems will usually have special users who are assigned the role of administrators and who have special permissions and responsibilities, such as that of managing the application users. Such systems would normally be highly concurrent and non-deterministic, with multiple user sessions being active at the same time, executing completely different actions. 3

We could describe one sample scenario, stating that, during an execution of the system: 30 users will log into the system over a span of 5 minutes of these, 5 users will be administrators while the rest will be normal users normal users will be carrying out different activities on the system a number will simply log in and log out again after a few seconds some might log in, upload a document and log out again some might log in, carry out activities and never log out etc. administrator users will also have a range of different transactions they will carry out on the system SMock allows straightforward scripting of case studies such as the one just described, giving control over types of transactions 1, timing issues, load buildup, usage patterns, etc., which can be used to benchmark the correctness and performance of different runtime verification tools. 1.1 The Architecture and Design of SMock SMock has been built in such a manner so as to enable the easy generation of families of transaction systems with two levels of configurability: (i) the forms and variations of the underlying transactions; and (ii) the scenarios depicting transaction firing as distributed over time. Figure 1.1 illustrates how the SMock architecture enables us to achieve this goal. SMock takes a script describing the various mock system architecture and execution options and generates a Java mock system behaving in the prescribed manner, generating events which can be captured by monitoring tools. Such a system is then typically monitored using a specification script in a runtime monitoring tool, enabling the measurement of overheads induced by the monitors in such a setting. Since the mock system may include parts which are randomly generated (e.g. the script may specify the launching of between 100 and 120 transactions launched in the first 10 seconds), its execution also outputs information (to the Replay Configuration file) to allow exact replayability. This enables, for instance, comparison of the performance of the 1 We use transaction to refer to an independent highly replicated unit of processing. 4

different versions of a runtime verification tool, or its performance against that of other tools, or as a regression test following a bug fix. SMock specification Profiling Report SMock GENERATED SYSTEM RUNTIME MONITOR RV specification Replay Configuration Figure 1.1: Proposed Test Platform Architecture The typical scenario in which the test platform will be used would be a sequence of actions which would involve: 1. Generating a mock transaction system through a user-defined specification 2. Executing the mock transaction system without any runtime monitoring 3. Saving the profiling reports output by the mock transaction system 4. Starting the runtime verification related testing: (a) Using an arbitrary runtime verification tool to instrument the mock transaction system to include runtime monitoring (b) Executing the mock transaction system, this time with included runtime monitoring functionality (c) Saving the profiling reports output by the mock transaction system 5. Comparing the output reports issued in steps (3) and (4c) to determine the effects of the runtime monitoring framework on the transaction system 5

Chapter 2 Preparing an SMock Specification One of the strengths of SMock is that it allows users to script scenarios to be used for testing. An execution of the test system is referred to as process P. Basic processes are individual actions which are characterised by three parameters their duration, memory consumption and CPU usage. These configurable parameters can be exact values or probability distributions, to enable the mock system to behave within a range of possible behaviours. Conf is therefore defined as: Conf ::=< duration, mem, cpu > Actions, which can now be represented using A Conf, can be named if one would want to be able to monitor their moment of entry and exit, but can also be left unnamed. Anonymous actions thus do not result in the triggering of system events during program execution. Processes can be combined using the following operators: (i) sequential composition; (ii) parallel composition of processes; and (iii) probability weighted choice between processes. The combinators come in a binary form and also generalised versions which allow the combination of multiple processes together. The following grammar describes these possible ways of composing actions into processes. P ::= A Conf P ; P P P P n P n ; x [...] do P (x) x [...] do P (x) A detailed, formal definition of this specification language is given in the SMock Technical Specification document??. This chapter explains the SMock specification language syntax and the way the various constructs can be combined to create 6

a complete script. 2.1 A domain specific language in Python SMock scripts are written in a small domain specific language which provides a number of constructors to enable the description of processes, as defined above, and how they are to be launched. The scripting language has been implemented as an embedded language [?] in Python. As an example, one of the simplest scripts that can be set up defines a program whose execution consists of only one call to an action named login, as shown here. Action( login, Delay(F ixed(3)), MemUsage(F ixed(2)), CP UUsage(Normal(50, 2))) 2.1.1 Action An action is used to describe a single operation being carried out on the system. Concretely, an action is translated into a system method call, with the method making use of time and system resources. This resource usage must therefore also be specified through the action configuration parameter. An action is defined by : an action name the action configuration settings(defined below) Action (actionname, ActionConfig ) 2.1.2 Action Configuration An action s resource usage is modelled through its configuration settings which specify (i) the time that elapses while the action is being processed (in seconds); (ii) memory resources utilised by the action s execution (in Kb) ; and (iii) percentage of the total CPU power utilised during execution of the action. Delay, MemUsage, CPUUsage) Memory usage, CPU utilisation and time delay settings are all randomized according to a specified distribution. The signatures of these constructs therefore all follow the same pattern of expecting one Distribution argument. The current implementation supports the following distributions: 7

Table 2.1: Action Configuration Parameters Language Signature Argument Description Construct Delay Delay(Distribution) Distribution - structure specifying distribution from which a random value can be generated CPUUsage CPUUsage(Distribution) Distribution - structure specifying distribution from which a random value can be generated MemUsage MemUsage(Distribution) Distribution - structure specifying distribution from which Fixed a random value can be generated (extends Distribution) Fixed(val) val - a fixed value Normal Normal(mean, sd) mean - mean of the distribution (extends Distribution) sd - standard deviation of the distribution Normal distribution Exponential distribution to be implemented Uniform distribution to be implemented Alternatively, fixed values can be specified. Table 2.1 summarizes usage of the mentioned constructs. The simple script proposed at the beginning of this chapter can therefore be explained as follows: The script defines the execution of one action, named login. The execution of login will have a fixed duration of 3ms. 2K of available memory will be utilised and the execution will require approximately 50% of the total CPU power availabled (the exact CPU usage will be a randomised value obtained from a normal distribution of mean 50 and standard deviation of 2). 2.1.3 Anonymous Action to be implemented Apart from the named actions just described, a process may also incorporate one or more anonymous actions. The definition of an anonymous action, similarly to that of the standard Action, is used to specify resource usage during a defined time interval. The difference between an unnamed (or anonymous) action and the standard action defined previously, is that execution of these anonymous actions does not result in the triggering of events. The definition of an anonymous execution would therefore be of the form: AnonAction(Delay(F ixed(3)), M emu sage(f ixed(2)), CP U U sage(n ormal(50, 2))) 8

or, more simply, Action(, Delay(F ixed(3)), MemUsage(F ixed(2)), CP UUsage(Normal(50, 2))) 2.1.4 Sequential Composition A realistic description of the run of a process will usually involve the sequential execution of multiple actions. A process can therefore be defined as a sequence of named, and possibly anonymous, actions, using the sequence construct Seq. As an illustrative example, we shall build a script to model the behavior of a mock document management system. Common functions made available by such systems are those of allowing users to log into their own session, upload documents, edit these documents and so forth. The SMock specification describes the sequence of actions that occurs on the system when it is being utilised. A simplified run of the system could therefore be modelled using the script below which specifies a sequence of three method calls login, upload and logout, each of which are configured to simulate different durations, memory and CPU utilisation. l o g i n A c t i o n = Action ( l o g i n, Delay ( Normal ( 3, 1 ) ), MemUsage( Normal ( 3 0, 5 ) ), CPUUsage( Fixed ( 2 0 ) ) ) ) uploadaction = Action ( upload, Delay ( Normal ( 5, 1 ) ), MemUsage ( 5 0 0, 2 5 ), CPUUsage( Fixed ( 3 5 ) ) ) logoutaction = Action ( logout, Delay ( Fixed ( 2 ) ), MemUsage( Normal ( 3 0, 5 ) ), CPUUsage( Fixed ( 1 0 ) ) ) ) program = Seq ( loginaction, uploadaction, logoutaction ) 9

2.1.5 Parallel Composition Real world scenarios propose complex systems where execution is multi-threaded and multiple actions are occurring concurrently on the executing system. This scenario is supported through the introduction of the Par construct that is used to define parallel process blocks. Assuming that initaction, uploadaction, saveaction, displayaction and logoutaction have already been defined within the same specification script, the following program listing describes a process made up of a login action, followed by an upload action. Execution then forks into two parallel processes where one thread carries out a save action while the other thread simulates the display function. The process terminates with the logout action. program = Seq ( loginaction, uploadaction, Par ( saveaction, d i s p l a y A c t i o n ), logoutaction ) Note that the implementation adopts the solution of synchronized joining, meaning that the final logoutaction is not executed until both forked threads have completed execution. 2.1.6 Choice The choice construct introduces the concept of alternative paths that can be taken during an execution. This can be specified using Choice(*args), where a Choice construct can have an arbitrary number of ChoiceItems as parameter. Each ChoiceItem is assigned a probability defining the probability with which that particular path would be selected. As an example, the following script describes an execution which can take one of two alternative paths: either a sequence of loginaction, uploadaction, saveaction and logoutaction, or else the sequence of loginaction, displayaction and logoutaction. program = Seq ( loginaction, Choice ( ChoiceItem ( 0. 3, Seq ( uploadaction, saveaction ) ), ChoiceItem ( 0. 7, d i s p l a y A c t i o n ) ), logoutaction ) finally i can understand why you were using a probability for each item... do we check that all probabilities add up to 1? should we maybe normalize the values? eg: 1,abc; 2,def means that def will occur twice as frequently as abc /C we have to take decision on the above... /R 10

So far, our simulation of the use of a document management system has been restricted to modelling the experience of one user. The simulated system s behaviour would be more realistic if it models the fact that, commonly, multiple users are logged into the system at any one time. The script may be extended to model multiple users and, in turn, the concurrent activity of these users using the constructs available for replicating processes, as described in the next sections. 2.1.7 Sequential ForEach The sequential foreach (loop) construct is used to indicate that a process segment is repeated sequentially over a number of iterations. In the following example we model 10 users which, in sequence, will each carry out a login, upload and logout action, with user at index 1 being processed first and user at index 10 last. program = ForAllSeq ( user, 10, Seq ( loginaction, uploadaction, logoutaction ) ) The signature of the ForAllSeq construct is therefore defined as: ForAllSeq (loopindex, iteration_count, program_construct) where: loopindex variable used as iteration index. iteration_count the number of times the program_construct will be repeated. program_construct defines one or more actions that will be carried out. Could be any of the constructs defined in this section. While the above models multiple user activity on the mock document management system, this activity is still sequential and fails to recognize the fact that normally multiple users are logged into the system concurrently. This concurrency can be modelled using the parallel foreach construct. 2.1.8 Parallel ForEach Contrary to the ForAllSeq() construct defined above, using a parallel foreach indicates that processing will occur in parallel. The previous script could therefore be improved by using: 11

program = ForAllPar ( user, 10, Seq ( loginaction, uploadaction, logoutaction ) ) which now indicates that all of the 10 users will be carry out their activity concurrently. 2.1.9 The Document Management System Specification The SMock language constructs introduced can be effectively combined together to specify a realistic model of the use of a mock system. The specification given below is a complete version of the document management system specification and models a scenario where thirty-five users are concurrently active on the system. In order to simulate the fact that, normally, users will not be carrying out their activities in synchrony, an anonymous action has been utilised at the beginning of each user s activity sequence. The definition of the anonymous action specifies that a user s activity will not commence until 2 user milliseconds have elapsed, where user refers to the user s index defined in the parallel foreach construct. l o g i n A c t i o n = Action ( l o g i n, Delay ( Normal ( 3, 1 ) ), MemUsage( Normal ( 3 0, 5 ) ), CPUUsage( Fixed ( 2 0 ) ) ) ) uploadaction = Action ( upload, Delay ( Normal ( 5, 1 ) ), MemUsage ( 5 0 0, 2 5 ), CPUUsage( Fixed ( 3 5 ) ) ) logoutaction = Action ( logout, Delay ( Fixed ( 2 ) ), MemUsage( Normal ( 3 0, 5 ) ), CPUUsage( Fixed ( 1 0 ) ) ) ) program = ForAllPar ( user, 35, Seq ( AnonAction ( Delay ( Fixed ( 2 user ) ), MemUsage( Fixed ( 0 ) ), CPUUsage( Fixed ( 0 ) ) ), 12

Choice ( ChoiceItem ( 0. 5, Seq ( loginaction, uploadaction, logoutaction ) ), ChoiceItem ( 0. 5, Seq ( loginaction, logoutaction ) ) ) ) ) 2.2 Further Example scenarios The following section lists a number of example program specifications using the language just defined. In particular, the examples focus on highlighting the advantages that can be obtained from the implementation of SMock as a Python embedded language. 2.2.1 Image Processing Example In this example we propose the simulation of a dummy system that handles concurrent processing of multiple images. We assume that the function that this system can carry out is that of image compression through a fictitious algorithm which segments the image, compresses the segments and then aggregates the result of each segment s compression to produce the final image. Compression of each segment occurs independently from that of the other segments therefore meaning that compression of segments can be carried out in parallel. The final merge will require compression of all segments making up the image to be completed first. We could describe one sample system execution as follows: 5 images are to be processed All 5 images will be processed in parallel openimage = Action ( open, Delay ( Normal ( 3, 1 ) ), MemUsage( Normal ( 3, 1 ) ), CPUUsage( Fixed ( 5 ) ) ) ) segmentimage = Action ( segment, Delay ( Normal ( 5, 1 ) ), MemUsage ( 5 0 0, 2 5 ), 13

CPUUsage( Fixed ( 3 5 ) ) ) processsegment = Action ( process, Delay ( Fixed ( 5 ) ), MemUsage( Normal ( 4 0, 5 ) ), CPUUsage( Normal ( 1 5, 3 ) ) ) mergeresults = Action ( merge, Delay ( Normal ( 5, 3 ) ), MemUsage( Normal ( 4 0, 5 ) ), CPUUsage( Normal ( 1 5, 3 ) ) ) program = ForAllPar ( image, 5, Seq ( openimage, segmentimage, ForAllPar ( segment, 1 0, processsegment ), mergeresults ) ) While the above is a straightforward example of a specification using the SMock language constructs, other scenarios could possibly impose more complex requirements. As an example, in our image processing example, we could have specified that we would like the segmentimage action to be modelled as a number of repeated action calls. Such a requirement can easily be met through the use of existing Python language constructs within the SMock embedded language. The script belows shows how standard Python function definitions are used to defined a new function segmentimage that, given as a parameter an indication of the required repetitions, will take care of instantiating a new segmentimageaction for the required number of times. def segmentimage ( r epeat ) : a c t i o n L i s t = [ ] f o r x in xrange ( r epeat ) : a c t i o n L i s t. append ( segmentimageaction ) r eturn Seq ( a c t i o n L i s t ) d e f segmentimageaction : r eturn Action ( segment, Delay ( Normal ( 5, 1 ) ), MemUsage ( 5 0 0, 2 5 ), CPUUsage( Fixed ( 3 5 ) ) ) 14

...... program = ForAllPar ( image, 5, Seq ( openimage, segmentimage ( 3 ), ForAllPar ( segment, 1 0, processsegment ), mergeresults ) ) We could extend this example even further to specify that the time taken to process the processsegment image is a function of the segment s index. Such a requirement is met in the extended script shown below. Contrary to the previous example, the segmentindex parameter used in this case, is a parameter which will be evaluated dynamically once the generated mock system is running and a randomized value for the action s duration is required to be calculated. def processsegment ( segmentindex ) : r eturn Action ( process, Delay ( ( Normal ( segmentindex, 1 ) ) ), MemUsage( Normal ( 4 0, 5 ) ), CPUUsage( Normal ( 1 5, 3 ) ) ) def segmentimage ( r epeat ) : a c t i o n L i s t = [ ] f o r x in xrange ( r epeat ) : a c t i o n L i s t. append ( segmentimageaction ) r eturn Seq ( a c t i o n L i s t ) d e f segmentimageaction : r eturn Action ( segment, Delay ( Normal ( 5, 1 ) ), MemUsage ( 5 0 0, 2 5 ), CPUUsage( Fixed ( 3 5 ) ) )...... program = ForAllPar ( image, 5, Seq ( openimage, segmentimage ( 3 ), ForAllPar ( segment, 1 0, processsegment ( segment ) ), 15

mergeresults ) ) 16

Chapter 3 Using SMock The first implementation of SMock is directed towards the generation of mock systems written in Java. This decision is mainly driven by the large amount of runtime monitoring tools that exist for the Java language. Such tools can only be tested in conjunction with SMock systems if these are Java systems. In the near future we propose to extend SMock to allow generation of mock systems in other languages. More information on how SMock can be extended for multi-language support is given in the SMock Technical Specification [?]. The SMock domain specific language is implemented in the Python library SMockLang.py. Importing this library in a Python script gives access to all the necessary language constructs defined in Chapter 2. Additionally, the SMockLang library also provides a central helper method, createjavatestprogram(script_expression, dir), which receives as parameter a script_expression and internally parses this expression and creates the resultant mock system code in dir. Generating a Java program using SMock will therefore involve the preparation and execution of a Python script, as shown in Program 3.0.1 In ➀ all commands defined in the SMockLang library are imported. ➁ specifies a simple mock system which is passed as parameter to createjavatestprogram in ➂. Running this Python script will automatically result in the generation of Java code that, when compiled, forms the complete mock transaction system. some kind of conclusion would be in order here /C 17

Program 3.0.1 Using Python for SMock system generation ➀ from RVTestScriptLang import * delay = Delay(Fixed(5)) cpuusage = CPUUsage(Normal(50, 2)) memusage = MemUsage(Fixed(2)) login = Action( login, ActionConfig(delay, cpuusage, memusage)) browse = Action( browse, ActionConfig(delay, cpuusage, memusage)) logout = Action( logout, ActionConfig(delay, cpuusage, memusage)) ➁ testprog = Seq(login, browse, logout) ➂ createjavatestprogram(testprog, c:\\ ) 18