Focused Diagnosis for Failing Software Tests

Transcription

1 Focused Diagnosis for Failing Software Tests Birgit Hofer, Seema Jehan, Ingo Pill and Franz Wotawa Graz University of Technology, Austria The final publication is available at Springer via Abstract. Ranging from firmware to cloud services, software is an essential part of almost any imaginable system, or at the very least assists us in their design or maintenance. The sheer complexity and sophisticated concepts of today s software products thus demand for solutions that assist us in assuring their quality. We aim to contribute in this direction by proposing a fault localization approach for failing test cases that draws on model-based diagnosis techniques from the AI community and focuses the search on dynamic executions. With this focus, we offer the scalability needed to consider also designs like service oriented architectures (SOAs). Furthermore, we opt for a flexible approach that allows a user to refine the reasoning by annotating our basic structure of a control flow graph with further information, e.g., for black box components. First experiments with standard software examples, as well as examples taken from the SOA domain show promising results. Keywords: Model-based Diagnosis, Software Debugging, Constraint Satisfaction Problem, Control Flow Graph 1 Introduction Software helps to meet challenges and offers flexible and cost-effective solutions - ranging from firmware that capitalizes a hardware s resources, via developer tools, to cloud solutions for complex, scalable demands. Ensuring a complex software product s quality is, however, a cumbersome task. For potentially huge designs, like service oriented architectures (SOAs) it becomes even more demanding (see [8]). Aside controllability and observability issues, we might have only partial knowledge about some system parts, e.g. invoked web services or 3rd party libraries. For SOAs we envisioned in [14] a corresponding, combined diagnosis and testing workflow that targets their BPEL processes (i.e., the descriptions of their business logic). Our first sketch at tackling diagnostic issues appeared at the DX Workshop [4]. For this paper, we rid ourselves of this focus and propose a diagnosis approach for failing test cases that is not aimed at some particular software architecture. Our underlying reasoning model is that of a software s control flow graph (CFG) that can be easily annotated with further knowledge about individual parts (like black boxes). We draw on model-based diagnosis techniques [6, 13] for localizing possible root cause variants explaining a failed test case. Most model-based diagnosis

2 2 approaches use a model of the correct system for their reasoning. However, we debug the faulty system under scrutiny (that is, in this case our knowledge about it as aggregated in the CFG). Furthermore, while usually the system s entire model is considered for such a diagnostic endeavor, similar to dynamic slicing techniques [7, 17], we focus our search to the scope of individual executions, i.e., those CFG parts executed by the given test case. In Section 3, we show how to implement such an approach. Similar to Wotawa et al. [16], we compute the diagnoses directly from a CFG s constraint representation, isolating sets of constraints possibly responsible for the issue at hand. Our first experiments as reported in Section 4 showed promising results for programs taken from the software landscape and service-oriented architectures. For illustration purposes, we use the computation of a circle s or square s area and circumference. For the input {type=circle,length=2}, the parameter length defines the circle s diameter, and the program s output is {area=3.14, circumference=3.14}. Obviously incorrect, this is due to a bug in line 6 s.t. the circumference is computed as radius pi(= 3.14) instead of 2 radius π(= 6.28). Example 1. Compute area and circumference of a circle or a square. 1 g e o m e t r i c F e a t u r e s ( type, l e n g t h ){ 2 p i = ; 3 i f ( type== c i r c l e ){ 4 r a d i u s = l e n g t h / 2 ; 5 area = r a d i u s r a d i u s p i ; 6 c i r c u m f e r e n c e = r a d i u s p i ; //FAULT 7 e l s e 8 area = l e n g t h l e n g t h ; 9 c i r c u m f e r e n c e = 4 l e n g t h ; 10 e n d i f 11 } 2 Preliminaries The central model for our reasoning is the following generic control flow graph. Please note that we assume a program to be deterministic and currently do not consider parallel computations but sequential programs only. Definition 1. A Control Flow Graph G is defined by a tuple G = {V, E, v 0, F, S, γ C (v V ), γ A (v V )}, where V is a finite set of vertices representing statements, E V V is a finite set of directed edges connecting the statements (edge e = (v 1, v 2 ) E connects v 1 to v 2 ), v 0 V is the start vertex, F V is the set of leaf vertices (with no outgoing edges), S V is the set of branching vertices (with more than one outgoing edge), and the functions γ C (v) and γ A (v) map vertices v V to a statement s conditions and assignments respectively. The conversion of a sequential program into a CFG is straightforward, where further information can be easily added to γ C (v) and γ A (v). With G capturing the control flow structure of a program, a path π in G defines a valid scenario.

3 3 Definition 2. A finite Path π of length n in a CFG G = {V, E, v 0, F, S, γ C, γ T } is a finite sequence π = π 1 π 2...π n such that we have that (1) for any 0 < i n, π i is in V, (2) π 1 = v 0, (3) for any 0 < i < n, the edge e = (π i, π i+1 ) is in E, and (4) π n F. The length of some path π is denoted by π, where we use f (π) to refer to the last vertex in sequence π. As we reason about finite computations only, per definition a path is finite. For such a path π, we define the set of Path Constraints C(π) = (0<i n) γ C(π i ), where variables are replaced by indexed variables in order to implement a static single assignment form. A path is feasible if its corresponding path constraints are satisfiable. In the path constraints C(π) we collect our knowledge about π for reasoning. The static single assignment form (SSA) [1] implemented for C(π) means that we use indexed variables s.t. whenever a variable is assigned a value, the index is incremented for further referrals along the path. This concept of temporal variable instances clocked by assignments ensures that every variable along a path is defined once only, but can be referenced as often as needed. For our example, C(π) would be {π 0 == 3.14, (type 0 == circle ) == true, radius 0 == length 0 /2, area 0 == radius 0 radius 0 pi 0, circumference 0 == radius 0 pi 0 }. For our focused reasoning, we establish a connection between a control flow graph G and an actual program execution Π via our definition of a trace. Definition 3. The trace T in G for a program execution Π is defined as the path in G allowed for the input part of Π. Assuming a program to be deterministic and the inputs to be complete, the input values define a single specific path to be taken by the program. For our example computation, this means taking the then-branch. For our target scenario of a failing test case, we extract the relevant inputs from the very test case itself. Definition 4. A test case is a tuple τ = (I, O) where I (sometimes we will write also I(τ)) is the set of value assignments to input variables, and O is the set of expected values for the output variables. A test case fails iff the observed output O deviates from O, s.t. O O. Our diagnostic reasoning for localizing the fault(s) implements the basic concept of consistency-oriented model-based diagnosis as defined in [6, 13, 3]: Given a system s set of components COMP, a system description SD defining the correct behavior AB(c i ) NominalBehavior(c i ) of components c i COMP, and observations OBS about the system s behavior, the system is considered to be faulty iff SD OBS { AB(c i ) c i COMP} is unsatisfiable. The predicates AB(c i ) represent the health state of components c i : If AB(c i ) is false, then c i behaves as expected. Otherwise c i is faulty and we assume to know nothing about it. While a minterm in the health predicates defines a specific system state, a diagnosis is a subset-minimal set of faulty components that explains the observed faulty behavior OBS. COMP is the set of individual constraints in C(π) and SD is a correspondingly tailored constraint encoding of C(π). Definition 5. A diagnosis for (SD, COMP, OBS) is a subset-minimal set COMP such that SD OBS { AB(c i ) c i COMP \ } is satisfiable.

4 4 3 Localizing the Root Causes for Failed Software Tests Based on a program s CFG, our aim was to isolate those program parts that could explain the deviation from expected behavior for the failed test case. With today s complex software designs, an important motivation was to focus the search for optimizing resources. In detail, and in contrast to a full temporal system model like the one we used for our work on LTL specification diagnosis [12], we thus focus on a specific path in the system s control structure, similar to the idea in [5]. This way, SD does not describe the program s entire CFG, but focuses on the trace T (a path) in the CFG that is exercised for the failed test case τ. In particular, we use the path constraints C(T ), and by encapsulating C(T ) s individual constraints c with corresponding health predicates AB(c), we aim at identifying those constraints c whose incorrectness could be responsible for T s deviation from expected behavior (as defined by the output part O of τ). Faults in branching conditions (or related variables) might cause a trace T to deviate from the correct path (in the repaired program). With our focused system model SD, this would mean that the corrected path as defined by a diagnosis, let us call it π corr, could leave SD at some branching vertex. For such a scenario, we have to (a) disable subsequent (in terms of temporal evolvement along T ) constraints in C(T ), and (b) make the user aware of this. This is motivated by the fact that SD does then describe π corr only up to this point, but afterwards describes an alternative route in the CFG. To address this, we divide the trace T into segments separated by the branching vertices, and introduce variables intrace i for all these segments. The variables values define whether an individual segment is part of π corr as defined by. Definition 6. A segmented path (π, S) in a CFG G is a tuple s.t. π is a path in G, and S is the set of branching vertices s i in π that divide the path into enumerated segments as follows. The branching vertices s i are numbered according to their distance from π 0, the enumeration starting with 1. The first segment, numbered 0, starts at π 0 and ends with s 1. Starting with segment 1, a segment i starts right after s i, and ends with either s i+1 or the path π s final vertex f(π). Our example computation s trace T has two segments. The first one contains the constraints {pi 0 == 3.14, (type 0 == circle ) == true}, the second one contains the constraints {radius 0 == length 0 /2, area 0 == radius 0 radius 0 pi 0, circumference 0 == radius 0 pi 0 }. Now let us formalize our specific diagnosis problem and introduce our corresponding technical implementation. Definition 7. A control flow graph diagnosis problem is a tuple (G, τ), where G is a CFG, and T is a trace from a failing test case τ. This describes the scenario where we have a program G that we find to be incorrect by observing unexpected behavior. For our automated reasoning, we define the following constraint encoding to be used with a corresponding solver. Definition 8. A constraint satisfaction encoding CSP(G, τ) for a control flow graph diagnosis problem (G, τ) is defined as follows:

5 5 1. let (T, S) be a segmented path for the trace T for τ in G 2. let the set of variables V contain all the variables in C(T ), as well as Boolean variables intrace i for all of (T, S) s segments as of Definition let C (T ) be the path constraints C(T ) altered s.t. for any individual constraint c C(T ) of segment i, we add a predicate AB c and do as follows: (a) if c is not a branching constraint from some s S, then c gets replaced by intrace i AB c c. (b) if c is the branching constraint from s i S, then c gets replaced by the following set of constraints: intrace i intrace i+1, AB c intrace i+1 and intrace i AB c (c intrace i+1 ) - with being the expected polarity of intrace i+1 when constraint c is satisfied. 4. then let CSP(G, τ) be the combined constraints of C (π) and τ as well as the constraint intrace 0. Example 2. The CSP for our example computation is as follows: intrace 0 AB 1 pi 0 == 3.14 intrace 0 intrace 1 AB 2 intrace 1 intrace 0 AB 2 (type 0 == circle ) intrace 1 intrace 1 AB 3 radius 0 == length 0 /2 intrace 1 AB 4 area 0 == radius 0 radius 0 pi 0 intrace 1 AB 5 circumference 0 == radius 0 pi 0 type 0 == circle length 0 == 2 area 0 == 3.14 circumference 0 == 6.28 intrace 0 We do not use a conflict-based diagnosis engine like the computation suggested by Reiter [13]. Similar to the idea presented in [16] for Java programs, we formulate a constraint satisfaction problem adopting the CSP as of Definition 8. The algorithm in Fig. 1 shows how we derive diagnoses directly via computing satisfying assignments that are limited in the amount of active health predicates. This is achieved via the constraint added in line 4 of the algorithm. A single query to the solver then returns all the solutions of size i, where a diagnosis (solution) is the set of active health predicates of a satisfying assignment. In the loop (lines 3 to 8), for each instance, we increase i (starting with one and stopping at a given bound) and save (in line 6) all the diagnoses of cardinality i. Furthermore, in line 7, we add for each new diagnosis a blocking clause to the constraint model, which is basically a logic OR over all negated health predicates in. This is needed to block all supersets of for satisfying assignments, so that the reported solutions are indeed subset-minimal (see Def. 5). After reaching the bound for i, we report all diagnoses. Deploying this algorithm on our running example s CSP as of Example 2 results in the computation of the two single fault diagnoses AB 2 and AB 5, where the latter catches the fault described in the introduction.

6 6 1: procedure GetDiagnoses(M, n) 2: Let DS be {} 3: for i = 1 to n{( do ) } π 4: M = M AB j == i j=1 5: D = Solve(M ) 6: DS = DS D 7: M = M ( j D ( AB k j ( AB k ))) 8: end for 9: return DS 10: end procedure Fig. 1. GetDiagnoses Algorithm Please note that newer developments such as [10], and our experiments in [11] showed that direct approaches such as the one we used for this paper can offer superior advantage compared to conflict-based computations. The setup used in this paper is referred to as Direct-MS CS in the comparison found in [11]. For practical purposes, we propose that the user should be made aware of the different segments being part (or being excluded) of the scenario described by a diagnosis. This is vital data when considering a diagnosis in practice, so that we suggest to report for a diagnosis also the corresponding minterm in the intrace variables, that is, the evaluation of all the corresponding intrace variables. Definition 9. An extended diagnosis for a control flow graph diagnosis problem (G, τ) as of Def. 7 is a tuple (, T S, INTRACE), where is a diagnosis (see Def. 5) in the predicates AB i for the CSP as of Def. 8 derived from (G, τ), T S refers to the segmented path/trace derived for CSP(G, τ), and INTRACE is the corresponding minterm in the intrace variables (cf Def. 8) describing which segments are in the path π corr depicted by. Considering our encoding, the restriction to deterministic programs is apparent in that we do not directly accommodate non-determinism in the program for scenarios where for one and the same input we could have different traces. However, for such scenarios one can apparently implement an exhaustive approach by considering all the options in a loop, and presenting the diagnoses grouped by the different choices. 4 First Empirical Experiments For our first experiments, we implemented a compiler that converts a SOAs business logic programs (BPEL processes) into our control flow graph format (see [4]), and considered also standard examples from the testing and debugging community. As constraint solver, we used MINION [2] that supports logic and arithmetic operations on Booleans and integers out-of-the-box. All the experiments were carried out on a MacBook Pro (Late 2011) with a 2.4 GHz Intel Core i5, 4 GB 1333 MHz DDR3, running OS X

7 7 For this paper, we report on our experience with six examples. Most of them come from the SOA domain. The Loan example is a typical SOA process which approves, delays, or rejects loan requests based on the amount requested and the client s history. The Body Mass Index (Bmi) example decides on given height and weight if the result should be underweight, healthy, overweight, obese, or very obese. The Triangle (Tri) example determines depending on the input the type of triangle e.g. equilateral, isosceles, or scalene. The Geo example is the running example discussed in this paper. Tcas is a SOA variant of the well-known traffic collision avoidance system used in many software engineering examples 1. The calculator (Calc) example demonstrates additions, subtractions, divisions and multiplications as well as control structures like if-then-else and while. For all the examples, we derived faulty versions containing a single or two faults each. The corresponding files can be downloaded from the following link 2. Table 1 shows our results when searching for single and double fault diagnoses. Depending on the injected fault and the specific scenario, there are not always single and double fault diagnoses. Column F indicates the number of faults injected for a specific sample. The total number of inputs and outputs to a program are given in column I/O. Column S indicates the number of statements in the diagnosis problem s execution trace. The number of constraints and variables derived for the trace are reported in the columns C and V. For single fault ( = 1) and double fault diagnoses ( = 2) we report in the corresponding columns on the number of solutions found, as well as the average time in milliseconds over three computations. The approach s diagnostic performance for the single fault scenarios is indicated in column R by means of the reduction in statements that have to be considered by using the following formula : Reduction = 1 single fault diagnoses executed statements In other words, this means how many statements out of the executed ones we could eliminate from consideration. For double fault scenarios, we need to establish a more appropriate metric and thus do not report corresponding numbers. Our approach allows for small computation times for the considered examples. While most times were in the range of mere milliseconds, for TCAS it took up to a few seconds to consider also double fault diagnoses. In most cases we also saw an attractive reduction in the number of statements that would have to be checked for correctness. On average, we achieved a reduction of % for our experiments. For most of the examples, the reduction ranged from 22 % to 50 %. For TCAS, the reduction got as high as 89.1 %. The specific structure of the Calc example did not allow for any reduction. Summarizing, we saw promising results for these standard examples, but further experiments are needed in order to determine the approach s attractiveness for larger programs. As discussed in the next section, first steps towards such 1 see also 2

8 8 Table 1. Results for programs with single and double faults. = 1 = 2 Program F I/O S C V R(%) # T avg(ms) # T avg(ms) Loan-V1 1 2/ Loan-V2 1 2/ Loan-V3 2 2/ Loan-V4 2 2/ Loan-V5 2 2/ Loan-V6 2 2/ Bmi-V1 1 2/ Bmi-V2 1 2/ Bmi-V3 1 2/ Bmi-V4 1 2/ Bmi-V5 2 2/ Bmi-V6 2 2/ Bmi-V7 2 2/ Bmi-V8 2 2/ Tri-V1 1 3/ Tri-V2 1 3/ Tri-V3 1 3/ Tri-V4 1 3/ Tri-V5 2 3/ Tri-V6 2 3/ Tri-V7 2 3/ Tri-V8 2 3/ Geo-V1 1 2/ Geo-V2 1 2/ Geo-V3 2 2/ Geo-V4 2 2/ Geo-V5 2 2/ Geo-V6 2 2/ Tcas-V1 1 12/ Tcas-V2 1 12/ Tcas-V3 1 12/ Tcas-V4 1 12/ Tcas-V5 1 12/ Tcas-V / Tcas-V / Calc-V1 1 3/ Calc-V2 1 3/ Calc-V3 1 3/ Calc-V4 2 3/ Calc-V5 2 3/ a study are the development of a modeling guide in order to support an efficient use of a solver, and we also need appropriate metrics for comparing the information delivered by our diagnoses with techniques like slicing.

9 9 5 Discussion and Future Work In this paper we propose a focused approach at the functional diagnosis of software programs for failing test cases. Implementing a consistency-oriented modelbased diagnosis approach, we compute corresponding diagnoses directly from a constraint satisfaction problem that encodes the program s control flow graph that can also be annotated with auxiliary data (e.g. for black box parts). Aiming to accommodate the sheer stunning complexity of today s programs, we focus our search on a single execution s trace in the control flow graph, instead of considering a full-fledged temporal model like an automaton. The diagnoses then tell which constraints involved in this trace could resolve the issue, or where a correct program might leave the trace. Mayer and Stumptner [9] give an overview of related work on model-based software debugging (MBSD). Wotawa et al. [16] suggest an approach that also relies on constraint solving. Our work differs in one major aspect: instead of the source code, we rely on a concrete program execution. Thus, our approach could not eliminate as many diagnoses as [16], but has lower computational complexity since non-executed branches are abstracted. Wotawa [15] presented a lightweight MBSD approach that combines dynamic slicing with hitting-set computation. This approach captures which components participated in the computation of an erroneous value. In contrast, our approach captures the values of variables computed in the single components. Another difference concerns the diagnosis computation: while [15] computes the diagnoses via a hitting-set algorithm, we obtain the diagnoses directly from the constraint solver. The approach presented in [5] also relies on a concrete execution instead of the whole program. However, our work differs from theirs in particular in the used encoding: our new approach explicitly considers the possibility of changing the execution trace due to a fault. First experiments with standard SOA-, testing-, and debugging examples showed that our focused approach allows for attractive runtimes. Future work will contain a more detailed experimental study, taking into account also the effects on the diagnostic performance of weakening and strengthening the auxiliary data. An important step for such a study will be to develop a modeling guideline that allows users to write models that are easily solvable and minimize consumed ressources (i.e. in terms of memory). While we currently focus on a weak fault model in that SD does not contain any information on faulty behavior, implementing strong fault models (encoding optional behavioral modes) is up to future work. In the context of reliability issues (and other non-functional properties of SOAs or cloud applications) moving from persistent faults to intermittent faults could be of interest. Extensions regarding concurrent and non-deterministic programs are further interesting topics. Additionally, a comparison of the approach with slicing techniques will be of interest. Acknowledgement The research leading to these results has received funding from the Austrian Science Fund (FWF) under project references P23313-N23 and P22959-N23.

10 10 References 1. Brandis, M.M., Mössenböck, H.: Single-pass generation of static assignment form for structured languages. ACM TOPLAS 16(6), (1994) 2. Gent, I.P., Jefferson, C., Miguel, I.: Minion: A fast scalable constraint solver. In: In: Proceedings of ECAI 2006, Riva del Garda. pp IOS Press (2006) 3. Greiner, R., Smith, B.A., Wilkerson, R.W.: A correction to the algorithm in Reiter s theory of diagnosis. Artificial Intelligence 41(1), (1989) 4. Hofer, B., Jehan, S., Pill, I., Wotawa, F.: Functional diagnosis of a SOA s BPEL processes. In: International Workshop on Principles of Diagnosis (DX) (2014) 5. Hofer, B., Wotawa, F.: Combining slicing and constraint solving for better debugging: The conbas approach. Adv. Software Engineering 2012 (2012) 6. de Kleer, J., Williams, B.C.: Diagnosing multiple faults. Artificial Intelligence 32(1), (1987) 7. Korel, B., Rilling, J.: Dynamic program slicing methods. Information & Software Technology 40(11-12), (1998) 8. Mayer, W., Friedrich, G., Stumptner, M.: On computing correct processes and repairs sing partial behavioral models. In: ECAI. Frontiers in Artificial Intelligence and Applications, vol. 242, pp IOS Press (2012) 9. Mayer, W., Stumptner, M.: Model-based debugging state of the art and future challenges. Electron. Notes Theor. Comput. Sci. 174(4), (May 2007), http: //dx.doi.org/ /j.entcs Metodi, A., Stern, R., Kalech, M., Codish, M.: Compiling model-based diagnosis to Boolean satisfaction. In: 26th AAAI Conference on Artificial Intelligence. pp (2012) 11. Nica, I., Pill, I., Quaritsch, T., Wotawa, F.: The route to success - a performance comparison of diagnosis algorithms. In: International Joint Conference on Artificial Intelligence. pp (2013) 12. Pill, I., Quaritsch, T.: Behavioral diagnosis of LTL specifications at operator level. In: International Conference on Artificial Intelligence. pp (2013) 13. Reiter, R.: A theory of diagnosis from first principles. Artif. Intelligence 32(1), (1987) 14. Wotawa, F., Schulz, M., Pill, I., Jehan, S., Leitner, P., Hummer, W., Schulte, S., Hoenisch, P., Dustdar, S.: Fifty shades of grey in SOA testing. In: 9th Workshop on Advances in Model Based Testing; 2013 IEEE Sixth International Conference on Software Testing, Verification and Validation Workshops (ICSTW). pp (2013) 15. Wotawa, F.: Fault localization based on dynamic slicing and hitting-set computation. In: The 10th International Conference on Quality Software (QSIC). IEEE (2010) 16. Wotawa, F., Nica, M., Moraru, I.: Automated debugging based on a constraint model of the program and a test case. The journal of logic and algebraic programming 81(4), (2012) 17. Zhang, X., He, H., Gupta, N., Gupta, R.: Experimental evaluation of using dynamic slices for fault localization. In: Sixth International Symposium on Automated & Analysis-Driven Debugging (AADEBUG). pp (2005)