SOFTWARE DEBUGGING WITH DYNAMIC INSTRUMENTATION AND TEST BASED KNOWLEDGE A Thesis Submitted to the Faculty of Purdue University by Hsin Pan In Partial Fulfillment of the Requirements for the Degree of Doctor of Philosohy August 1993
To My Family. ii
iii ACKNOWLEDGMENTS I would first like to exress my grateful acknowledgement to my major rofessors, Richard DeMillo and Eugene Safford, for their atience, suort, and friendshi. Professor DeMillo motivated me to study software testing and debugging, and who rovided many helful ideas in this research as well as the area of software engineering. The encouragement, valuable criticism, and exert advice on technical matters by my co-advisor, Eugene Safford, have sustained my rogress. In articular, I thank Professor Richard Liton of Princeton University for suggesting the idea of Critical Slicing. I thank my committee members Professors Aditya Mathur and Buster Dunsmore for their suggestions and discussion. I am also thankful to Stehen Chain, William Gorman, R. J. Martin, Vernon Rego, Janche Sang, Chonchanok Viravan, Michal Young, and many others with whom I had valuable discussion while working on this research. My ex-officemates, Hiralal Agrawal and Edward Krauser, deserve secial recognition for develoing the rototye debugging tool, SPYDER. Bellcore s agreement to make ATAC available for research use at Purdue is acknowledged. Finally, my sincere thanks go to my arents Chyang Luh Pan and Shu Shuan Lin, and my wife Shi Miin Liu, for their love, understanding, and suort. I am esecially grateful for my wife s efforts in roofreading art of this thesis and her advice on writing. This research was suorted, in art, by a grant from the Software Engineering Research Center at Purdue University, a National Science Foundation Industry/University Cooerative Research Center (NSF Grant ECD 8913133), and by NSF Grant CCR 8910306.
DISCARD THIS PAGE
iv TABLE OF CONTENTS Page LIST OF TABLES : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : vii LIST OF FIGURES : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : viii ABSTRACT : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : x 1. INTRODUCTION : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : 1 1.1 Problems in Locating Faults : : : : : : : : : : : : : : : : : : : : : : : : 1. Scoe of This Research : : : : : : : : : : : : : : : : : : : : : : : : : : 1.3 Contributions : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : 5 1.4 Organization of this Dissertation : : : : : : : : : : : : : : : : : : : : : : 5. RELATED WORK : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : 7.1 Traditional Techniques : : : : : : : : : : : : : : : : : : : : : : : : : : : 7. Algorithmic Aroaches : : : : : : : : : : : : : : : : : : : : : : : : : : 9.3 Knowledge Based Aroaches : : : : : : : : : : : : : : : : : : : : : : 10.4 Program Slicing Aroach : : : : : : : : : : : : : : : : : : : : : : : : : 1.5 Test Integrated Suort : : : : : : : : : : : : : : : : : : : : : : : : : : 14.6 Summary : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : 16 3. ANALYSIS OF A NEW DEBUGGING APPROACH : : : : : : : : : : : : : : 18 3.1 Background : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : 18 3. Heuristics vs. Deterministic Stes : : : : : : : : : : : : : : : : : : : : : 0 3.3 Exanded Dynamic Program Slicing : : : : : : : : : : : : : : : : : : : : 1 3.4 Generic Analysis of Testing Based Information : : : : : : : : : : : : : : 8 3.5 Summary : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : 31
v Page 4. HEURISTICS WITHOUT THE ASSISTANCE OF FURTHER TESTING IN- FORMATION : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : 33 4.1 Heuristics based on Relevant Test Cases : : : : : : : : : : : : : : : : : : 34 4. Further Analysis : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : 4 4.3 Summary : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : 45 5. HEURISTICS WITH THE ASSISTANCE OF MUTATION BASED TESTING 46 5.1 Critical Slicing : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : 50 5.1.1 Definitions : : : : : : : : : : : : : : : : : : : : : : : : : : : : : 50 5.1. Proerties of Critical Slicing : : : : : : : : : : : : : : : : : : : : 51 5. Debugging with Other Mutation Based Testing Information : : : : : : : 57 5..1 Methods for Analyzing the Information : : : : : : : : : : : : : : 57 5.. Statement Analysis MT 1 : : : : : : : : : : : : : : : : : : : 61 5..3 Domain Perturbation MT : : : : : : : : : : : : : : : : : : 65 5..4 Oerand Relacement on the Use art of a Statement MT 3 : 68 5..5 Oerand Relacement on the Define art of a Statement MT 4 7 5..6 Control Deendency Variation MT 5 : : : : : : : : : : : : : 76 5.3 Summary : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : 79 6. A PROTOTYPE IMPLEMENTATION AND EXPERIMENTATION : : : : : : 80 6.1 SPYDER: A Prototye Debugger : : : : : : : : : : : : : : : : : : : : : : 80 6.1.1 Screen of SPYDER : : : : : : : : : : : : : : : : : : : : : : : : : 8 6.1. Functions of SPYDER : : : : : : : : : : : : : : : : : : : : : : : 83 6.1.3 Imlementation Features : : : : : : : : : : : : : : : : : : : : : : 85 6. An Exeriment : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : 86 6..1 Evaluation Methods : : : : : : : : : : : : : : : : : : : : : : : : 87 6.. Tested Programs : : : : : : : : : : : : : : : : : : : : : : : : : : 90 6..3 Results of Heuristics without the Assistance of Further Testing Information : : : : : : : : : : : : : : : : : : : : : : : : : : : : 93 6..4 Results of Critical Slicing : : : : : : : : : : : : : : : : : : : : : 108 6.3 Summary : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : 114 7. CONCLUSIONS AND FUTURE WORK : : : : : : : : : : : : : : : : : : : : 118 7.1 Summary of Our Aroach : : : : : : : : : : : : : : : : : : : : : : : : 118 7. A New Debugging Paradigm : : : : : : : : : : : : : : : : : : : : : : : : 10 7.3 Limitation of the Paradigm : : : : : : : : : : : : : : : : : : : : : : : : : 11 7.4 Lessons Learned from the Prototye Imlementation : : : : : : : : : : : 11 7.5 Future Work : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : 1 7.5.1 Fault Guessing Heuristics : : : : : : : : : : : : : : : : : : : : : 1
vi Page 7.5. Exhaustive Exeriments : : : : : : : : : : : : : : : : : : : : : : 13 7.5.3 Failure Analysis : : : : : : : : : : : : : : : : : : : : : : : : : : 13 7.5.4 Large Scale Programs : : : : : : : : : : : : : : : : : : : : : : : 13 7.5.5 Other Alications : : : : : : : : : : : : : : : : : : : : : : : : : 14 BIBLIOGRAPHY : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : 15 APPENDICES Aendix A: Notation and Terminology : : : : : : : : : : : : : : : : : : : 136 Aendix B: A List of Heuristics Proosed in Chater 4 : : : : : : : : : : : 138 Aendix C: Programs Tested : : : : : : : : : : : : : : : : : : : : : : : : 139 Aendix D: Figures of Exerimental Results Presented by the Order of Programs Tested : : : : : : : : : : : : : : : : : : : : : : : : : : 141 Aendix E: Algorithms for Alying the Proosed Heuristics : : : : : : : : 145 VITA : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : 148
vii LIST OF TABLES Table Page 5.1 Mutant oerators used by MOTHRA for FORTRAN 77 : : : : : : : : : : : : 47 6.1 Software comonents of SPYDER : : : : : : : : : : : : : : : : : : : : : : : 8 6. Tested rograms : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : 90 6.3 ATAC s measurement of test case adequacy : : : : : : : : : : : : : : : : : : 9 6.4 Coverage analysis for Heuristics 1 4, 6 10, and 13 14 roosed in Chater 4 94 6.5 Coverage analysis for Heuristic 16 : : : : : : : : : : : : : : : : : : : : : : 95 6.6 Coverage analysis for heuristics under H (success set) without threshold requirements roosed in Chater 4 : : : : : : : : : : : : : : : : : : : : : : 96 6.7 Coverage Analysis for Critical Slicing : : : : : : : : : : : : : : : : : : : : 111 Aendix Table
viii LIST OF FIGURES Figure Page 1.1 A new debugging scenario : : : : : : : : : : : : : : : : : : : : : : : : : : 3 3.1 An examle for the tye of otential effect (PE) : : : : : : : : : : : : : : : 3 3. Relationshis among DPS, EDPS, and SPS : : : : : : : : : : : : : : : : : : 7 4.1 The family tree of roosed heuristics in Chater 4 : : : : : : : : : : : : : : 34 5.1 An examle derived from Figure 3.4 with indication of a critical slice : : : : 5 5. Relationshis between CS, DPS, EDPS, and SPS for rograms without array and ointer variables : : : : : : : : : : : : : : : : : : : : : : : : : : : : : 55 5.3 Examles of the different execution aths between the scoe of Dyn(P; v; $; t) and Dyn(M; v; $; t) after line S : : : : : : : : : : : : : : : : : : : : : : : : 63 6.1 X Window screen dum from SPYDER during a software debugging session : 81 6. Effectiveness comarison of heuristics, art I : : : : : : : : : : : : : : : : : 99 6.3 Effectiveness comarison of heuristics, art II : : : : : : : : : : : : : : : : 100 6.4 Effectiveness comarison of heuristics, art III : : : : : : : : : : : : : : : : 10 6.5 Effectiveness comarison of Heuristic 15 : : : : : : : : : : : : : : : : : : : 104 6.6 Effectiveness comarison of Heuristic 16 : : : : : : : : : : : : : : : : : : : 105 6.7 Effectiveness comarison between rank and general threshold for Heuristics 3, 4, 7, and 13 : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : 107 6.8 Effectiveness comarison between rank and general threshold for Heuristic 15 109 6.9 Effectiveness comarison between rank and general threshold for Heuristic 16 110 6.10 Effectiveness comarison of critical slices : : : : : : : : : : : : : : : : : : 113
ix Figure Page 6.11 A grahic summary of the effectiveness comarison between R a and R cs for all tested rograms with ositive coverage analysis : : : : : : : : : : : : : : 115 6.1 A grahic summary of the effectiveness comarison between R h and R d for all tested rograms with ositive coverage analysis : : : : : : : : : : : : : : 116 Aendix Figure D.1 Effectiveness comarison of heuristics in the order of tested rograms : : : : 14 D.1 Continued : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : 143 D. Effectiveness comarison between rank and general threshold in Figure 6.9 resented by the order of tested rograms : : : : : : : : : : : : : : : : : : : 144
x ABSTRACT Pan, Hsin. Ph.D., Purdue University, August 1993. Software Debugging with Dynamic Instrumentation and Test Based Knowledge. Major Professors: Richard A. DeMillo and Eugene H. Safford. Develoing effective debugging strategies to guarantee the reliability of software is imortant. By analyzing the debugging rocess used by exerienced rogrammers, we have found that four distinct tasks are consistently erformed: (1) determining statements involved in rogram failures, () selecting susicious statements that might contain faults, (3) making hyotheses about susicious faults (variables and locations), and (4) restoring rogram state to a secific statement for verification. This dissertation focuses on the second task, reducing the search domain for faults, referred to as fault localization. A new aroach to enhancing the rocess of fault localization is exlored based on dynamic rogram slicing and mutation based testing. In this new scenario, a set of heuristics was develoed to enable debuggers to highlight susicious statements and thus to confine the search domain to a small region. A rototye debugging tool, SPYDER, was reviously constructed to suort the first task by using dynamic rogram slicing and the fourth task by backward execution; some of the heuristics were integrated into SPYDER to demonstrate our aroach. An exeriment confirms the effectiveness and feasibility of the roosed heuristics. Furthermore, a decision algorithm was constructed as a ma to convey the idea of alying those heuristics. A new debugging aradigm equied with our roosed fault localization strategies is exected to reduce human interaction time significantly and aid in the debugging of comlex software.
1 1. INTRODUCTION The existence of errors in software develoment is inevitable because of human inability to erform tasks or to communicate erfectly.[deu79] Schwartz ointed out that in real world software systems, rogram errors are a fundamental henomenon, and a bug free rogram is an abstract theoretical concet.[sch71] As the size and the comlexity of rograms increase, more errors are introduced during software develoment. Along with the raid evolution of hardware, software lays an increasingly significant role in the success of many roducts, systems, and businesses.[pre87] In the software life cycle, more than 50% of the total cost is exended in the testing and debugging hases.[boe81, Mye79, Pre87] Develoing effective and efficient testing and debugging strategies to guarantee the reliability of software is thus imortant. In [ANS83], errors are defined as inaroriate actions committed by a rogrammer or designer. Faults or bugs are the manifestations and results of errors during the coding of a rogram. A rogram failure occurs when an unexected result is obtained while the rogram is executed on a certain inut because of the existence of errors and faults. In other words, a failure is a symtom of faults. Testing exlores the inut sace of a rogram that causes the rogram to fail, and debugging tries to locate and fix faults (bugs) after failures are detected during testing or use. Although testing and debugging are closely related, none of the existing debugging tools attemt to interface with testing tools. Conventional debugging tools (e.g., ADB and DBX [Dun86]) are command driven symbolic debugging tools and tend to be stand alone. Many fault localization strategies used in current well-known debugging tools (e.g., setting break-oints and tracing) were develoed in the 1960s and have changed little. [AS89] These debugging tools do not locate faults efficiently. Users have to discover by themselves information useful for debugging. Debugging rocesses are still labor intensive, and these
tools are thus little used.[joh83] To debug software systematically, it is imortant that a tool use techniques based on the debugging rocess of exerienced rogrammers, make hidden information available to users, and be directly accessible from the testing environment. However, these criteria have not been fully met by existing debugging tools. 1.1 Problems in Locating Faults Two major stes involved in the debugging rocess are locating and then correcting faults. Myers [Mye79] ointed out that the fault locating asect reresents 95% of the debugging effort. Several studies [GD74, Gou75, Ves85, Mye79, MM83, ST83], including behavioral research, suggest that locating faults is the most difficult and imortant task in the debugging rocess. Different strategies for locating faults would therefore affect the erformance of debugging. [SCML79] A reresentative aragrah from Martin [MM83] states: Traditionally, rogrammers have sent too much time looking for errors in the wrong laces. Myers [Mye78] found that rogrammers focused their attention on normal rocessing at the exense of considering secial rocessing situations and invalid inuts. Weinberg [Wei71] found that rogrammers have difficulty finding errors because their conjectures become rematurely fixed, blinding them to other ossibilities. Knowing what tyes of errors are likely to occur and where they are likely to occur in the rogram can avoid these roblems and greatly simlify the debugging rocess. All these studies conclude that locating faults is the most human-intensive and exensive task in the debugging rocess. 1. Scoe of This Research The major goal of this research is to find an efficient way to accomlish the task of locating faults. By analyzing the debugging rocess used by exerienced rogrammers, four
3 Failure Detected & Analyzed Further Testing Fault Localization More Testing Information? Y N N With Mutation-based Testing? Y N Same Search Domain? Heuristics without the Assistance of Further Testing Information Y (reduced search domain) Fault Guessing based on the search domain Heuristics with the Assistance of Mutation-based Testing Verifying the Guess N Fault Identified? Y Fix the Fault Figure 1.1 A new debugging scenario. Focus of this research, fault localization, is in the shaded area.
4 distinct tasks are found to be consistently erformed: 1) determining statements involved in rogram failures, ) selecting susicious statements that might contain faults, 3) making hyotheses about susicious faults (variables and locations), and 4) restoring rogram state to a secific statement for verification. If all four tasks are erformed with direct assistance from a debugging tool, the debugging effort becomes much easier. This dissertation focuses on the second task, reducing the search domain for faults, referred to as fault localization. To assist users in conducting the first and last tasks, two techniques Dynamic Program Slicing [ADS91a, AH90] and Execution Backtracking [ADS91b, AS88] were develoed by Agrawal, DeMillo, and Safford. A rototye debugging tool, SPYDER [ADS91a, ADS91b, AS88, Agr91], has been constructed with those techniques. To achieve the first task, SPYDER hels users automatically find the dynamic slice of a rogram for any given variables, locations, and test cases in terms of data and control deendency analysis, where a dynamic slice of a rogram is the set of statements that actually affect the value of a selected variable at a secific location after the rogram is executed against a given test case. 1 For the fourth task, SPYDER rovides a facility to restore the rogram state to a desired location by backtracking the rogram execution to that location and avoids the need to reexecute the rogram from the beginning. In this dissertation, a new aroach to enhancing the rocess of fault localization is exlored based on dynamic rogram slicing and mutation based testing. In the new scenario, a set of heuristics was develoed according to different situations and highlights susicious statements to confine the search domain to a reduced region. Some of the heuristics were integrated into SPYDER to demonstrate the feasibility and usefulness of our aroach. Figure 1.1 deicts stes in the roosed debugging aradigm. develoing owerful debugging tools is suggested as a result of this study. A new direction in 1 An informal definition of Dynamic Program Slicing is described in Chater 3.3. See [AH90, ADS91a, Agr91] for the formal definition. Program mutation has been studied for over a decade. It is well documented in the literature [CDK + 89, DGK + 88, BDLS80, Bud80, DLS78]. and is described in Chater 5.
5 1.3 Contributions The rincial contribution of this dissertation is a new debugging aradigm for enhancing the rocess of fault localization by reducing the search domain. Our aroach is based on information obtained from dynamic rogram slicing and mutation based testing, and does not guarantee that faults will always be recisely located in the reduced search domain. However, the reduced search sace containing faults or the information leading to fault discovery will hel users in debugging. This work makes contributions to both testing and debugging. From the debugging oint of view, the roosed aroach for fault localization rovides a reduced search domain for faults and imroves human interaction in debugging. The debugging field benefits from this new direction in develoing owerful tools. The effectiveness of integrating debugging and testing tools to enhancing the caability of locating faults has been demonstrated. From the testing oint of view, this new aroach can let the context be urosely switched between debugging and testing. Knowledge obtained from testing has been shown beneficial for debugging. This gives the testing field a new way to think of the usage of testing methodologies. The feasibility of using test based information to benefit debugging is demonstrated in this dissertation. 1.4 Organization of this Dissertation The rest of this dissertation is organized as follows. A brief survey of software debugging techniques is given in the next chater. Chater 3 describes the analysis of our new debugging aroach. We also define Exanded Dynamic Program Slicing, an enhancement of Dynamic Program Slicing, for debugging uroses. In Chater 4, we roose a set of heuristics without the assistance of further testing information. These heuristics are based on test cases and dynamic rogram slices. Chater 5 resents heuristics with the assistance of mutation based testing. An effective debugging instrument, Critical Slicing, derived from the simlest mutant oerator statement deletion is illustrated. In Chater 6,
6 we describe the integration of heuristics into a rototye debugging tool, SPYDER, along with results of a reliminary exeriment that was conducted to confirm the effectiveness and feasibility of the roosed heuristics. From the exerimental results, algorithms for alying the roosed aroaches are resented in Aendix E. Finally, the conclusion of this dissertation and future directions of this research are given in Chater 7.
7. RELATED WORK tools. We now briefly survey the debugging techniques used in existing software debugging.1 Traditional Techniques There are two brute force traditional debugging techniques according to Myer.[Mye79] One is analyzing memory dums that usually dislay all storage contents in octal or hexadecimal format. The other is scattering rint statements around susicious laces in a faulty rogram to dislay the value of variables and the flow of rogram execution. Users are exected to understand where the rogram goes wrong and the abnormal behavior of the rogram by using these techniques. However, these techniques are often inefficient because of a massive amount of data to be analyzed. It is time consuming to verify the correctness of a rediction by reeating the rocess of executing and analyzing. While the technique of analyzing memory dums is not often used now, the technique of scattering rint statements is still emloyed, esecially when debugging tools are not used. One debugging technique setting break-oints by users has been the main facility of many debugging tools for both low-level and high-level languages since the early 1960s (e.g., DDT [SW65] and FLIT [SD60] for assembly languages, and DBX [Dun86] for C). In order to avoid dealing with machine codes as well as scattering rint statements in a rogram, interactive symbolic debugging tools were develoed. These tools rovide not only the basic facility, setting break-oints, but also some of the following caabilities: dislaying values of variables, tracing reset trace-oints during execution, continuing execution of the rogram from a break-oint, executing single stes from a break-oint, modifying rogram states such as value of variables, and reexecuting a rogram (e.g., [Kat79, MB79, Bea83, Dun86]). These utilities can be emloyed by users to investigate a faulty rogram
8 interactively. First, users can set break-oints in the rogram. Then, the execution of the rogram will be susended at these break-oints. Users are allowed to examine the current rogram state (e.g., value of variables) at a susended oint and decide whether to examine the next statement, to continue forward execution, or to set new break-oints. These tools only rovide utilities to examine a snashot of rogram execution. Users have to conduct their own strategies to debug without roducing a massive amount of information for analysis. These drawbacks increase the difficulty of debugging. Another debugging technique kees track of execution history for backtracking. The concet of execution backtracking has been imlemented in database systems (e.g., rollback recovery for system failure [Ver78, HR83]) and in fault tolerant software systems (e.g., the recovery block technique [Ran75]). In the above imlementations, system states to be rolled back for later analysis must be set at the beginning. However, from the software debugging standoint, execution backtracking should be able to gradually backtrack rogram execution from any break-oint under the control of users. EXDAMS [Bal69], for examle, is an interactive debugging tool equied with this technique. The major roblem with imlementing backtracking is the consumtion of large amounts of sace for storing the execution history and rogram states. Agrawal, DeMillo, and Safford [ADS91b] roosed a structured backtracking aroach to imlement execution backtracking without storing the whole execution history. Their aroach saves the latest value of variables changed in a statement and only allows backtracking to the rogram state rior to a comlete statement. The idea of a user-friendly interface has been built into some debugging tools such as Dbxtool [AM86]. Windows and a mouse are used to handle the selections in debugging rocesses instead of the traditional command driven aroach which accets tyed commands only. Being able to dislay information (e.g., rogram execution flow, break-oints, and rogram states) simultaneously makes the debugging rocess more convenient and efficient.
9. Algorithmic Aroaches Shairo [Sha83] roosed an interactive fault diagnosis algorithm, the Divide and Query algorithm, for debugging. The algorithm will recursively search a comutation tree that reresents the target rogram until bugs are located and fixed. At each node of the comutation tree, a query of node n will divide the tree rooted at node n into two roughly equal subtrees. If the result of an intermediate rocedure call at node n is correct, the algorithm omits the tree rooted at n and iterates; otherwise the algorithm will be recursively alied to the two subtrees of node n. Shairo roved that if a rogram is correct, every subrogram of the rogram is also correct because the comutation tree of a subrogram is a subset of the comutation tree of the whole rogram. If a rogram is not correct, then there exists at least one erroneous subrogram in it. The correctness of each intermediate rocedure call at the node of a comutation tree will be verified by users. This aroach is suitable for debugging rograms that can be well reresented as a comutation tree. Logic rograms, such as those written in Prolog, are the best candidates. A few enhanced debugging systems for Prolog rograms have been develoed based on this aroach.[per86, Fer87] Renner tried to aly this aroach to locating faults in Pascal rograms.[ren8] In order to verify the results of intermediate rocedure calls mentioned above, an oracle is imlemented to ask users the correctness of the return values of rocedures in Pascal. Afterwards, the debugging system would investigate and execute each rocedure in a to down direction until a rocedure fails or generates incorrect results detected by the oracle. A rocedure containing bugs is thus located and the goal is achieved. The rimary limitation of alying this aroach to rograms written in structured languages such as Pascal is that it can only oint out the rocedure containing bugs. Other debugging tools are needed to debug the faulty rocedure. A similar result can be found in [FGKS91].
10.3 Knowledge Based Aroaches This aroach attemts to automate the debugging rocess by using the techniques of artificial intelligence and knowledge engineering. Many existing automated rototye systems for rogram debugging have been develoed based on this aroach since the early 1980s.[DE88, Sev87] Knowledge of both the classified faults and the nature of rogram behavior is usually required in this aroach. However, the knowledge required for real world rograms is too comlicated. The rototye systems mentioned in [DE88, Sev87] can only handle restricted fault classes and very simle rograms. A few reresentative debugging tools using knowledge based aroaches are reviewed in this section. PUDSY (Program Understanding and Debugging SYstem) [Luk80] analyzes a rogram before starting the rocess of debugging. Inuts to PUDSY are a rogram and a secification of the rogram. A knowledge base is maintained for the first hase analyzing and understanding a rogram. The knowledge base kees a set of rogramming schemas, which describe the simle tyical behavior of each statement block (e.g., schema to find the maximum element in an array, and schema to exchange values of two elements). For each schema, there exists a set of assertions to formalize the function of that schema. After the target rogram is automatically decomosed into chunks by heuristic methods, PUDSY will search the knowledge base to match rogramming schemas with those chunks. If a matched air is found, the related assertion of the schema is constructed for the matched chunk of the given rogram. Then the debugging hase starts by comaring these assertions with the given secification. If any of these assertions violates the given secification, bugs are located in a corresonding chunk of code. Tyes of the bugs can be found by comaring the assertion with the given secification. Because PUDSY needs the secification of the faulty rogram, only one subrogram (e.g., a rocedure or a function) can be handled at a time. Otherwise, the corresonding secification for a comlete rogram is too comlicated to be described. The major limitation of this system is the variety of the rogramming schemas in the knowledge base. Only a few tyical schemas can be reresenteded well.
11 Laura [AL80] uses a straightforward aroach to debug students rograms. The secification of a rogram given to the system by teachers is the correct rogram model. The system comares rograms written by students with the correct model. Programs for comarison are transformed into internal reresentation forms flow grahs. Then, the flow grahs are systematically comared by Laura to automatically locate bugs. Because of the limitation of flow grahs to reresenting comlicated rograms, this system is designed for tutorial uroses in classrooms. PROUST [JS84, JS85] is an intention based diagnosis system to hel novice rogrammers learn how to write correct rograms. It does online analysis and understanding of Pascal rograms in order to identify nonsyntactic bugs. PROUST takes a rogram and a descrition of the rogram s intentions as inut. The descrition is not an algorithm; it is just a list of goals (intentions). The goals are then synthesized and imlemented by rogramming lans retrieved from a knowledge base. A rogramming lan contains statements and subgoals to imlement a tyical function. Then, the synthesized goals are comared with the code of the given rogram. If the matching results are negative, all arties involved are analyzed. Diagnosis of the roblem and suggestions for correcting the faults are reorted to novices. However, if there are no roer rogramming lans to synthesize the given goal (intention), bugs are reorted even if the code of the given rogram is correct. This roblem becomes serious with comlicated rograms. TALUS [Mur85, Mur86b, Mur86a] emloys an aroach similar to PROUST s, but uses a theorem rover to do the transformation from rogramming lans to codes. This method artially solves the roblem of restricted rogramming lans in PROUST. However, TALUS can only handle small LISP rograms. Generally seaking, the above systems, classified as rogram analysis techniques [Sev87], can only handle relatively small rograms because they have to fully understand and statically analyze the rograms to be debugged. The necessity of detailed understanding of rograms revents these techniques from being alied to ractical rograms. Instead of statically analyzing the entire rogram, FALOSY (FAult LOcalization SYstem) [ST83] emhasizes fault localization by comaring the actual outut with the exected
1 outut. Automatic fault location is erformed by analyzing outut discreancies with the hel of restored fault cause effect knowledge. Two major resources are maintained in the knowledge base. One is a set of heuristics describing the maing between outut discreancies (fault symtoms) and ossible causes. The other is a set of functional rototyes of certain standard rogram tasks (e.g., sorting). Based on outut discreancies, FALOSY searches the ossible causes of the fault symtoms and comares the functional rototyes retrieved from its knowledge base with the given rogram. Then, the faults are reorted in terms of the rototye schema (similar to the rogramming lans in PROUST) rather than the buggy code. Unfortunately, only a limited class of rograms and faults are imlemented in this system. Harandi [Har83] resented a heuristic model for knowledge based rogram debugging. The system aims to debug comile time errors and certain run time errors. It is assumed that most of the debugging knowledge of exerienced rogrammers can be encoded as heuristic rules in the form of situation action airs. The situation art summarizes the ossible symtoms of errors. The action art includes ossible causes of errors and suggestions for fixing the errors. The system matches the resent error symtoms and the information, which is rovided by users or obtained through analysis of the rogram, with the situation art of the rules. Then, the corresonding actions of the rules are invoked. Because comilation errors can easily be fixed by exerienced rogrammers using the error messages rovided by the comiler, this system is mainly used for tutoring uroses..4 Program Slicing Aroach Program slicing was roosed by Weiser [Wei8, Wei84] as another aroach for debugging. Weiser s rogram slicing (also called static slicing) decomoses a rogram by statically analyzing data flow and control flow of the rogram. A static rogram slice for a given variable at a given statement contains all the executable statements that could influence the value of that variable at the given statement. The slice is a subset of the original rogram. The exact execution ath for a given inut is a subset of the static rogram slice with resect to the outut variables at the given checkoint. If a rogram fails on a test case
13 and a variable is found incorrect at a statement, we can hyothesize that bugs are highly likely in the rogram slice regarding that variable statement air because only statements having influence on that variable statement air could cause the incorrect result. Thus, the search domain for faults is reduced from the entire rogram to the rogram slice. Focus [LW87] is an automatic debugging tool based on static rogram slicing to locate bugs. It attemts to combine rogram slices with test cases to find the susicious region containing faults, which is assumed to be a subset of a rogram slice. Two grous of slices are formed based on the rogram oututs after executing selected test cases referred to as rogram dicing by Lyle and Weiser.[Lyl84, WL86] One grou contains slices with resect to erroneous variables. The other grou contains slices with resect to variables having correct values. Focus tries to confine the susicious region for faults by choosing statements in the former slice grou but not in the latter one for a given susicious variable statement air. However, the slice generated by Focus contains many statements with no influence on the susicious variable statement air because of the feature of static rogram slicing. The static rogram slicing aroach cannot resolve runtime ambiguities, thus highlights many surious statements with no influence on the incorrect results. In this case, the faulty statements cannot be effectively identified. Korel and Laski [KL88a, KL90] extended Weiser s static rogram slicing to dynamic rogram slicing (K-L slicing). K-L slicing defines an executable subset of an original rogram that comutes the same function for given variables and inuts. In this case, their aroach does not show the exact influence on a articular value of the given variable, location, and test case. For examle, for a given variable V at the end of a loo and a given test data d, statement S 1 in the loo affects the value of V during the first iteration, and statement S in the loo, not S 1, affects the value of V during the second iteration. According to the definition of K-L slicing, both statements S 1 and S will be included in the K-L dynamic rogram slice for variable V and test data d in order to form an executable subrogram. However, if we want to know the exact influence on the articular value of V at the end of the second iteration, K L slicing cannot give the information because of the existence of S 1 in this situation.
14 Agrawal and Horgan [AH90] claimed the usefulness of a dynamic slice lies not [only] in the fact that one can execute it, but in the fact that it isolates only those statements that [actually] affected a articular value observed at a articular location. They use the rogram deendency grah and an extended static slicing algorithm to construct dynamic rogram slices. Dynamic rogram slices obtained in this way are similar to those defined by Korel and Laski. Thereafter, in order to comute accurate dynamic slices such as in the examle mentioned above, a dynamic deendency grah and secialized algorithms are emloyed. Accurate dynamic rogram slicing can isolate statements that actually affect a articular value of a susicious variable at a given location for a given inut. Thus, in the above examle, the exact influence on the articular value of V at the end of the second iteration, which is statement S not statement S 1, will be shown in the accurate dynamic rogram slice. Dynamic rogram slicing for ointers based on the same aroach has been imlemented in the rototye debugging tool SPYDER [ADS91a, ADS91b, Agr91]. Prior the work reorted here, dynamic rogram slicing had not been systematically alied to fault localization, although in Agrawal s dissertation [Agr91] he briefly alluded to the idea of combining dynamic rogram slices and data slices for fault localization. Part of our heuristics are based on dynamic slices that are collected by varying test cases, variables, and location of variables..5 Test Integrated Suort The relationshi between testing and debugging has never been clearly defined. Current testing and debugging tools are indeendent of each other. Even if they are integrated in one tool, strengthening the caability of this tool to detect and to locate faults needs to be seriously studied. This section will focus on the ossibility of using the information derived from existing testing methodologies for debugging uroses. Osterweil [Ost84] attemted to integrate testing, analysis, and debugging, but gave no solid conclusion about how to transform information between testing and debugging to benefit each other. An interesting result is suggested by Osterweil s research: debugging
15 is robably best suorted by a mix of static and dynamic caabilities, just as is the case for testing and verification. This oints out a valuable direction for building new debugging tools. Clark and Richardson [CR83] were the first to suggest certain testing strategies and classified failure tyes can be used for debugging uroses. Only one examle was given to describe their idea. No further detailed study was conducted. They described how certain test data selection strategies based on symbolic evaluation [How77, CHT79, CR81] can hel the debugging rocess. Tyical error tyes classified by them include: erroneous reference to an inut value; erroneous rocessing of secial inut values; erroneous rocessing of tyical/atyical values; erroneous loo rocessing (e.g., never terminated, never executed); and erroneous roduction of secial/tyical/atyical outut values. For each error tye, test data are selected according to the result of symbolic evaluation. If an error in a rogram is detected after selected test data are alied to the rogram, we can know the otential tye of the error and then locate the bugs through the attributes of the test data and the erroneous results. STAD (System for Testing And Debugging) [KL88b, Las90, Kor86] is the first tool to integrate debugging with testing. Nevertheless, its testing and debugging arts do not share much information together excet for imlementation uroses (e.g., they share the results of data flow analysis). Information obtained from the testing hase for debugging uroses consists of the execution history of rogram failures and the test cases causing rogram failures. In this situation, the caability of STAD for locating bugs is no better than that of indeendent and unintegrated tools. The data flow testing methodology is the testing technique of STAD. The debugging art will be invoked once a fault is detected during a testing session. STAD uses the structure of the rogram and the execution trajectory of a failure as its knowledge. The knowledge is reflected by a rogram deendence network, which is derived from deendency analysis (data flow and control flow deendency). A set of hyotheses indicating otential locations of faults is generated by using the rogram deendence
16 network. Then, all hyotheses in the set are verified by users at reset break-oints while reexecuting the rogram. The main goal of fault localization in the debugging session of STAD is to hel users focus on the ossible erroneous region, rather than recisely locating faults. PELAS (Program Error-Locating Assistant System) [Kor88] is an imlementation of the debugging art of STAD. Korel and Laski roosed an algorithm based on the hyothesis and test cycle and the above knowledge to localize faults interactively.[kl91] Only a subset of Pascal is suorted, and limited rogram errors are considered in STAD and PELAS. Collofello and Cousins [CC87] roosed a set of heuristics to locate susicious statement blocks after a thorough test. A rogram is first artitioned into many decision to decision aths (DD aths), which are comosite statements existing between redicates. After testing, two test data sets are obtained: one detects the existence of faults and the other does not. Then, heuristics are emloyed to redict ossible DD aths containing bugs based on the number of times that DD-aths are involved in those two test data sets. The idea of these heuristics heled us develo our roosed fault localization strategies. The main restriction of their heuristics is that only execution aths that are a secial case of dynamic rogram slicing [AH90] are examined. After the search domain is reduced to a few statement blocks (DD-aths), no further suggestion is rovided for locating bugs..6 Summary Araki, Furukawa and Cheng summarized the debugging stes conducted by exerienced rogrammers and roosed a debugging rocess model as a general framework. [AFC91] They also ointed out that debugging tools must suort each stage in the debugging rocess: hyothesis verification, hyothesis set modification, and hyothesis selection. However, they did not describe what kinds of facilities and functions will be used to suort each stage. Unlike the aroaches roosed by Lyle Weiser and Collofello Cousins, which are only based on susicious variables and test cases, resectively, our heuristics are develoed by considering test cases, variables, and location of variables together.
17 The evolution of software debugging has not made much rogress during the last three decades. The most oular debugging techniques emloyed by commonly used debugging tools (e.g., DBX), setting break-oints by users and tracing, were introduced around the early 1960s. [Sto67] From the history of the develoment of fault localization, we find that techniques in some rototye systems work only for rograms with restricted structure and solve only limited roblems. An efficient debugging aradigm that deals with a broader scoe of faults is needed.
18 3. ANALYSIS OF A NEW DEBUGGING APPROACH Most studies of fault analysis [Knu89, BP84, OW84, Bow80, Li79, AC73, JSCD83, SPL + 85] classify faults collected from long term rojects. However, no further studies have been conducted based on the categorized information. We believe that there is value to debugging research in such analysis. In order to observe and analyze rogram failures, dynamic instrumentation caable of doing (dynamic) rogram deendency analysis is chosen as a basic facility of the new debugging aroach. [Pan91] Dynamic Program Slicing can determine statements actually affecting rogram failures so that the search domain for faults will be reduced. Although it is not guaranteed that dynamic slices always contain the faults (e.g., missing statement or secification faults), to investigate statements actually affecting rogram failures is a reasonable strategy in debugging. By analyzing semantics and values of variables in susicious statements of dynamic slices, we might discover valuable information for debugging. Therefore, we choose dynamic slices as the search domain to locate faults. At the same time, the knowledge derived from the testing hase (e.g., test cases and fault analysis from fault based testing methodology) would be helful to the debugging rocess. In this chater, the general background of our analysis is first described. Then, we enhance the existing dynamic rogram slicing technique by develoing Exanded Dynamic Program Slicing (EDPS) that has better ability to include faulty statements for debugging. In addition, we conduct a generic analysis for test based knowledge to hel us construct debugging strategies based on available information from the testing hase. 3.1 Background In the testing hase, multile test cases executed against P resent different kinds of information. Our goal is to extract as much of that information as ossible for debugging.
19 The more test cases we get, the better results we can have by investigating the information obtained from testing. Therefore, we refer a thorough test finishing the testing rocess to satisfy as many criteria of a selected testing method as ossible. After a thorough test, if the existence of faults in rogram P is detected, then at least one test case will cause P to fail. Such a test case is called an error revealing test case, and T f is the set of all such test cases. Likewise, the test cases on which P generates correct results are called non error revealing test cases, and T s is the set of them. A set of error revealing test cases is an indisensable resource for debugging. On the other hand, not every non error revealing test case is useful. Partition analysis on the inut domain hels us identify the subdomains associated with rogram failures. [WO80, RC85, HT90, WJ91] After a thorough test, users can artition the inut domain of P based on the secification of P and the results from testing. If an inut subdomain of secification contains both error revealing and non revealing test cases after testing, then those non revealing test cases are considered for debugging. Test cases in the inut subdomain indicate the divergence between the exected and abnormal behavior of P after being executed against P. The divergence of rogram behavior with resect to the test cases rovide valuable clues for fault localization. If all test cases, which are constructed for the thorough test, in an inut subdomain of secification are non error revealing ones, they are merely evidence to suort the correctness of P for the certain inut domain. It is likely that they do not rovide direct hel for locating faults. We thus ignore those non error revealing test cases at the beginning of debugging rocess to save effort. In order to reduce the size of susicious inut subdomains to a minimum, we refer a well defined inut domain and secification of a given faulty rogram. Analyzing the results of rogram failures will hel identify susicious variables (e.g., outut variables) that contain unexected values. Dynamic slices with resect to these susicious variables and corresonding test cases are then constructed for the new debugging aroach. Based on the selected test cases obtained from the artition analysis as mentioned
0 above, we analyze the divergence among dynamic slices to understand rogram failures. This analysis hels us construct heuristics in the next chater. In brief, the general analysis can be summarized as follows. Emloy a thorough test to collect as many relevant test cases as ossible for an effective debugging rocess. Construct a well-defined inut domain and secification of a given faulty rogram to reduce the susicious inut domain to a minimum. We can then deal with minimum dynamic rogram slices with resect to related test cases, and the task of fault localization will be more efficient. Focus on the smallest set of related test cases one at a time, esecially for the non error revealing test cases. Conduct logical oerations, such as intersection, union, and difference, on dynamic rogram slices with resect to selected test cases to study similarities or differences among these dynamic slices. 3. Heuristics vs. Deterministic Stes Unlike testing, debugging is an unstructured, sontaneous, and mystical rocess.[tra79, Lau79] The tasks erformed in the rocess of debugging are to collect valuable information for locating faults. The information may come from rogram secification, rogram behavior during execution, results of rogram execution, available test cases, etc. Deterministic decisions (the aroach that systematically analyzes comlicated information to benefit software debugging) may not be ossible for the general case, and those for secific alications do not yet exist. We thus adot a heuristic aroach as a feasible solution to enhance the debugging rocess. We believe that heuristics, which gather useful information from different cases, can cover varied situations and hel us localize faults. The heuristic aroach is like a forward reasoning technique to locate faults based on existing information. On the other hand, to construct deterministic stes, we need first
1 examine all actual behavior and understand information flow made by faults, then induce systematical solutions for locating faults. This aroach is like a backward reasoning technique. However, understanding all actual behavior and information flow made by faults is a difficult task. The exact maing between results and causes for localizing faults is another obstacle to overcome. Thus the heuristic aroach is a feasible way for debugging at the current stage. Although each heuristic does not rovide a general solution and is only suitable for some secific situations, the overall debugging ower from uniting these heuristics is exected to surass that of currently used debugging tools. 3.3 Exanded Dynamic Program Slicing In this section, we enhance Dynamic Program Slicing [AH90, ADS91a, Agr91], which was chosen as our basic instrument, to handle certain tyes of faulty statements. An informal definition of Dynamic Program Slicing mentioned in [ADS93] is quoted and summarized as follows. 1 There are two major comonents to constructing a dynamic rogram slice: the dynamic data slice and the dynamic control slice. A dynamic data slice with resect to a given exression, location, and test case is a set of all assignments whose comutations have roagated into the current value of the given exression at the given location. This is done by taking the transitive closure of the dynamic reaching definitions of the variables used in the exression at the given location. The set of all assignments that belong to this closure forms the dynamic data slice. On the other hand, a dynamic control slice with resect to a given location and test case is a set of all redicates that enclose the given location after executing the test case. This is done by taking the transitive closure of the enclosing redicates starting with the given location. The set of all redicates that belong to this closure forms the dynamic control slice. Thus 1 Readers are referred to [AH90, ADS91a, Agr91] for the formal definition.