Lokalisieren von Defekten in Software mit Hilfe von Data Mining

Transcription

1 Lokalisieren von Defekten in Software mit Hilfe von Data Mining Klemens Böhm NAME INSTITUTE OF INSTITUTE, FOR FACULTY, PROGRAMME DEPARTMENT STRUCTURES (change AND DATA masterorganisation view) (IPD) Please insert a figure in the master transparency. KIT University of the State of Baden-Wuerttemerg and Photo: Stadt Karlsruhe National Research Center of the Helmholtz Association KIT University of the State of Baden-Wuerttemerg and National Laoratory of the Helmholtz Association

2 A prolem has een detected and windows has een shut down to rpevent damage to your computer. Please click on to add your Title DRIVER_IRQL_NOT_LESS_OR_EQUAL Please click on to add your Title If this is the first time you`ve seen this stop error screen, restart your computer, If this screen appears again, follow these steps: Check to make sure any new hardware or software is properly installed. If this is a new installation, ask your hardware or software manufacturer for any windows updates you might need. If prolems continue, disale or remove any newly installed hardware or software. Disale BIOS memory options such as caching or shadowing. If NAME you OF need INSTITUTE, to use FACULTY, safe DEPARTMENT Mode to (change remove master or view) disale components, restart your computer, press F8 to select Advanced Startup Options, and then Select Safe Mode. Technical information: Please insert a figure in the master transparency. *** STOP: 0x000000D1 (0x C, 0x , 0x , 0xF86B5A89) *** gv3.sys Adress F86B5A89 ase at F86B5000, DateStam 3dd991e Beginning dump of physical memory Physical memory dump complete. Contact your system administrator or technical support group for further assistance KIT University of the State Prof. Baden-Wuerttemerg Max Mustermannand - Title National Research Center of the Helmholtz Association

3 Locating Bugs in Software Software is almost never shipped ug-free, even if tested extensively. Deugging is time consuming and expensive. Automated localisation would e of great help. Idea: Locate ugs with data mining techniques. Use an approach ased on weighted graph mining. 3Motivation Klemens Böhm Lokalisieren von Defekten in Software mit Hilfe von Data Mining 1/25

4 Outline 1. Motivation 2. Call Graphs 3. Defect Localization 4. Dataflow-Enaled Defect Localization 5. Evaluation 6. Conclusions 4Motivation Klemens Böhm Lokalisieren von Defekten in Software mit Hilfe von Data Mining 2/25

5 Call Graphs Program executions as call graphs: methods nodes method calls edges a c 5Motivation Klemens Böhm Lokalisieren von Defekten in Software mit Hilfe von Data Mining 3/25

6 Call Graphs Program executions as call graphs: methods nodes method calls edges a Bugs in the call graph: Structure affecting (existing approaches) E.g., a ug in an if-condition in a c 6Motivation Klemens Böhm Lokalisieren von Defekten in Software mit Hilfe von Data Mining 4/25

7 Call Graphs Program executions as call graphs: methods nodes method calls edges a Bugs in the call graph: Structure affecting (existing approaches) E.g., a ug in an if-condition in a c Call frequency affecting (new in our contriution) E.g., a ug in a loop-condition in c 7Motivation Klemens Böhm Lokalisieren von Defekten in Software mit Hilfe von Data Mining 5/25

8 Reduction of Call Graphs printout showdelete showdelete showsame showinsert showdelete showsame showinsert showinsert showinsert showdelete showdelete showdelete showsame showsymol showsymol printin showsymol printin showsymol printin showsymol showsymol showsymol printin showsymol showsymol showsymol Millions of method calls are very common. Reduction of call graphs is necessary. a a c 1 1 c 3 Example of reduced call graph. Weights have not een used in previous reductions. 8Motivation Klemens Böhm Lokalisieren von Defekten in Software mit Hilfe von Data Mining 6/25

9 Outline 1. Motivation 2. Call Graphs 3. Defect Localization 4. Dataflow-Enaled Defect Localization 5. Evaluation 6. Conclusions 9Motivation Klemens Böhm Lokalisieren von Defekten in Software mit Hilfe von Data Mining 2/25

10 How to Use Call Graphs? Existing idea: Look at program executions (correct ones as well as failing ones) represented as call graphs and analyse the graph structure (i.e., graphs without weights). Identify patterns typical of failing executions. Frequent sugraph mining. In a nutshell, this gives way to finding structure-affecting ugs. New aspect: Explicitly analyse call frequencies (edge weights) esides graph structures. 10 Motivation Klemens Böhm Lokalisieren von Defekten in Software mit Hilfe von Data Mining 7/25

11 Weighted Graph Mining No algorithm availale for weighted graph mining (as well as no prolem formulation). How to make use of edge weights? Preprocessing Discretisation of weights (We are currently investigating this.) Postprocessing (our approach already studied and presented susequently) Graph mining without weights Susequent detailed analysis of weights 11 Motivation Klemens Böhm Lokalisieren von Defekten in Software mit Hilfe von Data Mining 8/25

12 Finding Discriminating Edge Weights (1) Apply frequent sugraph mining to reduced call graphs of correct and failing program executions (ignore the weights in this step) (2) Consider only sugraphs occurring in oth correct and failing executions (3) Analyse the edge weights Example graph found y frequent sugraph mining: a c Average weight of a c in correct executions: 1 Average weight of a c in failing executions: 1 Average weight of c in correct executions: 3 Average weight of c in failing executions: Motivation Klemens Böhm Lokalisieren von Defekten in Software mit Hilfe von Data Mining 9/25

13 Finding Discriminating Edge Weights (1) Apply frequent sugraph mining to reduced call graphs of correct and failing program executions (ignore the weights in this step) (2) Consider only sugraphs occurring in oth correct and failing executions (3) Analyse the edge weights Example graph found y frequent sugraph mining: a c Average weight of a c in correct executions: 1 Average weight of a c in failing executions: 1 Average weight of c in correct executions: 3 Average weight of c in failing executions: Motivation Klemens Böhm Lokalisieren von Defekten in Software mit Hilfe von Data Mining 10/25

14 Entropy Based Scoring Assemle a tale which contains every edge in every frequent sugraph: a c a SG 1 SG 2 SG 1 SG 1 SG 2 a c c a Class Execution failing Execution correct Application of the Information Gain feature-selection algorithm, ased on entropy. Result: Ranking of the columns (edges) 14 Motivation Klemens Böhm Lokalisieren von Defekten in Software mit Hilfe von Data Mining 11/25

15 Integration of Structural Evidence reduced call graphs in failing and correct frequent sugraph mining (not considering weights) in failing only entropy ased scoring (ased on edge weights) structural scoring (ased on support) comination comined method ranking 15 Motivation Klemens Böhm Lokalisieren von Defekten in Software mit Hilfe von Data Mining 12/25

16 Integration of Structural Evidence reduced call graphs in failing and correct frequent sugraph mining (not considering weights) in failing only entropy ased scoring (ased on edge weights) structural scoring (ased on support) comination comined method ranking 16 Motivation Klemens Böhm Lokalisieren von Defekten in Software mit Hilfe von Data Mining 13/25

17 Integration of Structural Evidence in failing and correct entropy ased scoring (ased on edge weights) reduced call graphs frequent sugraph mining (not considering weights) in failing only structural scoring (ased on support) Usage of two kinds of evidence: frequency (left) and structure (right) Both types of ugs canelocated: call frequency and structure affecting ones comination comined method ranking 17 Motivation Klemens Böhm Lokalisieren von Defekten in Software mit Hilfe von Data Mining 14/25

18 Results Example output: METHOD SCORE 1 inputscan() showinsert() showdelete() oldconsume() addsymol() Motivation Klemens Böhm Lokalisieren von Defekten in Software mit Hilfe von Data Mining 15/25

19 Results Example output: METHOD SCORE 1 inputscan() showinsert() showdelete() oldconsume() addsymol() The ug has een instrumented in showinsert(). A software developer has to check two methods only. We do not try to explain/fix ugs, just to locate them! 19 Motivation Klemens Böhm Lokalisieren von Defekten in Software mit Hilfe von Data Mining 16/25

20 Outline 1. Motivation 2. Call Graphs 3. Defect Localization 4. Dataflow-Enaled Defect Localization 5. Evaluation 6. Conclusions 20 Motivation Klemens Böhm Lokalisieren von Defekten in Software mit Hilfe von Data Mining 2/25

21 Challenge: Dataflow-Affecting Bugs Not all defects affect the call graph. Example (failing execution): voidmain() void a () void () 1 int c(12, 33) 1. Method calls c(12, 33). 2. Method c calculates a wrong return value Method prints value 8. Graph structure and weights remain unaffected. Call-graph mining cannot find the defect. Approach: Extend call graphs with dataflow information and analyse these extended graphs. 21 Motivation Klemens Böhm Lokalisieren von Defekten in Software mit Hilfe von Data Mining 17/25

22 Dataflow Information Availale Example dataflows (parameter and return values) of int c(int p1, int p2) from : In real executions, one method can e called million times. To deal with this data, we need to aggregate it. 22 Motivation Klemens Böhm Lokalisieren von Defekten in Software mit Hilfe von Data Mining 18/25

23 Our Approach: Discretisation with CAIM CAIM (class-attriute interdependence maximization) class-aware (correct, failing) parameter-free, i.e., it automatically determines a possily small numer of intervals, homogeneous wrt. the class (a) Exemplar call data. Exec. p1 p2 r class correct correct correct failing failing failing failing correct correct () Intervals generated. Value Intervals p1 i 1 : [1; 11.5] i 2 : (11.5; 23] p2 i 1 : [2; 13.5] i 2 : (13.5; 38] i 3 : (38; 47] r i 1 : [5; 8.5] i 2 : (8.5; 13] (c) Discretised data. Exec. p1 p2 r 1 i 1 i 3 i 2 1 i 1 i 3 i 2 1 i 1 i 1 i 2 2 i 2 i 2 i 1 3 i 2 i 2 i 1 3 i 2 i 2 i 1 3 i 2 i 2 i 1 4 i 1 i 1 i 2 4 i 1 i 3 i 2 23 Motivation Klemens Böhm Lokalisieren von Defekten in Software mit Hilfe von Data Mining 19/25

24 Dataflow-Enaled Call Graphs (DEC Graphs) How to represent discretised dataflow data in call graphs? Attach tuples of weights (i.e., counters) to edges: (t; p i 1 1 ; ; p i n1 1 ; ; p i 1 m ; ; p i nm m ; r i 1 ;...; r i nr ) - t: total numer of calls (as efore) - p 1 ; ; p m ; r: parameter values and return value - i 1 ; ; i n x : intervals of the parameter/return values 52 voidmain() void a () 4 2 void () 1,0,1,0,1,0,1, 0 int c(int p1, int p2) Exec. p1 p2 r 1 i 1 i 3 i 2 1 i 1 i 3 i 2 1 i 1 i 1 i 2 2 i 2 i 2 i 1 3 i 2 i 2 i 1 3 i 2 i 2 i 1 3 i 2 i 2 i 1 4 i 1 i 1 i 2 4 i 1 i 3 i 2 Example call c: (t; p i 1 1 ; p i 2 1 ; p i 1 2 ; p i 2 2 ; p i 3 2 ; r i 1 ; r i 2 ) 24 Motivation Klemens Böhm Lokalisieren von Defekten in Software mit Hilfe von Data Mining 20/25

25 Defect Localisation with DEC Graphs 1. Assemle classified DEC graphs for all executions. 2. Perform frequent sugraph mining (CloseGraph). We use sugraphs as contexts to detect defects that occur in certain situations only. 3. Assemle a feature tale with all tuple elements from all edges in all frequent sugraphs separately: 4. Derive a ranking of the columns (and thus the methods) with entropy-ased feature selection (GainRatio). Various optimisations. 25 Motivation Klemens Böhm Lokalisieren von Defekten in Software mit Hilfe von Data Mining 21/25

26 Outline 1. Motivation 2. Call Graphs 3. Defect Localization 4. Dataflow-Enaled Defect Localization 5. Evaluation 6. Conclusions 26 Motivation Klemens Böhm Lokalisieren von Defekten in Software mit Hilfe von Data Mining 2/25

27 Evaluation of Defect Localisation with DEC Graphs A data-centric application: Weka (class DecisionStump) 16 instrumented defects, similar to related evaluations. Most dataflow-affecting ugs indirectly affect the call graph. 90 executions of every version with UCI data The result tale contains the ranking position, i.e., numer of methods one has to inspect to find a defect: With improvements only 1.5 methods on average (out of 30 actually executed methods). 27 Motivation Klemens Böhm Lokalisieren von Defekten in Software mit Hilfe von Data Mining 23/25

28 Conclusions A domain-specific data-mining prolem: software-defect localisation Contriutions: A domain and prolem-specific data representation: DEC graphs (discretisation) An analysis process for DEC graphs (sugraph mining, feature selection) Localization of dataflow-affecting ugs esides other ug classes Future improvements: Dataflows through gloal variales Scalale analysis of large software projects Parallel programs 28 Motivation Klemens Böhm Lokalisieren von Defekten in Software mit Hilfe von Data Mining 24/25

29 Acknowledgements Thanks to Frank Eichinger for letting me use his slides. 29 Motivation Klemens Böhm Lokalisieren von Defekten in Software mit Hilfe von Data Mining 25/25