A TraceLab-based Solution for Creating, Conducting, Experiments

Transcription

1 A TraceLab-based Solution for Creating, Conducting, and Sharing Feature Location Experiments Bogdan Dit, Evan Moritz, Denys Poshyvanyk 20 th IEEE International Conference on Program Comprehension (ICPC 12) Passau, Germany

2 Have you ever tried to reproduce a case study and you couldn't? 2

3 Case Study in SE In general, it is difficult to reproduce case studies Some details may be missing Results of case studies produce contradictory results * *T T. Menzies and M. Shepperd, "Special Issue on Repeatable Results in Software Engineering Prediction", Empirical Software Engineering (ESE), vol. 17, no. 1-2, 2012, pp

4 In Feature Location (FL) Case studies are hard to reproduce** Different datasets Different implementations Different metrics ** Dit, B., Revelle, M., Gethers, M., and Poshyvanyk, D., "Feature Location in Source Code: A Taxonomy and Survey", Journal of Software Maintenance and Evolution: Research and Practice (JSME), 2012, doi: /smr

5 Different Datasets Out of 60 papers 5% Datasets 5

6 Different Datasets Out of 60 papers 5% Datasets 95% Different software systems Different versions of the systems Different data points from the same system 6

7 Different Implementation Often source code of FLTs is not available Configuration parameters are missing 7

8 Different Metrics Precision Recall F-measure Effectiveness measure 8

9 Other Observations Out of 60 papers Compared their technique against a 38% limited number of existing techniques 9

10 Other Observations Out of 60 papers 62% 38% Compared their technique against a limited number of existing techniques No Comparison 10

11 Why is Reproducibility of Case Studies Important? 11

12 Why is Reproducibility of Case Studies Important? Confirm if field is making progress 12

13 Why is Reproducibility of Case Studies Important? Confirm if field is making progress (Many requests for tools and data at ICPC 12) 13

14 Why is Reproducibility of Case Studies Important? Results can be checked by reviewers Results can verified by other researchers Clarify some aspects not explicitly i l stated in the paper Facilitate publishing new papers (others can use the data) 14

15 How to Publish a new FL Paper? New FLT 15

16 How to Publish a new FL Paper? Datasets New FLT 16

17 How to Publish a new FL Paper? Datasets New FLT Few available Hard to generate 17

18 How to Publish a new FL Paper? Datasets Results New FLT 18

19 How to Publish a new FL Paper? Datasets Results? New FLT 19

20 How to Publish a new FL Paper? Datasets Results New FLT FLT#1 FLT#2 FLT#3 FLT#4 FLT#5 FLT#N 20

21 How to Publish a new FL Paper? Datasets Results New FLT FLT#1 FLT#2 FLT#3 FLT#4 FLT#5 FLT#N No Implementation 21

22 How to Publish a new FL Paper? Datasets Results New FLT FLT#1 FLT#2 FLT#3 FLT#4 FLT#5 FLT#N No Implementation Implement from scratch 22

23 How to Publish a new FL Paper? Datasets Results New FLT FLT#1 FLT#2 FLT#3 FLT#4 FLT#5 FLT#N No Implementation Implementation Implement from scratch 23

24 How to Publish a new FL Paper? Datasets Results New FLT FLT#1 FLT#2 FLT#3 FLT#4 FLT#5 FLT#N No Implementation Implementation Different metrics Different inputs, etc. Implement from scratch 24

25 How to Publish a new FL Paper? Datasets Results New FLT FLT#1 FLT#2 FLT#3 FLT#4 FLT#5 FLT#N No Implementation Implementation Different metrics Different inputs, etc. Implement from scratch Modify the tool 25

26 How to Publish a new FL Paper? Datasets Results New FLT FLT#1 FLT#2 FLT#3 FLT#4 FLT#5 FLT#N No Implementation Implementation Implementation Different metrics Different inputs, etc. Implement from scratch Modify the tool 26

27 How to Publish a new FL Paper? Datasets Results New FLT FLT#1 FLT#2 FLT#3 FLT#4 FLT#5 FLT#N No Implementation Implementation Implementation Different metrics Different inputs, etc. Details missing Implement from scratch Modify the tool 27

28 How to Publish a new FL Paper? Datasets Results New FLT FLT#1 FLT#2 FLT#3 FLT#4 FLT#5 FLT#N No Implementation Implementation Implementation Different metrics Different inputs, etc. Details missing Implement from scratch Modify the tool guesstimate? 28

29 How to Publish a new FL Paper? Datasets Results Tedious and error prone New FLT FLT#1 FLT#2 FLT#3 FLT#4 FLT#5 FLT#N No Implementation Implementation Implementation Different metrics Different inputs, etc. Details missing Implement from scratch Modify the tool guesstimate? 29

30 How to Publish a new FL Paper? Datasets Results Tedious and error prone New FLT FLT#1 FLT#2 FLT#3 FLT#4 FLT#5 FLT#N No Implementation Is there a Implementation better way? Implementation Different metrics Different inputs, etc. Details missing Implement from scratch Modify the tool guesstimate? 30

31 How to Publish a new FL Paper? Datasets Results Tedious and error prone New FLT FLT#1 FLT#2 FLT#3 FLT#4 FLT#5 FLT#N No Implementation Is there a Implementation better way? Implementation Implement from scratch Different metrics Different inputs, etc. TraceLab Modify the tool Details missing guesstimate? 31

32 TraceLab Framework for: Creating Conducting experiments Sharing Designed to support traceability link recovery Developed by: 32

33 TraceLab 33

34 TraceLab Experiment 34

35 Components (C#, Java) TraceLab Experiment 35

36 Components (C#, Java) TraceLab Experiment Supports scripting languages: R, Python 36

37 How we Envision Publishing FL Papers with TraceLab? 37

38 How we Envision Publishing FL Papers with TraceLab? New FLT 38

39 How we Envision Publishing FL Papers with TraceLab? New FLT 39

40 Running the Experiment 40

41 Running the Experiment 41

42 How to Create a Composite Component? 42

43 How to Create a Composite Component? 43

44 How to Create a Composite Component? 44

45 How to Create a Composite Component? = 45

46 Comparing the Results 46

47 Comparing the Results 47

48 Comparing the Results 48

49 Comparing the Results 49

50 Comparing the Results 50

51 TraceLab Vision for FL Platform for creating, reproducing and sharing experiments In the future, submit a paper with a link to a TraceLab experiment: Data Code Configuration parameters Etc. 51

52 Status of TraceLab Beta version available by request TraceLab will be released in July 2012 Features under development: Packaging feature that bundles up: Data Code Settings Performance and GUI enhancements 52

53 Our Contribution Datasets for 5 systems: Features to methods mapping Execution traces Queries Corpora Reusable components and sample experiments for FL Evaluation metrics 53

54 Conclusions and Future Work TraceLab & Feature Location = Reproducible experiments Research community welcomed to share: More datasets More components and experiments 54

55 Conclusions and Future Work Adapt TraceLab for other SE areas: Impact analysis Detection of duplicate bug reports Developer recommendation At ICSE 12 TraceLab got a lot of interest t from large software groups Hopefully adaption by ICPC community 55

56 Thank you! Questions? William and Mary 56

57 Generating the jedit Datasets SVN Commits between v4.2-v4.3 57

58 Generating the jedit Datasets SVN commit message Title + Description = Query

59 Generating the jedit Datasets Changed files Previous Version Compare using Eclipse AST Current Version Modified methods (gold set) 59

60 Configuring Components 60

61 Comparing the Results (Alternative) 61