Towards a Semantic Knowledge Base on Threats to Validity and Control Actions in Controlled Experiments

Size: px

Start display at page:

Download "Towards a Semantic Knowledge Base on Threats to Validity and Control Actions in Controlled Experiments"

Anabel Baldwin
8 years ago
Views:

1 Towards a Semantic Knowledge Base on Threats to Validity and Control Actions in Controlled Experiments Stefan Biffl 1 Marcos Kalinowski 2 Fajar Ekaputra 1 Amadeu Anderlin Neto 3 Tayana Conte 3 Dietmar Winkler 1 1 Vienna University of Technology, Austria 2 Fluminense Federal University, Brazil 3 Amazonas Federal University, Brazil 2 kalinowski@acm.org

Anderlin Neto 3 Tayana Conte 3 Dietmar Winkler 1 1 Vienna University of Technology,

2 Introduction Validity is a property of inferences and every experiment faces Threats to Validity (TTVs). Risk Management Steps Risk Identification Risk Assessment Risk Mitigation Risk Monitoring Risk Control Intensity of Risk Management Tasks in Experiment Process Risk Assessment Risk Monitoring Experiment Process Steps Experimt Definition Experimt Planning Experimt Operation Analysis & Interpretation Presentation & Packaging Examples of typical experiment risk elements Subject selection, Subject characteristics, Subject groups Process, Pretest, Training, Treatment, Replication Artifact, Material, Environment, Instrumentation Cause/effect, Metrics, Observation, Data collection Hypothesis, Statistical test 2 We assume that experiment planners can benefit from a structured overview on TTVs and control actions collected from a sufficiently wide range of actual experimental SE research.

Experiment Process Steps Experimt Definition Experimt Planning Experimt Operation Analysis & Interpretation Presentation & Packaging Examples of typical experiment risk elements Subject selection,

3 Introduction Context and challenges: 3 (1) No semantic access based on domain concepts. Risk of missing important TTVs related to their experiment plan. (2) SLRs publish static reports. There is no way for systematically integrating evidence and sharing it with other researchers to reuse and extend.

Risk of missing important TTVs related to their experiment plan.

4 Introduction A semantic knowledge base (KB) on TTVs and control actions could provide access using domain concepts, helping to identify: Relevant TTVs related to their research context; and Information on how TTVs have been addressed. A good TTV KB should be flexible enough to allow adding new evidence and the analysis of data considering SE research areas and domain concepts. 4

and Information on how TTVs have been addressed.

5 Research Issues RI-1: TTV KB Stakeholder Needs. The starting point for developing a TTV KB is: What are the most relevant needs and requirements for a TTV KB prototype from the viewpoint of EMSE process stakeholders? RI-2: TTV KB Design. Which data elements are necessary to address the most relevant queries from EMSE stakeholders on TTVs and control actions? RI-3: TTV KB Content Analysis. How can the TTV KB content bring new insights into the use of TTVs in SE experiments compared to the available textbooks? 5

viewpoint of EMSE process stakeholders? RI-2: TTV KB Design.

6 RI-1: TTV KB Stakeholder Needs We identified stakeholder queries in an informal survey with 10 EMSE experts from six different research groups. Three most relevant Stakeholder Queries (SQ) SQ1: Which research areas have been addressed by the experimental studies reporting TTVs? SQ2: Which TTVs have been reported in a given research area (filtered by TTV type and experiment domain concept)? SQ3: Which control actions have been reported for a specific TTV? Other requirements Query interface (via a web user interface); Data import/export interface (e.g., via spreadsheets). 6

SQ2: Which TTVs have been reported in a given research area (filtered by TTV type and experiment domain concept)?

7 RI-2: TTV KB Design 7 The TTV KB prototype was implemented with semantic technologies using the Protégé framework. Interfaces: A web prototype for querying. An interface to import data from spreadsheets. Glossary to add synonyms for domain concepts.

Interfaces: A web prototype for querying.

8 RI-2: TTV KB Design To populate the TTV KB, we used a TTV SLR spreadsheet containing data on 206 experiments, their related publications, TTVs, and control actions. Data extensions to address the requirements: SWEBOK Chapters and BOK Topics related to each of the 206 experiments; Experiment domain concepts related to each TTV and control action; Generic TTVs reported by (Wohlin et al., 2012) (most-cited TTV identification source, cited as source by 47 experiments); and Mappings between the specific and generic TTVs. 8

Data extensions to address the requirements: SWEBOK Chapters and BOK Topics related to each of the 206 experiments; Experiment

9 RI-2: TTV KB Design 9

10 RI-3: TTV KB Content Analysis The TTV KB content analysis showed a high variation in the number of specific TTVs reported in an experiment. There are only a few SE research areas with a significant number of experiments reporting TTVs. Most SWEBOK chapters are associated to five or less experiment reports. 10

There are only a few SE research areas with a significant number of

11 RI-3: TTV KB Content Analysis 11 Even in research areas with a medium-sized set of experiment reports, this set discusses a wide range of specific TTVs, which can be useful to consolidate. Topic UML associated to just 15 experiments still covers 64% of the specific TTVs and 75% of the generic TTVs found in all 206 experiments. A small set of TTVs has been reported in many experiments and the relevance of TTVs may vary notably for different research areas. More than half of the 206 experiment reports discussed specific TTVs that could not conservatively be mapped to generic TTVs in the most-cited textbook. Five of the generic textbook TTVs were not directly reported in any of the 206 experiment reports.

A small set of TTVs has been reported in many experiments and the relevance of TTVs may vary notably for different research areas.

12 Discussion and Conclusion Experiment planners may benefit from a structured overview on TTVs and control actions collected from actual experimental SE research Complementing other valuable sources for identifying TTVs, such as guidelines and generic checklists. We defined a strategy of providing such overview by introducing a semantic TTV KB populated with data from a TTV SLR. Concerning the TTV KB content quality: the SLR s primary studies were selected based on strict quality criteria; and data extensions (e.g., mapping specific to generic TTVs) were added using a peer-reviewed process. 12

We defined a strategy of providing such overview by introducing a semantic TTV KB populated with data from a TTV SLR.

13 Discussion and Conclusion The resulting TTV KB is extensible and available online. Enables experts in the EMSE community to incrementally improve its content for their research areas. Future work: Integrate other experiment-related data (e.g., experiment design, hypotheses, and response variables) into the KB, significantly extending the querying capabilities. Provide collective intelligence functions to foster community involvement for frequent updates and quality assurance. 13 Software engineering requires a community supported living experience base V.R. Basili, A personal perspective on the evolution of empirical software engineering In: J. Munch and K. Schmid (editors), Perspectives on the Future of Software Engineering, pp Springer, 2013.

Provide collective intelligence functions to foster community involvement for frequent updates and quality assurance.

14 Towards a Semantic Knowledge Base on Threats to Validity and Control Actions in Controlled Experiments Stefan Biffl 1 Marcos Kalinowski 2 Fajar Ekaputra 1 Amadeu Anderlin Neto 3 Tayana Conte 3 Dietmar Winkler 1 1 Vienna University of Technology, Austria 2 Fluminense Federal University, Brazil 3 Amazonas Federal University, Brazil

Anderlin Neto 3 Tayana Conte 3 Dietmar Winkler 1 1 Vienna University of

Building Empirical Software Engineering Bodies of Knowledge with Systematic Knowledge Engineering

Building Empirical Software Engineering Bodies of Knowledge with Systematic Knowledge Engineering Stefan Biffl a, Marcos Kalinowski b, Fajar Ekaputra a, Estefanía Serral a, Dietmar Winkler a a CDL-Flex,