Identification and Analysis of Combined Quality Assurance Approaches

Transcription

1 Master Thesis Software Engineering Thesis no: MSE-2010:33 November 2010 Identification and Analysis of Combined Quality Assurance Approaches Vi Tran Ngoc Nha School of Computing Blekinge Institute of Technology SE Karlskrona Sweden

2 This thesis is submitted to the School of Engineering at Blekinge Institute of Technology in partial fulfillment of the requirements for the degree of Master of Science in Software Engineering. The thesis is equivalent to 24 weeks of full time studies. Contact Information: Author: Vi Tran Ngoc Nha Address: Kaiserslautern, Germany External advisor(s): Frank Elberzhager Fraunhofer Institute for Experimental Software Engineering, Kaiserslautern, Germany External advisor(s): Prof. Dr. Dr. h. c. Dieter Rombach Executive Director, Fraunhofer IESE, Kaiserslautern Germany University advisor: Dr. Tony Gorschek Blekinge Institute of Technology School of Computing Blekinge Institute of Technology SE Karlskrona Sweden Internet : Phone : Fax : ii

3 ABSTRACT Context: Due to the increasing size and complexity of software today, the amount of effort for software quality assurance (QA) is growing and getting more and more expensive. There are many techniques lead to the improvement in software QA. Static analysis can obtain very good coverage while analyze program without execution, but it has the weakness of imprecision by false errors. In contrast, dynamic analysis can obtain only partial coverage due to a large number of possible test cases, but the reported errors are more precise. Static and dynamic analyses can complement each other by providing valuable information that would be missed by using isolated analysis technique. Although many studies investigate the QA approaches that combine static and dynamic QA techniques, it is unclear what we have learned from these studies, because no systematic synthesis exists to date. Method: This thesis is intended to provide basic key concepts for combined QA approaches. A major part of this thesis presents the systematic review that brings details discussion about state of the art on the approaches that combine static and dynamic QA techniques. The systematic review is aimed at the identification of the existed combined QA approaches, how to classify them, their purposes and input as well as introduce which combination is available. Result: The results show that, there are two relations in the combination of static and dynamic techniques such as integration and separation. Besides, the objectives of combined QA approaches were introduced according to QA process quality and product quality. The most common inputs for combined approaches were also discussed. Moreover, we identified which combination of static and dynamic techniques should or should not be used as well as the potential combination for further research. Keywords: Quality assurance, Static analysis, Dynamic analysis, Testing, Inspection, Systematic review.

4 ACKNOWLEDGEMENTS This thesis is performed with the sincere help and contributions of several people. I would like use this opportunity to express my gratitude to them. It is a pleasure to thank my thesis supervisor Frank Elberzhager for invaluable guidance and support throughout the thesis. His encouragement and advice developed my inspiration and understanding of the topic. I am also thankful to my EMSE co-supervisor Dr. Tony Gorschek from Blekinge Institute of Technology for his review and important comments. Moreover, I would like to thank the library staffs of Fraunhofer Institute for Experimental Software Engineering and Blekinge Institute of Technology for their help to get useful papers and books. Lastly, I am deeply grateful to my family and EMSE friends for their support, encouragement and always being with me through all the challenges I faced. ii

5 CONTENTS 1 INTRODUCTION MOTIVATION AIMS AND OBJECTIVES RESEARCH QUESTIONS EXPECTED OUTCOMES STRUCTURE OF THESIS TERMINOLOGIES BACKGROUND SOFTWARE QUALITY QUALITY ASSURANCE STATIC QA TECHNIQUES Inspection Walkthrough Static code analysis DYNAMIC QA TECHNIQUES COMBINED QUALITY ASSURANCE APPROACH RELATED WORK RESEARCH METHODOLOGY BRIEF DESCRIPTION OF RESEARCH METHODOLOGY RESEARCH METHODOLOGY IN THE CONTEXT OF THIS TOPIC MAPPING BETWEEN RESEARCH METHODOLOGIES AND RESEARCH QUESTIONS SYSTEMATIC REVIEW PLANNING AND CONDUCTING SYSTEMATIC REVIEW PROCESS Planning the review Conducting the review Reporting the review PLANNING THE REVIEW The need of systematic review Defining research question Review protocol Study quality criteria Data extraction strategy Synthesis of the extracted data Review protocol evaluation CONDUCTING REVIEW Primary study selection Data extraction Study quality assessment SYSTEMATIC REVIEW RESULT CHARACTERISTIC OF PRIMARY STUDIES Publication year Publication channel RESEARCH RESULTS SQ1. What are existing approaches that combine static and dynamic quality assurance techniques? SQ2. Which kinds of evidence is there that support the combined quality assurance (CQA) approaches? SQ3. What are the objectives of CQA approaches? SQ4. Which CQA approaches exist? What are the common static techniques and dynamic techniques used in CQA approaches? SQ5. Which input is used for static techniques in CQA approaches? iii

6 7 THREATS TO VALIDITY INTERNAL VALIDITY EXTERNAL VALIDITY CONCLUSION SUMMARY OF FINDINGS SQ1. What are existing approaches that combine static and dynamic quality assurance techniques? SQ2. Which kinds of evidence is there that support the combined quality assurance (CQA) approaches? SQ3. Which CQA approaches exist? What are the common static techniques and dynamic techniques used in CQA approaches? SQ4. Which CQA approaches exist? What are the common static techniques and dynamic techniques used in CQA approaches? SQ5. Which input is used for static techniques in CQA approaches? INTERPRETATION FUTURE WORK Create a framework to select suitable quality assurance strategy Extension of the systematic review Conduct a systematic review for quality assurance effort reduction approaches BIBLIOGRAPHY APPENDIX A Test case prioritization (TCP) Test case reduction (TCR) Automated testing Fault-proneness prediction Combined quality assurance approach APPENDIX B APPENDIX C APPENDIX D iv

7 LIST OF FIGURES Figure 1: Thesis structure... 3 Figure 2: Five steps of Inspection... 8 Figure 3: Technical dimensions of software inspection... 9 Figure 4: Five steps in walkthrough Figure 5: The two basic dimensions of test techniques Figure 6: Systematic review process Figure 7: Review protocol Figure 8: Selection of primary studies Figure 9: Distribution of studies by publication year Figure 10: Distribution of studies by publication channel Figure 11: Classification of Combined Quality Assurance approaches Figure 12: Classification based on type of study Figure 13: Distribution of studies by year Figure 14: Type of study- Direct Investigation Figure 15: Type of Empirical study Figure 16: Study objectives QA process quality Figure 17: Study Objectives - Product quality Figure 18: Static techniques Figure 19: Dynamic techniques Figure 20: Input of static techniques v

8 LIST OF TABLES Table 1: Terminologies used in the thesis report... 4 Table 2: Research methodology mapping Table 3: Question structure Table 4: Search string attributes Table 5: Search string formation Table 6: Quality checklist Table 7: A sample of data extraction form Table 8: Quality checklist summary Table 9: List of publication channel Table 10: Static techniques Table 11: Dynamic techniques Table 12: Combination of static and dynamic techniques in CQA Table 13: QA effort reduction approaches vi

9 1 INTRODUCTION 1.1 Motivation Nowadays software system is playing an increasingly important role in our society. Since software systems are growing up in both size and complexity, developing high quality and reliable systems is becoming more challenging and more expensive. Building software products on time, within budget and with highest quality is the demand in the area of software quality assurance [6]. The significant cost for software development is for quality assurance activities. Studying the techniques that lead to improvement of quality assurance or testing effectiveness will helps to reduce cost and effort. Combined QA that use both static and dynamic QA techniques is one of the approaches. This master thesis will explore the quality assurance (QA) approaches that combine static and dynamic QA techniques. Except from the goal of reducing cost and effort, the combined QA approaches which concern about other goals in literature are also taken into account. There are many techniques used to improve testing efficiency in software quality assurance. Quality assurance techniques are divided into 2 types: static and dynamic QA techniques. Static QA techniques is a technique which help to check the software without executing the system while dynamic QA techniques test the system by running it [5,6]. However, most of existing approaches is often performed static QA techniques separately from dynamic QA techniques that often lead to the waste of effort in software quality assurance. Combining static and dynamic QA techniques by using both static QA techniques and dynamic QA techniques is one of the approach that can helps tackle this problem [8]. There are already significant empirical studies discussing about testing effectiveness and quality [1,3,4,7] but no systematic review have been done for summary and classification of all the combined approaches. In this thesis the state of art about combined software quality assurance will be presented. In details, the existing approaches for combining static and dynamic QA activities will be identified and analyzed. Later, a classification of these approaches will be provided. This thesis will cover the empirical research in this area. The state of the art will be examined on the principles of a systematic literature review from Kichenham [9] to gain a solid overall impression. 1.2 Aims and objectives The aim of this master thesis is to study combined quality assurance (QA) approaches focusing on combining static and dynamic QA techniques. In order to achieve this target, the following tasks need to be done: Identify the quality assurance approaches that combine or integrate static and dynamic QA techniques. 1

10 Identify the objective of combined approaches. Identify which static QA techniques are used in combined approaches. Identify which dynamic QA techniques are used in combined approaches. Identify the input of combined approaches. Analyze and synthesize the result of these studies. Classify those techniques based on the manner and behavior of techniques Point at gaps in current knowledge regarding combined quality assurance approach 1.3 Research questions In the following, a set of research questions is listed. The major research question (RQ) is divided into several sub questions (SQ). Answering the SQs allows us to answer the main research question RQ. RQ: Which software quality assurance approaches / methods / techniques that combine static and dynamic quality assurance techniques? SQ1. What are existing approaches that combine static and dynamic quality assurance techniques? SQ2. Which kinds of evidence is there that support the combined quality assurance (CQA) approaches? SQ3. What are the objectives of CQA approaches? SQ4. Which CQA approaches exist? What are the common static techniques and dynamic techniques used in CQA approaches? SQ5. Which input is used for static techniques in CQA approaches? 1.4 Expected outcomes The expected outcome is a thesis report which contains three main aspects. Firstly, this report will introduce the available combined quality approaches that combine static and dynamic QA techniques in literature. Secondly, the objectives and environment factors related to the approaches will be identified and analysis. Finally, the classification of the reviewed approaches will be proposed by following the principles of a systematic literature review. 2

11 1.5 Structure of thesis Figure 1 shows the overall structure of this thesis. The content is logically divided into three main parts, namely background research, research design and research contribution. Figure 1: Thesis structure In Background Research the reader is equipped with the essential information to follow the topics in the following parts. Chapter 1 (Introduction) present briefly the topic area, the necessity of the work and research questions. Chapter 2 (Background) introduces the software quality, software quality assurance, quality assurance techniques, static and dynamic, testing and inspection. The chapters in Background Research can be skipped if the reader is already familiar with these topics. Chapter 3 (Related Work) presents a brief summary of the related work regarding the combined quality assurance approaches. In Research Design the implementation details of the conducted research are presented. Chapter 4 (Research Methodology) depicts the strategy how the stipulated research questions will be answered. Furthermore, the design of the systematic review is illustrated in Chapter 5 (Systematic Review Planning and Conducting). It includes a detailed review protocol which aims to add traceability to the review and to support a possible replication in the future. In Research Contribution the original work of this thesis is presented. Chapter 6 (Systematic review results) analyses the gathered information through the systematic review and presents the findings. In Chapter 7 (Threats to validity), validity threats regarding the research work are discussed. The thesis work closes with Chapter 8 (Conclusion and Future work). 3

12 1.6 Terminologies In this work, there are several terms which are used frequently. Table 1 provides the short definition of terminologies used in the thesis report. The background details of each terminology are introduced in Chapter 2. Table 1: Terminologies used in the thesis report Term Quality assurance Static quality assurance techniques Dynamic quality assurance techniques Combined quality assurance Evidence Systematic Literature review Definitions Quality assurance involves evaluating overall project performance to provide the confidence that the project will satisfy the relevant quality standards [13]. The objective of quality assurance activities is to quantify the quality of software artifacts and providing guideline for quality improvement. Static techniques contain all techniques that do not need to run the software but operate on the code and/or other documents to observe their properties, mainly detect their defects. Dynamic analysis is the process of examining the properties of a program based on its behavior during runtime and requires program execution. Static and dynamic QA techniques usually are applied in QA activities separately and replaceable for each other. Combined QA is all the approaches that use both static and dynamic for defect detection. Type of study that primary studies conduct. A means of identifying, evaluating and interpreting all available research relevant to a particular research question, topic area, or phenomenon of interest [23]. 4

13 2 BACKGROUND This chapter provides the background of frequently used term throughout this report as well as the background of related concepts, e.g. software quality, software quality assurance, quality assurance techniques, static and dynamic quality assurance, testing and inspection. 2.1 Software quality Software quality is an open topic for discussion and hence, there are many definitions of software quality in practitioner community. IEEE standard defines software quality as: (1) The degree to which a system, component, or process meets specified requirements. [10] (2) The degree to which a system, component, or process meets customer or user needs or expectations. [10] In other definition, Software quality is seen as measurement of how well software is designed and how well the software conforms to that design [11]. Both definitions highlight the role of quality of requirement (how to build a right software) and quality of implementation (how to build a software rightly). Therefore, in order to ensure the quality of software, it is concerned two aspects, quality assurance of requirement (validation) and quality assurance of implementation (verification). The software quality can be expressed by quality attributes, those are a feature or characteristic that affects an item s quality [10]. To best of our knowledge, there is no generally accepted set and description of quality attributes but a well-known set is defined in the standard ISO 9126 [12]: Functionality Reliability Usability Efficiency Maintainability Portability 2.2 Quality assurance Quality assurance (QA) is defined by the IEEE in [10] as following: (1) A planned and systematic pattern of all actions necessary to provide adequate confidence that an item or product conforms to established technical requirements. 5

14 (2) A set of activities designed to evaluate the process by which products are developed or manufactured. Quality assurance involves evaluating overall project performance to provide the confidence that the project will satisfy the relevant quality standards [13]. Quality assurance can be focused on the product or the process. In this work, we focus on the first aspect of quality assurance. The objective of quality assurance activities is to quantify the quality of software artifacts and providing guideline for quality improvement. Quality assurance differs from quality control in that quality control is a set of activities, which focus on evaluating quality of developed product. The process is conducted during or after the production of product. Meanwhile, quality assurance focuses on reducing the cost of ensuring quality by a variety of activities performed throughout the development and manufacturing process. Therefore, the main objective of quality assurance is cost effectiveness of QA activities. There are various kind of QA techniques and methods used to quarantine software quality, i.e., preventing defects or detecting exiting defects. The QA activities that focuses on detecting exiting defects is also called Verification and validation, which is defined by the IEEE [10] as follows: Verification and validation is the process of determining whether the requirements for a system or component are complete and correct, the products of each development phase fulfill the requirements or conditions imposed by the previous phase, and the final system or component complies with specified requirements. We distinguish the two relevant concepts: verification and validation: Verification is the process of evaluating a system or component to determine whether the products of a given development phase satisfy the conditions imposed at the start of that phase. Validation is the process of evaluating a system or component during or at the end of the development process to determine whether it satisfies specific requirements. Within verification and validation techniques, there are two main categories, namely (1) static and (2) dynamic techniques. Static techniques contain all techniques that do not need to run the software but operate on the code and/or other documents and detect their defects. On the contrary, dynamic analysis runs the compiled software with the aim to observe failures. The main dynamic analysis method is software testing. In this report, we use the term static QA techniques and dynamic QA techniques to refer to the term static and dynamic techniques mentioned above. 6

15 2.3 Static QA techniques The popular static QA techniques are inspection, review, walkthrough and code static analysis Inspection Inspections are one of the most well-known forms of static analysis because they can detect defect in early stages of development process. The first structured software inspection method was introduced by Michael Fagan [14]. He proposed the approach of a formal inspection process, which includes five main steps as following: Overview: An overview is the beginning step of inspection process. The meeting is hold with the presence of author and the inspection team. During the meeting, the author of the produced document describes the general area of work and gives detailed information of inspected document. Preparation: Each inspector analyzes the artifact and any supporting documentation independently to detect potential defects. All potential defects are recorded. Inspection: An inspection meeting is held with all team members. One reader paraphrases the artifact. During this process, inspectors can stop the reader and raise any issue they have found, either in preparation time or at the meeting. The issue is discussed until agreement is reached. The agreed defects will be classified based on defect characteristic as well as its severity (major or minor). After the meeting, the detail report during inspection will be documented by the moderator. The author of this artifact will rework based on the report. Rework: The author of the artifact made modifications based on the meeting report to correct all found defects. Follow-up: All required alterations are ensured to be made by the moderator. The moderator then decides whether the artifact needs to be re-inspected. The decision depends on the quantity and quality of the rework. 7

16 Figure 2: Five steps of Inspection [14] According to Fagan, the results of inspections should never be used as criteria for assessment of programmers [14]. Inspections are self-improvement processes because inspection results are used as feedback for both defects removal and professional development. Inspection statistics, such as inspection rates and error density, are collected and used for determining characteristics of inspections and for process improvement as well as defect hindrance. Since then many variations on the process and the reading techniques have been proposed. We refers to the classification of inspection techniques in [15]. The work uses four dimensions to classify inspection techniques, which are: Technical dimension: methodological variations Economic dimension: economic effects on the project and vice versa Organizational dimension: effect on the organization and vice versa Tool dimension: support with tools The process of an inspection typically consists of the phases planning, overview, detection, collection, correction, and follow up. In the planning phase, a particular inspection is organized when artifacts pass the entry criteria to the inspection. We need to select the participants, assign them to roles and schedule the meetings. The overview phase sometimes called kickoff meeting is not compulsory and the literature differs in this respect. This phase is intended to explain the artifact and its relationships to the inspectors. This can be helpful for very complex artifacts or for early documents such as requirements descriptions. The next phase defect detection is the main part of the inspection. This is often divided into 8

17 preparation and a meeting. How the details of this phase look like differs in the literature. Some authors suggest that already the preparation part should be used for defect detection while others propose to only try to understand the artifact and do the actual defect detection in the meeting part. Besides the difference in the steps, sequences of processes of inspections, the other difference lies in the use of reading techniques. There are many available reading techniques used in inspection, such as perspective based reading, checklist based reading, defect based reading and etc Walkthrough Figure 3: Technical dimensions of software inspection [15] A structured walkthrough is an organized procedure for a group of people to review and discuss the technical aspects of software development and maintenance work products [16]. The major objectives of a structured walkthrough are to find errors and to improve the quality of the product. The basic purpose of a walkthrough is error detection, not error correction. When the walkthrough is completed, the author of the work product is responsible for taking the necessary actions to correct the errors. Walkthrough is considered as a simplified version of formal inspection process. Each participant in the structured walkthrough process has a specific role. For a small size project, a person may have multiple roles. These roles are; coordinator, producer, reviewer and scribe. The coordinator contacts participants, prepares and distributes documentation, and selects a schedule for the walkthrough. Participants spend time preparing for the walkthrough by examining the product. The walkthrough, which is a group activity, is the central component of the review process where all participants meet and discuss or comment on the product. Then, a list of the comments is made and passed to the producer. This list is used to make changes in the product. A final management summary is also produced by the coordinator. Feedback is not used to control the development process. Walkthrough process and its participants are given in Figure 2. 9

18 2.3.3 Static code analysis Figure 4: Five steps in walkthrough [16] Static code analysis is a method that aims to find defects in code without actually executing programs. The analysis is often performed with supports of software tools. There are various techniques of static code analysis, which can be classified into four main categories, namely data flow analysis, control flow analysis, program slicing and formal methods. Data flow analysis: to detect ineffective code and identify data flows that is not suit to practice such as variable are written before they are read. It is also known as symbolic form of analysis without considering specific data values. Control flow analysis: translate the program into a flow graph to detect poorly structured code and identify inaccessible code and non-termination classes. Program slicing: A slice is the set of program statements that affected a given value [11]. Program slicing focuses on a particular subset of variables involving in a given part of the program. Program slicing improves re-testing by selecting test case for regression tests or improve program comprehension. Formal methods: Formal methods use mathematical logic and reasoning (proof) to establish correctness and point out the gap between abstract designs and concrete code. Formal methods can be applied to design specification and code. Model checking and symbolic analysis are common types of formal methods. 10

19 2.4 Dynamic QA techniques The main dynamic analysis is software testing. There are many definitions about testing, few of them are: 1. the process of using a system or component under given circumstances, of observing or noting the results, and carrying out an evaluation of the system or component from a given perspective [17] 2. the process of executing a program with the purpose of detecting defects [18] 3. is questioning a product in order to evaluate it [19] Software testing also provides an objective, independent view of the software to allow the business to appreciate and understand the risks at implementation of the software. There are various possibilities to classify different test techniques. We base our classification on standard books about testing [18, 20] and the classification in [2]. One can identify at least two dimensions to structure the techniques: (1) The granularity of the test object and (2) the test case derivation technique. Fig. 2.2 shows these two dimensions and contains some concrete examples and how they can be placed according to these dimensions. The types of test case derivation can be divided on the top level into (1) functional and (2) structural test techniques. The first only uses program specification to generate test cases, whereas the latter relies on the source code and the specification. In functional testing generally techniques such as equivalence partitioning and boundary value analysis are used. The tested component is viewed as a black box, and their behaviors are determined by studying its inputs and associated outputs [13]. Structural testing is often divided into control-flow and data-flow techniques. Control flow testing techniques require understanding of source code. This family studies a series of paths throughout the program, thereby examining the program control model. Data flow testing aims at examining program paths and sequence events, which consider the data state, number and type of uses of variables. On the granularity dimension, we often see the phases unit, module or component test, integration test, and system test. In unit tests, only basic components of the system are tested using stubs to simulate the environment. Unit testing includes white box (clear box or glass box), black box and gray box. The white-box approach generates test cases according to the information derived from the source code of the software under test. White box use control flow testing and data flow testing when the source code is already developed. The black-box approach, on the other hand, generates test cases from information derived from the specification without understanding about the software structure. the classification-tree method (CTM) 11

20 During integration tests the components are combined and their interaction is analyzed as a group. It occurs after unit testing and before system testing. Integration testing takes as its input modules that have been unit tested, groups them in larger aggregates, applies tests defined in an integration test plan to those aggregates, and delivers as its output the integrated system ready for system testing [21]. Finally, in system testing the whole system is tested, often with some similarity to the later operational profile. The system testing is normally conducted on a complete, integrated system to evaluate the system's compliance with its specified requirements. The testing techniques using the scope of black box testing, and as such, should require no knowledge of the inner design of the code or logic. Besides, there are some special types of testing either with a special purpose or with the aim to simplify or automate certain parts of the test process. Random testing technique generates random test cases without following any prioritization or pre-established guidelines but use a stochastic approach to cover the input range. Mutation operators base mutation testing techniques on modeling typical programming faults, which is dependent on the programming language. Each mutation operator can applied to generate several mutants (programs that are the same as the original program except one modified sentence by the mutation operator). Model-based testing i.e. uses explicit behavior models as basis for test case derivation, possibly with an automatic generation. Figure 5: The two basic dimensions of test techniques [15] 2.5 Combined quality assurance approach Static and dynamic QA techniques usually are applied in Quality Assurance (QA) activities separately and replaceable for each other. Some of studies have shown the advantages to combine static and dynamic QA techniques. To get an overview picture of combined QA approaches and to reason about the performance of the reviewed approaches, a systematic review is conducted. Literature provides various examples of combined approaches with varying methodologies. The aim of this study is to identify the 12

21 different approaches and to put them in a systematic context. The main outcomes from this review are to find out the methods that combined quality assurance activities and their purpose. Apart from that, it is also subject of interest to know the context of these methods, i.e. to see what is the focus of the reviewed approaches, the requirement to use them as long as their performance. 13

22 3 RELATED WORK A preliminary study on the literature indicates that no systematic reviews on combined quality assurance (CQA) approaches have been conducted before. However, several studies which are related to the area of quality assurance, especially testing and inspection are identified. Although the studies reside in the same area, they do not focus on the issues addressed in this thesis work. In 1987, Basily and Selby [77] conducted a control experiment to compare three software testing techniques, e.g. code reading, functional testing and structural testing. The comparison strategy is based on three aspects, e.g. fault detection effectiveness, fault detection cost and class of detected fault. The subjects of this study are 32 professional programmers and 42 advance students. The result shows that code reading and functional testing is both superior than structural testing. Code reading detected more interface faults and functional testing detected more control faults. For the purpose of estimation the percentage of faults detected, code readers gave the most accurate estimates while functional testers gave the least accurate estimates. This experiment is considered as the milestone for comparing testing techniques and giving motivation for using both code reading and testing techniques to detect different types of defects. Oliver Laitenberger [78] in 1998 contributes a controlled experiment to characterize the effects of code inspection and structural testing on software quality. In this experiment, C-module is test with 20 subjects. The result shows that inspection significantly outperforms the defect detection effectiveness of structural testing as well as inspection and structural testing are not complemented each other well. Then, the suggestion is given that inspection should be used along with other testing techniques, such as boundary value analysis, to achieve a better defect coverage. This experiment is a starting point and encourages other researchers to investigate the optimal mix of defect detection techniques. A study by Per Runeson, Carina Andersson, and Thomas Thelin entitled What do we know about defect detection methods provides the survey about empirical studies which compare inspection and testing. They summarize 12 studies nine experiments on code defects, one experiment on design defects, and two case studies on detection process to search for evidence that would help developers choose between defect detection methods. The comparison data about average values of effectiveness and efficiency for defect detection is also given. This summary of the empirical studies can be considered as a general guide for which defect detection methods to use for different purposes and at different development stages. Although it mentions strategy of selecting defect detection techniques but this result is not novel or strictly empirically based. Furthermore, the study does not provide overall picture of CQA approaches. This thesis is intended to identify CQA approaches that use both static and dynamic techniques for defect detection and put them in a systematic manner. This thesis will explore the objective and procedure of 14

23 each approach in more details. Furthermore, the common combination of static and dynamic techniques as well as the potential combination for further research is also discussed. 15

24 4 RESEARCH METHODOLOGY To answer the research questions in this topic, two research methods were used: literature review and systematic literature review. Before describing how each of these research methods is used, their definition and process are introduced in Section 4.1. Section 4.2 provides a flow chart to illustrate how the chosen research methods were applied in this study to answer the research questions. In Section 4.3, every research questions is mapped to research method which is used to answer that research question. 4.1 Brief description of Research Methodology This thesis conducts a systematic review on combined quality assurance approaches. The brief definition and the process of systematic review is given as following. According to Kitchenham et al., Systematic review (also referred as a systematic literature review) is considered a secondary study which reviews all studies related to a specific research topic, to answer the research question by identifying, analyzing and interpreting all relevant evidence [23]. Systematic review is performed by following a well-defined methodology that is unbiased and (to some extent) repeatable. The most common reasons for performing a systematic review are: to summarize the existing evidence of the research question, to identify a gap in current research and to provide a background for new research activities [23]. Kitchenham recommend three phases to do systematic review: planning the review, conducting the review and report the review. In first phase, the most important steps are to define the research question and develop a review protocol. Review protocol is the procedure and purpose to do the systematic review [24]. The review protocol helps to reduce the possibility of researcher bias as well as for researcher to focus on their research goals. In conducting review phase, the main activities are selecting the primary studies based on defined search strategy, accessing the quality of the studies, extracting data from the studies for further analysis and synthesizing data to draw final answer for the research question. And then, in the last phase, the review is documented in an appropriate format and evaluates the report to ensure the validity and quality of the report. 4.2 Research Methodology in the context of this topic To answer the research question, a systematic review was conducted. All the steps of the systematic review were performed with the consideration of these aspects, from forming the research questions, design the review protocol in planning phase, select related primary studies, and extract and synthesis 16

25 data from these studies in the conducting phase, as well as document the results in the reporting phase. This method incorporates the data extraction strategy of the systematic review so that the data can be collected in a systematic way and is usable for later use in conceptual analysis. According to the findings from the data synthesis, the categorization and generalization are proposed to do the classification of combined quality assurance approaches. The detail description of each phase for this systematic review is provided in the following chapter. Chapter 5 provides the planning and conducting for the review and chapter 6 is the summary and synthesis of the result. 4.3 Mapping between Research methodologies and research questions Table 2 describes mapping between research methodologies and research questions to explain the purpose for each research question and which research methodology is applied to answer it. Table 2: Research methodology mapping Research Question RQ. Which software quality assurance approaches / methods / techniques that combine static and dynamic quality assurance techniques? SQ1. What are existing approaches that combine static and dynamic quality assurance techniques? SQ2. Which kinds of evidence is there that support the combined quality assurance (CQA) approaches? SQ3. What are the objectives of CQA approaches? Rationale To explore the combined QA approaches that combines static and dynamic QA techniques. Detailed steps followed to answer this question are elaborated in RQ2.1 to RQ2.6. To explore which existing combined quality assurance approaches that are available in literature To identify the evidence or type of study for each approaches To identify the goal and detail objective for each CQA approaches. Research Methodology Systematic review Systematic review Systematic review SQ4. Which CQA approaches exist? To identify which static QA Systematic 17

26 What are the common static techniques and dynamic techniques used in CQA approaches? SQ5. Which input is used for static techniques in CQA approaches? techniques used in CQA approaches and which dynamic techniques used in CQA approaches. To explore which combination of static and dynamic QA techniques are available or which combinations are potential to be performed. To identify which input static QA techniques use in CQA. review Systematic review 18

27 5 SYSTEMATIC REVIEW PLANNING AND CONDUCTING The systematic review process consists of three phases. Those phases are: plan the review, conduct the review and document the review (Figure 6). The execution of each phase requires iteration, feedback and modification to achieve acceptable results. Each of these phases contains a sequence of stages. The general information about systematic review procedure is explained in section 5.1. Section 5.2 is the design of this systematic review and section 5.3 is the details of systematic review conducting. 5.1 Systematic review process Figure 6 illustrates in detail three phases of a systematic review applied in this thesis Planning the review The first phase is planning the review. In this phase the plan of the complete process is made. There are 4 main steps in this phase. Identify the needs for a systematic review: In the beginning, researchers need to investigate whether a systematic review in the same research area has already been conducted before or not. The evidences will be studied for deciding whether the systematic review is still needed and to which extent. Specifying research questions: The research questions of the systematic review are defined in this step. Primary studies identified in this systematic review and the final result must be able to answer these research questions. Development of the review protocol: In this step the review protocol is developed. It describes the details and methods with which the entire systematic review is conducted. A well-documented review protocol ensures that the systematic review is repeatable. Evaluating the review protocol: This step is to ensure that the review protocol has fulfilled certain quality criteria and the review is unbiased. The researcher without involving in developing the protocol will inspect, evaluate and provide feedback to refine the protocol until its quality is satisfied Conducting the review The following phase is conducting the review. There are five steps that are required to be executed in sequence. 19

28 Perform the search: The resource for searching such as digital libraries, journals, and proceedings to conferences, are identified. The search strategy is applied on these resources to collect potential primary studies. Selection of primary studies: In this step, all studies extracted from the previous step are selected according to inclusion and exclusion criteria. Evaluate quality of primary studies: This step is about assessing the quality of the selected studies from the previous stage based on quality checklist or questions defined in the research protocol. Extract necessary data: In this step, the necessary in the selected studies is extracted for further analysis. Analysis and synthesis data: The gathered information in the extraction forms is analyzed and synthesized to answers the research questions Reporting the review The final phase is reporting the review which mainly focus on prepare the results of the systematic review in an appropriate format. The systematic review guideline explains that there are two stages in this phase. Formatting the main systematic review report: There are two type of formats can be used for reporting a systematic review: technical report or PhD/ Master thesis and journal or conference paper. Evaluating the systematic review report: The report should be evaluated for ensuring its quality and validity. 20

29 21

30 5.2 Planning the review The need of systematic review Figure 6: Systematic review process To identify the need of this systematic review, a preliminary search on the topic was performed. The search tried to answer those questions: i. Were systematic reviews regarding combined quality assurance approaches already existed and to which extend? ii. Were literature reviews regarding combined quality assurance approaches already existed and to which extent? To answer these questions, searching on 2 databases Compendex and Inspec is conducted. The search strings were applied on the field: Subject, Title and Abstract ("WN KY" in the search strings). The synonyms for "systematic review" used in the search string. The results obtained 89 papers but none of them answer the research question in Section 4.2. The result shows that performing the systematic review is necessary. Here is the search string used to perform preliminary search: (((("systematic review" OR "research review" OR "research synthesis" OR "research integration" OR "systematic overview" OR "systematic research synthesis" OR "integrative research review" OR "integrative review") WN KY) AND ((inspection or review or "static analysis" or "static quality assurance") WN KY)) AND ((test* or "dynamic analysis" or "dynamic quality assurance") WN KY)) Defining research question This review questions drive the entire systematic review methodology. The question structure is shown in Table 3. Following Kitchenham [23], the question structure is formed with 4 main aspects. Population is the group or team who use the concept in research question. Intervention is the software methodology/tool/technology/procedure that addresses the issue. Comparison is the software engineering methodology/tool/technology/procedure with which the intervention is being compared. Outcomes should relate to factors that has important impact to practitioner. Context is the environment in which the comparison takes place. Table 3: Question structure Population Intervention project manager, tester, developer, quality assurance team combined quality assurance approaches which combine static QA 22

31 techniques and dynamic QA techniques Comparison Outcomes Context quality assurance approaches which use static QA techniques only or dynamic QA techniques only quality assurance improvement software quality assurance RQ: Which software quality assurance approaches / methods / techniques that combine static and dynamic quality assurance techniques? SQ1. What are existing approaches that combine static and dynamic quality assurance techniques? SQ2. Which kinds of evidence is there that support the combined quality assurance (CQA) approaches? SQ3. What are the objectives of CQA approaches? SQ4. Which CQA approaches exist? What are the common static techniques and dynamic techniques used in CQA approaches? SQ5. Which input is used for static techniques in CQA approaches? Review protocol The Figure 7 describes the review procedure which is applied for this systematic review. Firstly, a literature review was performed to have certain background knowledge about methods/techniques which reduce QA effort. These approaches found during this review gave us an orientation and overview for doing a systematic review in combined QA approaches. Before systematic review was conducted, a preliminary search is performed to explore whether there is already systematic review about combined QA approaches and to which extent. This step ensured that this systematic review is necessary and not a duplicate work. After that, the research questions were defined according to the result of literature review. The search strategy including search term, data source, study selection criteria and study quality assessment are indentified before performing study searching. The search strategy was continuously modified during the evaluation until it was satisfied. The review applies search strategy to get the search result from chosen databases and resources. The study selection criteria defined in search strategy are considered to choose the relevant paper based on 23

32 title/ abstract/ full-text. Thus the selected papers resulted from the searching will be reviewed to get the insight of the topic. The important data from those chosen paper will be extracted and analyzed. Based on the extracted data, the classification of combined QA approaches is proposed and the research questions are answered. Preliminary review search The need of systematic review Define research question Research Questions DatasourceI nspec Compendex Define search strategy Search terms Missing Combine search result Develop assess strategy Evaluate search strategy Satisfied Study quality Criteria Select papers based on tiltle/ abstract/ fulltext Study selection criteria Study extracted data Review paper Data Extraction Strategy Synthesis data Classification Framework Figure 7: Review protocol 24

33 a. Search strategies In order to perform the search in this review, four databases were identified. The main two chosen databases are Inspec and Compendex. These databases are the comprehensive database that contains millions of publication, mostly in engineering and computer science domain. Another reason for choosing Compendex and Inspect is Engineering Village interface for using Compendex and Inspect provides advance search feature a user friendly interface. b. Search string The search string is formed based on the key word derived from the research question. There are four attributes in the search string. Search attributes are derived from the thesis title: Identification and analysis of Combined Quality Assurance approaches. Table 4 shows the steps that are applied to form the search string and rationale for each step. Table 5 presents the relevant terms for each search attribute. Table 4: Search string attributes Search attributes Steps Rationale Attribute 1 Synonyms for static QA techniques", different common type of "static QA To minimize the effect of heterogeneous terminology techniques Attribute 2 Synonyms of software To constraint the search in software engineering Attribute 3 Synonyms of dynamic QA To minimize the effect of techniques, different common type heterogeneous terminology of dynamic QA techniques Attribute 4 Relevant term of combine Increase the coverage of search string Table 5: Search string formation Attribute Steps Relevant terms 1 Static QA techniques inspection or review or static analysis or static quality assurance 2 Software Software 3 Dynamic QA test* or dynamic quality assurance or dynamic techniques analysis 4 Combine combin* or integrat* or synergy or trade off or 25