EMPIRICAL EVALUATION IN SOFTWARE PRODUCT LINE ENGINEERING

Transcription

1 EMPIRICAL EVALUATION IN SOFTWARE PRODUCT LINE ENGINEERING Alvin Ahnassay, Ebrahim Bagheri, Dragan Gasevic Laboratory for Systems, Software and Semantics, Ryerson University Abstract: Context: Software Product Line (SPL) engineering is a specific approach to software development that offers strategic methodologies and technologies tailored for reuse of core through the capturing of variabilities and commonalities. It is imperative to evaluate software product line engineering practices through real empirical studies and investigate how each of the proposed work contribute to research and development communities. Objectives: The objectives of this research is to paint a big picture of the empirical evaluations that have been undertaken in the field of Software Product Lines (SPL) in reference to development of methods, techniques, and approaches 1. Furthermore, we are interested in determining the quality of the empirical evaluations and how they can be improved. Methods: We carried out a systematic literature review of the software product line methods, techniques, or approaches reported from January 1, 2006 until December 31, The adopted method follows well established guidelines in the literature for conducting systematic literature reviews. Results: 79 distinct studies representing a diversity of unique software product line methods, techniques, and approaches were selected according to the inclusion and exclusion criteria. The results revealed a significant number of evaluations conducted in academia with only 25 studies conducted in an industrial setting. However, the majority of the evaluations effectively avoided scalability issues by using industrial sized examples. This investigation also revealed quality issues in various aspects of the quality assessment criteria for research design and reporting. Conclusions: This systematic literature review found that a large majority of evaluations had not been sufficiently designed or reported. This makes it difficult to rely on the available evidence. The findings highlight unresolved quality issues in empirical studies of software product lines. The need for improvement in quality research design and quality reporting is only reinforced further with the findings of this review. Keywords: Empirical Software Engineering, Software Product Line Engineering, Systematic Literature Review 1 In this the paper, the phrase methods, techniques and approaches is intended to refer to any form of soft or hard means proposed to address a challenge in the area of software product lines. 1

2 1. Introduction The diversity and complexity of stakeholders requirements and demands for high quality software systems urge the need for approaches which enable effective reusability in developing software systems. Software Product Line Engineering (SPLE) has proven to be a paradigm which provides strategic software reuse [1], allowing organizations to reduce development costs and time to market and increase product quality [36]. SPLE shifts from developing a single software system to developing a set of software systems which share a common and managed set of features [2]. Hence, SPLE can handle various stakeholders requirements within a particular domain of interest and builds new products using a stable collection of existing core assets. In software product lines, reusability is considered as a first class notion; a lifecycle is introduced for developing reusable artifacts (development for reuse) and a lifecycle is introduced for developing software systems from reusable artifacts (development with reuse) [3]. On the one hand, commonality and variability modeling are utilized in the first lifecycle (also known as domain engineering lifecycle) to document common assets and their variability. On the other hand, configuration and decision making mechanisms are employed in the second life-cycle (also known as application engineering lifecycle) to derive software systems fitted to stakeholders requirements. There are many studies in software product lines sharing similar aspects that confer various methods, techniques, and approaches of software product line engineering and the assured benefits of the practices. These studies propose new and enhanced methods, techniques, and approaches to software product line development. A common characteristic of these studies is their advantages and improvements over traditional software development and existing software product line practices. In addition, many software engineering firms put claim to the fortune of realizing the benefits of adopting and utilizing the software product line approach [36]. However, not much is known about what real evidence exists that supports the proclaimed benefits inherited by software product line engineering practices. This systematic review seeks to review and summarize empirical evidence reported between 2006 and 2011 from academia and industry on software product lines methods, techniques, and approaches. The objective of this review is to: Search and consolidate empirical evidence supporting software product line methods, techniques, and approaches evaluated in academia and industry; Assess the evaluation strategies/procedures and the quality of reporting; Summarize and analyze the results; and Recommend areas for future research. This review defines three research questions (see Section 2) that are answered based on the gathered evidence. Each research question is supported by an underpinning of sub-research questions which will help define the quality of research evaluations. Based on the nature of this research, the contribution of this review paper should be of interest to those who want to learn from experience while planning future studies on software product lines and those who seek 2

3 evidence to support their decision-making process for adopting software product line practices or to improve on their existing practices based on the accumulated results. This paper is structured as follows. Section 2 explains our motivation and the research questions that this work intends to answer. Section 3 provides an overview of the design of this systematic literature review. Section 4 discusses the execution of the review along with threats to validity. Section 5 presents and discusses the results of the review in reference to the research questions outlined in Section 2. Section 6 highlights and discusses the main findings derived from the analysis of the results and points out the recommendations for empirical studies in software product lines. After reviewing the related work and comparing our systematic review in that context in Section 7, we conclude the paper and highlight possible future work. 2. Motivation and Research Questions Systematic reviews are becoming a standard research method amongst software engineers[6],[4]. However, practitioners still are lacking in significant knowledge about this research method and the number of explored topics remains limited [4]. The deficiency in explored topics holds true in the area of software product lines and justifies a need for more systematic literature reviews of software product line methods, techniques, and approaches. Systematic reviews are performed to summarize existing evidence concerning a specific research question, topic area or phenomenon of interest, and identify areas where research is lacking [5]. Using a fair and unbiased approach, this systematic literature review is intended to summarize evidence of various software product line methods, techniques, and approaches subjected to an empirical evaluation. In addition, there is a need to identify possible holes in existing software product line evaluation methods in order to recommend further examination of empirical work in the area of software product lines and provide a context for new software product line evaluation activities. The research questions being asked in this review are designed to encapsulate what software product line methods, techniques, and approaches exist today, what their current state of evaluations are, and to what extent do the evaluations support adoption of the software product line methods, techniques, or approaches. The specific research questions that this study is addressing are: RQ1: What software product line methods, techniques, and/or approaches exist today and what is the context in which they have been proposed? RQ2: What is the state of evaluations being conducted on software product line methods, techniques, and/or approaches? RQ3: To what extent do the evaluations support the adoption of software product line methods, techniques, and/or approaches? In order to provide proper levels of details for abovementioned research questions, these questions are refined into several research questions. All research questions and their descriptions are recorded in Table 1. 3

4 Table 1: Designated research questions for the study # Research Question Description 1 What software product line methods, techniques, or approaches exist? 1.1 What problem area does the software product line method, technique or approach target? 1.2 What is the project lifecycle focus of the software product line method, technique or approach? 1.3 What project activities does the software product line method, technique or approach target? 1.4 To what degree is the evaluation of the method described? 2 What is the state of the software product line method, technique or approach evaluation? 2.1 What research methods are used in the evaluation of software product line method, technique or approach? To provide an inventory of software product line methods, techniques, or approaches evaluated. To identify the problem area targeted by the software product line method, technique or approach. To identify when the software product line method, technique, or approach is used during the software product line lifecycle. To identify the software product line development activity targeted by the software product line method, technique, or approach. To assess the quality of reporting used to describe the software product line method, technique or approach. A summary of research questions formulating the state of evaluation. Identify the type of research method used to evaluate the software product line method, technique or approach. For example, experiments or case studies. 2.2 In what context is the software product line method, technique or approach evaluated in? 2.3 What types of subjects are used during the evaluation of the software product line method, technique or approach? 2.4 What scale does the evaluation of the software product line method, technique or approach have? 2.5 What degree of realism does the evaluation of the software product line method, technique or approach have? 3 To what extend does the reported empirical evaluation enable practitioners to decide about the actual adoption of the software product line method, technique or approach? 3.1 To what degree is the software product line method, technique or approach evaluation described? Identify if the evaluation was performed from an industry or academic perspective. Identify the subjects used during the evaluation of the software product line method, technique or approach. For example, students, researchers, or practitioners. Identify the scale of which the software product line method, technique or approach is evaluated. Take into consideration the magnitude of the evaluation conducted. For example, can the evaluation be considered as industrial in size or a toy-example? Measurement of how realistic the evaluation of the software product line method, technique or approach is in terms of being applicable. A combination of research method, context, subjects and scale gives the realism of the evaluation. Determine how well the reported empirical evaluation enables practitioners to assess the probability or likelihood that the software product line method, technique or approach is viable enough to be adopted as a practicable approach in the software product line community. This question is influenced by the findings of RQ 3.1. The degree by which the evaluation is described can be measured by combining the context described, study design described, and the validity of the outcomes. 3. Review Method This section provides the details surrounding the review protocol developed to guide the conduct of this review. It discusses the systematic review design, data source and search strategy, study selection criteria, quality assessment criteria, data extraction procedures, and data synthesis procedures Systematic Review Design Following the guidelines described in [5], this study has been carried out according to the three main phases of systematic literature reviews: 1) Planning the review, 2) Conducting the review, and 3) Reporting the review. These phases are illustrated below: 4

5 Figure 1 Systematic Literature Review Phases The Planning Review Phase involves identifying the need for a review and developing a review protocol. Identifying the need for this study has been discussed in the previous section. This section provides a discussion on the development of the review protocol used for this study. The review protocol specifies the methods used for finding primary studies (including search terms and data sources), study selection criteria, study quality assessment criteria, and procedures for data extraction and data synthesis. The Conducting Review Phase involves identifying and selecting primary studies, assessing the quality of the primary studies, and extracting and synthesizing the information reported in the primary studies. The review execution is discussed in the next section. The Reporting Review Phase involves synthesizing the data extracted during the review execution and summarizing the results of the included primary studies. The results and analysis of this review is reported in Section 5. The extracted information is tabulated in accordance with the review questions and is supported by descriptive analysis, aggregated totals and percentages, and graphical representation. Overall, this paper follows the recommendations for structuring of reports of systematic reviews outlined in Table 9 of [5] Data Sources and Search Strategy The process of identifying relevant papers that undertake an empirical evaluation of a software product line method, technique or approach needed the capability to recover research articles and papers made available through scientific publication databases. Specific publication databases were selected on the basis that they included research papers in the area of computing and information technology sciences. These digital databases were chosen because they provide the most important and highest impact journals and conference proceedings and cover the fields of software product line engineering and software engineering [23]. Table 2 identifies and describes the digital databases chosen as adequate sources to identify research papers. Table 2: Digital Libraries used to search for primary studies Database Name ACM Digital Library IEEE/IEE Electronic Library ScienceDirect SpringerLink Description Search this database from ACM (Association for Computing Machinery) to access quality information in the scientific computing field. Includes full text articles from ACM magazines, journals and conference proceedings as well as publications from affiliated organizations. Search this database to find research materials in the areas of computing science and electrical engineering. ScienceDirect is Elsevier s digital library that is one of the world's largest providers of scientific, technical and medical (STM) literature. Access a selection of journals from Elsevier Science covering all fields of science (including journal titles from Business, Psychology, Arts and Humanities, and Social Science). Formerly known as Springer-Verlag Journals, from this site one can access hundreds of Springer-Verlag and Kluwer journal titles, with considerable full text online. Coverage includes Chemical Sciences, Computer Sciences, Economics, Engineering, Environmental Sciences, Geosciences, Law, Life Sciences, Mathematics, Medicine, Physics, and Astronomy. 5

6 Wiley Online Library Wiley Online Library, formerly known as Wiley InterScience now includes journal content formerly on Blackwell Synergy, providing access to more than 3 million articles across journals. Coverage includes business, computer science, criminology, education, health, law, mathematics, psychology, sciences and social sciences. Retrieving research papers from the above databases required a specific combination of keywords that would establish the identity of papers that suggest an evaluation of a Software Product Line method, technique, or approach was conducted. The search terms used were tested and tried to establish accuracy of paper identification and to ensure the identification process was robust enough to minimize the risk of missing important and relevant papers. This was ensured by several rounds of revision and based on the input from all of the authors of this paper. The structure of the search strategy is based on the main concept being examined in the review with the goal of finding empirical evaluations of Software Product Line methods, techniques, or approaches. Table 3 presents the search terms used to identify potentially relevant articles in the computing and information technology sciences literature. The population signifies the search for Software Product Lines as well as Software Product Families as these two terms of reference are interchangeable. The intervention signifies that an evaluation should have been performed in the research paper. Table 3: Search Terms used to find software product line literature Population AND Intervention software product line OR software product famil* AND empiric* OR experiment* OR experience* OR lesson learned OR lessons learned OR lesson learnt OR lessons learnt OR evaluat* OR validat* OR stud* OR case* OR example* OR survey* OR investigat* OR analy* OR demo* 3.3. Study Selection In addition to executing the automated search based on the search terms in Table 3, the results required further investigation to determine which papers should be included and excluded from this study. Hence, we applied a set of inclusion (see Table 4) and exclusion (see Table 5) criteria to decide proper papers for the study: Inclusion Criteria Papers where the search terms are found Table 4: Inclusion Criteria for determining the papers for the study Rationale If the main objective of the paper is to evaluate a software product line method, technique, or approach than it would most likely be mentioned in the title or abstract. Papers published from Jan to Dec Papers where the full-text is available. Papers written in English. Only interested in current and relevant evaluations conducted in recent years. The year 2006 was recommended because it is a common practice in other related studies to investigate and focus on a 5-year review period [51]. If the full-text is not available for review then there is no information to review and extract. If there is some information it is most likely unreliable. Time constraints and language barriers restrict this review to consider papers written in English only because the author is unilingual and does not 6

7 Papers that are either a research paper, peer-reviewed paper, technical reports or academic manuscript. Papers that specify in the abstract that it benefits from evaluation of a software product line method, technique or approach. have the resources available for translation of other languages. Due to quality restrictions this review was limited to conducted searches in academic electronic databases and technical reports. Other sources of evidence such as works-in-progress were excluded. If the main objective of the paper is to evaluate a software product line method, technique, or approach then it would most likely be mentioned in the title or abstract. Exclusion Criteria Papers that are duplicates of papers already included. Papers that are systematic literature reviews. Papers that present a theoretical evaluations of a software product line method, technique or approach. Papers that are solely evaluations of software product line technologies rather than a method, technique or approach. Papers that are evaluations of software engineering methods, techniques, or approaches that are not a software product line method, technique or approach but are evaluated in the context of software product line development (i.e. Agile or Model-driven development practices used to develop a software product line). Table 5: Exclusion criteria for filtering out papers for the study Rationale Including duplications will skew the results of this review. If duplicate papers are found, only the latest version will be included and all others excluded. Systematic literature reviews that study other systematic literature reviews are considered tertiary studies. This Systematic literature review is a secondary study such that it reviews primary studies. Theoretical evaluations do not produce empirical evidence. The main focus of this study is on empirical evaluations of software product line methods, techniques, or approaches. Technologies are tools that are used to assist in the execution of methods, techniques, or approaches. They are not methods, techniques, or approaches themselves. The main focus of this study is on empirical evaluations of software product line methods, techniques, or approaches. Papers that evaluate methods, techniques or approaches adopted from other software engineering methodologies other than software product lines must be rejected even if it is applied in the context of software product line development Study Quality Assessment In addition to the inclusion and exclusion criteria it is important to assess the quality of the selected primary studies for a variety of reasons one of which is to provide more detailed inclusion and exclusion criteria [5]. However, because the inclusion and exclusion criteria presented in Section 3.3 are quite vigorous as is, the quality assessment criteria presented in this section will serve to render the variances in study quality by providing a way of scoring the quality of individual primary studies. Furthermore, the quality assessment criteria will also serve to provide guidance in interpreting findings of this review and to make recommendations for future work [5]. Utilizing the recommendations outlined in [24], the quality assessment criteria are intended to appraise the attributes of research design and reporting of the selected primary studies. These attributes are incorporated into the data extraction form presented in Table 7 of Section 3.5 and include the following components: Theoretical context describing the theoretical aspects of the proposed software product line method, technique, or approach creates a frame of reference from which the research objectives are derived [24][25]. Research Design combining the guidelines for empirical research design recommended in [24][25][26] consensus was given to specific aspects of research design that were deemed appropriate for assessing the quality of evaluations. These aspects include clearly stated research objectives supported by research questions, domain context, samples and 7

8 instruments used to carry out the evaluation, and the data collection and analysis procedures. Describing the research design includes stating research objectives that are supported by research questions [24][25][26]. Stating the objective sets the focal point of the research [27]. This can be stated in various ways such as in a problem statement, hypotheses or objective/goal statement. The research objective should be supported by research questions that state what is needed to fulfill the objective [24][25]. Moreover, Kitchenham [24] asserts defining the hypothesis correctly is the most important part of designing empirical research. Equally important is describing the data collection and analysis procedures. Describing the data collection and analysis procedures ensures that the evaluation can be easily replicated and the results are analyzed correctly [24]. A welldefined research design has the advantage for subsequent analysis to be straightforward and clear [24]. Research Analysis interpreting the results of an evaluation and relating the results back to the research objectives ensures the results are statistically or practically significant and valid [24]. Therefore, reporting the results with clear interpretation, communicating assumptions made, threats to validity and lessons learned is important for the reader to make sure the findings are reasonable and trustworthy [24][25][26]. Research Conclusions Researchers should conclude their findings into a context of implications by forming theories that constitute a framework for their analysis [25]. This is where researchers discuss practical impact, relate their work and results to earlier research, and give footing towards future work [26]. Based on the attributes listed above, the following quality assessment checklist was created (see Table 6). Table 6: Quality Assessment Checklist Attribute Criteria Theoretical context Is the purpose of the method explained? Is the theoretical aspect of the method explained? Is the implementation of the method explained? Are the advantages of the method explained? Are the limitations of the method explained? Research Design Does the study describe a problem, hypotheses or objective/goal statement? Does the study provide research questions that will support or reject the problem, hypotheses or objective/goal statement? Does the study describe the domain which the evaluation was conducted in? Does the study describe samples and instruments used to conduct the evaluation? Does the study describe its data collection procedures? Does the study describe its data analysis procedures? Research Analysis Does the study provide an interpretation or analysis of the results? Does the study describe its results relative to its problem, hypotheses or objective statement? Does the study describe assumptions made during the execution of the evaluation? Does the study discuss threats to validity? Does the study discuss lessons learned? Research Conclusions Does the study discuss the practical implications? Does the study discuss related work? Does the study provide recommendations for future work? In order for the criteria in the Quality Assessment Checklist to provide a way of weighting the significance of individual primary studies, each question will be scored based on the answer 8

9 provided. The questions are intended to be close-ended questions meaning there are limited responses (Yes/Somewhat/No), according to [6], which are respectively assigned scoring points of (1.0/0.5/0.0). This scoring mechanism assists with the rendition of variances in study outcomes which will in turn determine the quality of each individual study and the strength of inferences Data Extraction The data extraction form in Table 7 was designed to accrue all the necessary information required to address the research questions and quality assessment criteria. In addition to acquiring the information needed to address the research questions and quality assessment criteria, the following standard information was also extracted from each primary study: Title of the Paper, Sources (Database and Journal), Date Published, Paper URL, Document Object Identifier (DOI) and Authors. The purpose of collecting the aforementioned information is to provide analysis of the meta-data of the studies themselves. For instance, distinguishing the time frames of the studies (i.e. how many studies were published in the year 2006 versus 2011). This measurement will provide insight into the growth and interest in software product line research. Other points of interest that can be answered include who the main players are in software product line research, how readers can access the studies via URL or DOI, and what sources are more likely to publish software product line research, and more importantly, publish high quality research. However, this review has limited its work to reporting the findings associated with answering the research questions stated in Section 2. Table 7 Data Extraction Properties # Property Values Description RQ Mappings 1 Method The method name provided in The name of the method under evaluation (if RQ 1 the reviewed literature. the author provides one). 2 Problem Area The area of software product line The area of software product line RQ 1.1 development. development which the method is intended to provide a solution for. 3 Project Focus Product Management, Domain Engineering, Product Engineering, Not Mentioned Project focus specifies what software product line development phase the method is meant to be used. RQ Project (Lifecycle) Activity 5 Method described Business Vision Evaluation, Scope Definition, Domain Requirements Analysis, Common Requirements, Variable Requirements, Domain Design, Domain Implementation, Domain Testing, Product Requirements, Product Design, Product Construction, Product Testing, Product Delivery/Support, Not Mentioned Yes No Somewhat Development process specifies whether the method is meant to be used during a specific stage of the software development process. These activities can be seen as sub-processes of the three phases identified in item Is the purpose of the method explained? 2. Is the theoretical aspect of the method explained? 3. Is the implementation of the method explained? RQ 1.3 RQ 1.4 9

10 6 Research - Method 7 Context Academia, Industry Case Study, Experimental, Survey, Post-mortem Analysis, Other 8 Subjects Practitioner, Researcher, Student 9 Scale of evaluation 10 Study designed described 11 Study reporting described Toy example/prototype, Down-scaled real example, Industrial Yes No Somewhat Yes No Somewhat 4. Are the benefits of the method explained? 5. Are the limitations of the method explained? The research method used to evaluate the software product line method. The type of evaluation classification. Specifies the context in which the evaluation is made. Specifies who uses the method in the evaluation Specifies the scale on which the evaluation is made. In combination with Study Context, does the study also provide information that address the following questions: 1. Does the study describe a problem, hypotheses or objective statement? 2. Does the study provide research questions that will support or reject the problem, hypotheses or objective statement? 3. Does the study describe the domain which the evaluation was conducted in? 4. Does the study describe samples and instruments used to conduct the evaluation? 5. Does the study describe its data collection procedures? 6. Does the study describe its data analysis procedures? In combination with Study Context and Study Design, does the study also provide information that address the following questions: 1. Does the study describe assumptions made during the execution of the evaluation? 2. Does the study provide an interpretation or analysis of the results? 3. Does the study describe its results relative to its problem, hypotheses or objective statement? 4. Does the study discuss threats to validity? 5. Does the study discuss the practical implications? 6. Does the study discuss lessons learned? 7. Does the study discuss related work? 8. Does the study provide recommendations for future work? RQ 2, RQ 2.1, RQ 2.5 RQ 3 RQ 2, RQ 2.2, RQ 2.5 RQ 3 RQ 2, RQ 2.3, RQ 2.5 RQ 3 RQ 2, RQ 2.4, RQ 2.5 RQ 3 RQ 3, RQ 3.1 RQ 3, RQ 3.1 Ø Properties 1-5 are intended to provide insight into RQ1: What software product line methods, techniques, and/or approaches exist today? Creating an inventory of software product line methods, techniques, and approaches that includes method names, method descriptions, targeted problem areas, targeted project lifecycles and activities gives a snapshot of existing software product line methods, techniques, and approaches being evaluated. 10

11 Ø Properties 6-9 are intended to provide insight into RQ2: What is the state of evaluations beings conducted on software product line methods, techniques, and/or approaches? Determining the state of software product line method evaluations takes into account the research method used, the context which the evaluation took place, the subjects that took part in the evaluation and the scale of the evaluation. These four properties describe the context of the evaluations studied and provide adequate information to develop an understanding of the state of evaluations. Ø Properties 6-12 are intended to provide insight into RQ3: To what extent do the evaluations support the adoption of software product line methods, techniques, and/or approaches? For a software product line method, technique or approach to be considered for adoption a few things need to be considered. The context which the method, technique or approach was evaluated is an important factor because the degree of realism influences how applicable the evidence is in other organizational environments. The quality of research design and reporting is also a major influential factor to adoption because it directly affects the trustworthiness of evidence. Here we provide further description of each of the properties: Property #1 Method: This property s value is in accordance with what is presented in the research study as the software product line method, technique or approach being evaluated. If the author(s) has provided the software product line method with a specific name then this value will be extracted and reported in accordance with what is stated in the study. If a specific name is not available for reporting then this property will provide a brief description of the method, technique or approach. Example data entry: Textual Variability Language (TVL) - a textual feature modeling dialect geared towards software architects and engineers. Property #2 Problem Area: This property s value identifies the problem area which the software product line method is focused. This value should highlight a specific problem area which the software product line method is attempting to address. Example data entries: Product Derivation, Variability Management, Visualization, and Architecture. Property #3 Project Focus: This property s value indicates which phase the software product line method is subjected to in relation to software product line development. The primary phases of development are Product Management, Domain Engineering, or Product Engineering Error! Reference source not found.. Although the three phases listed is the closest consensus practitioners and researchers share regarding the phases of software product line development, some authors may report the phases differently, mention alternative phases, or not mention a phase. In this event, a general impression will be applied based on the reviewer s conclusions. If a value cannot be determined then unknown will be applied. Property #4 Project (Lifecycle) Activity: This property s value categorizes the software product line method with respect to what development activity of software product line engineering it is intended to be used in. The values for this property are adapted from Error! Reference source not 11

12 found.. Some authors may report the activities differently, mention alternative activities, or not mention an activity. In this event, a general impression will be applied based on the reviewers conclusions. If a value cannot be determined then unknown will be applied. Property #5 Method described: This property s value indicates how the method was reported. Six questions related to the method description are asked and will contribute to the scoring of this property. These questions are listed under the study quality assessment criteria section. Property #6 Research Method: This property s values are in accordance to the empirical research methods discussed in [27][30]. These research methods are namely experiment, case-study, survey, and post-mortem analysis. Other types of research methods are identified (i.e. Action Research) will be tagged as Other but reported based on their classification. Property #7 Context: This property s values indicate whether the evaluation occurred in an industrial setting or an academic setting. If the context is not mentioned then it will be assumed the evaluation was conducted in an academic setting. Property #8 Subjects: This property s values indicate the type of participants observed during the evaluation. Participants may include practitioners, researchers, or students. If no participants are identified then it will be assumed the participants are the authors of the study and will be classified as researchers. Property #9 Scale of Evaluation: This property s values signify the size of evaluation conducted during the study. Evaluations are conducted on a small, medium, or large scale ranging from prototypes/toy examples, down-scaled real-world examples, to industrial size projects. Open source software product lines such as Berkeley DB will be classified as industrial scale evaluations. Property #10: Study Design described: This property s value indicates how the study was designed. Five questions related to the design of the study are asked and will contribute to the scoring of this property. These questions are listed in the study quality assessment criteria section. Property #11: Study Reporting described: This property s value indicates how the study was reported. Eight questions related to the reporting aspect of the study are asked and will contribute to the scoring of this property. These questions are listed in the study quality assessment criteria section Data Synthesis The results of the review are presented based on the order of the research questions. Syntheses of the results are descriptive in nature and complemented by quantitative summaries of classifications presented in tabular format and graphical representation. The information extracted from the primary studies is tabled in accordance with the review questions. The tabled information will focus on similarities and differences between the primary studies and will include counts and percentages of study properties and values identified in the data extraction form. If there is specific information available to be extracted that will support the quantitative synthesis of results then the results are presented using graphical plots, i.e., forest plot. Annotations will be used on the forest plot to support sensitivity analysis [24]. 12

13 4. Conducting the Review This section provides a description of how the review was executed (the steps taken to arrive at the final selection of primary studies to be interpreted and analyzed) and the circumstances which presented as threats to validity of this review Inclusion and Exclusion of Studies The review execution followed an automated approach by applying the search strategy discussed in Section 3.2. In addition to the automated search on the databases, the authors manually checked the returned results in order to make sure that no significant part of the literature is missing. References were also crosschecked manually by one of the authors by browsing through the online databases to ensure that no significant work in this area is overlooked by the automated search. The overall search resulted in the identification of 615 articles. Of the 615 papers found, 128 were from the ACM Digital Library, 271 were from the IEEE Xplore Digital Library, 48 were from ScienceDirect, 122 were from SpringerLink, and 46 were from the Wiley Online Library. 347 articles were rejected when the inclusion criteria had been applied to the title and abstract of the papers. Another 180 articles were rejected after further assessment of the quality of the papers on the basis of the quality assessment criteria (explained in Table 22), 8 of which were tagged as being related work. After the application of the inclusion, exclusion, and quality assessment criteria, a total of 79 papers remained for data extraction and classification with the intent to be subjected to analysis for the purpose of addressing the research questions. As a reference management technique, a free service for managing and discovering scholarly references called CiteULike 2, was used to store and manage the studies. A reference management system like CiteULike helps in the removal of duplicate studies and provides a list of references for readers [30]. The complete, unfiltered search results that include all 615 primary studies initially found have been retained for future access in the event that reanalysis is required. The unfiltered search results can be obtained by visiting Threats to Validity The main threats to validity of this review are publication and selection bias, and data extraction and classification. An explanation about each limitation is provided along with the actions taken to minimize the impact on the threats on this review Validation of the review protocol The review protocol developed for this systematic literature review was created and validated prior to conducting the review. Several guidelines were consulted including Kitchenham [5], Wright et

14 al. [30], Biolchini et al.[31], and Petersen [41] to create the review protocol. However, it was Kitchenham [5] that was the primary source of guidance. The review protocol was prepared by the first author and was then reviewed by the second author and consequently by the other two authors to verify whether research questions were formulated correctly; whether the search strings were according to the research questions, and whether the data to be extracted would be proper to address the research questions. Based on feedback provided in the discussion over research protocol, improvements were made to the research questions, search strategy, study selection criteria, quality assessment checklist, and data extraction strategy Validation of publication and primary study selection Systematic reviews continue to be subjected to publication bias because researchers are more likely to publish studies that produce positive results and refrain from publishing studies that produce negative results [5][30]. This type of bias has the potential of distorting the results of this review. Kitchenham [5] recommends performing manual searches in addition with automated searches to include reference lists from relevant primary studies and articles, company journals (i.e. IBM Journal of Research), Grey Literature (i.e. technical reports and works-progress), Conference Proceedings, and the Internet. To address this, besides the automated search on the databases, the authors manually checked the results of the automated search in order to make sure that no significant part of the literature is missing. References were also cross-checked manually by one of the authors by browsing through the online databases to ensure that no significant work in this area is overlooked by the automated search. Therefore, we believe publication bias was kept to a minimum in our work. Nevertheless, the effects on the results are considered to be insignificant because the combined use of digital libraries gives access to quality scientific and technical information found in over 3 million articles across over 2,300 journals. The inclusion and exclusion criteria of this review include segments that also produce a threat of selection bias such as selecting studies only written in the English language. Although this type of threat can be levied with bias, it has minimal impact on the conclusions [30]. In order to validate the inclusion and exclusion criteria, a random set of five studies were reviewed based on the inclusion and exclusion criteria. The results were carefully analyzed and validated by all researchers involved in the study. All 615 studies were subjected to the selection process. Three of the authors were involved in the selection process and 79 studies were deemed acceptable and tagged as selected using the reference management system mentioned in Section 4.1. The remaining studies were either rejected or classified as related work. Reasons for acceptance and rejection were noted on all studies Validation of data extraction criteria and classification Vague explanations of the data extraction criteria and problems with misclassification exemplifies as threats to the validity of the results of this review. Data extraction criteria should be clearly defined and relevant to providing the information needed to answer the study s objective. 14

15 Consequently, problems with data extraction descriptions will cause problems in classification. Published research may be of poor quality meaning the reports are written poorly or ambiguously and at times fail to include relevant data [30]. This makes it difficult to fit the data extracted from the papers into the chosen categories of the data extraction properties. Hence, it was necessary to validate the data extraction properties against credible sources (i.e. experts in the field of empirical research and software product lines). The data extraction properties 1 and 2 were sourced directly from the primary studies reviewed in this study. Each study reported the name and description of their proposed method, technique or approach differently and the problem areas which their solution targeted were also reported differently. Therefore, at best the verbatim names and descriptions of methods, techniques or approaches and the targeted problem area were reported based on the information provided in the source papers. In this circumstance, the extracted information was discussed between the authors and the information was verified. Therefore, we had full agreement on names and descriptions of the methods at the end of data extraction process. The data extraction properties 3 and 4 which identified Project Timelines and Project Activities were sourced from the Wikipedia article for Product Family Engineering. However, knowing that Wikipedia is not a credible source of information, further investigation was conducted into one of the article s listed references [36]. Pohl et al. [36] published book Software product line engineering foundations, principles, and techniques was used for validation and found the information in the Wikipedia article to be acceptable. The data extraction property 6 was sourced by [27]. Wohlin et al. [27] provided a discussion on empirical research methods in web and software engineering. In their discussion they identified four research methods: experiments, case studies, surveys, and post-mortem analysis. Furthermore, these research methods are supported by Easterbrook et al. [37] on selection of empirical methods for software engineering research. The data extraction properties 7-11 were chosen from a consensual perspective after reviewing the recommendations made in [24][25][26]Error! Reference source not found.. Kitchenham et al. [24] produced a set of preliminary guidelines for empirical research in software engineering which are intended to assist researchers in designing, conducting, and evaluating empirical studies. Runeson and Host [25] produced guidelines for conducting and reporting case study research in software engineering which provides empirically derived and evaluated checklists for researchers and readers of case study research. Jedlitschka and Ciolkowski [26] produced reporting guidelines for experiments in software engineering which provides guidance on the expected content of reporting sections of empirical studies. Finally, Wieringa [38] created a unified checklist for observational and experimental research in software engineering which identifies important questions to be answered in experimental and case study research design and reports. Interestingly, Wieringa [38] cites [24][25][26] in his work and examines commonalities between [24][25][26] and CONSORT [39][40] in order to form his unified checklist. Data classification proved to be without certainty since the studies under review did not provide precise answers to the data extraction criteria. Many properties were not described correctly or 15

16 mentioned at all. In these circumstances, Kitchenham [5] and Biolchini et al. [31] recommend contacting the author of a questionable study to assist in resolving uncertainties and provide clarity to unknowns. However, Biolchini et al. [31] provides an alternative suggestion to contacting authors which allows for general impressions of subjective evidence to be made by the reviewer. Due to time constraints, the option to make general impressions on subjective evidence was used. Again, in this circumstance, the extracted information was discussed between the authors and agreement on the information was obtained. 5. Results and Analysis This section provides a discussion and analysis surrounding the results of this systematic literature review based on the 79 primary studies selected. The discussion is structured based on the research questions presented in Section 2.2. In the following the number of the research question related to each section and/or subsection is given in parenthesis that can be traced back to Table What Software Product Line methods Exist (RQ1)? This review has identified 79 unique software product line methods, techniques, and approaches that have undergone an empirical evaluation and are presented in Table 29 found in Appendix C. Table 29 provides the method name, if one is specified, along with a description of the method. The method name and description are described as reported by the author. 20 articles did not provide a specific or formal name for the reported method being evaluated. In addition, each method is also classified in terms of the problem area which the method is intended to address, the project timeline focus, and project activities. Each method is mapped to its associated article using the article id What problem area does the Software Product Line method target (RQ1.1)? Unlike the project lifecycles and project activities, the problem areas targeted by the software product line methods were regularly mentioned in the reviewed articles. Many methods targeted a variety of problem areas. Table 8 shows that Feature Modeling is represented the most by 21 papers, followed by Commonality and Variability Management (CVM) with 15 papers and Requirements Management with 14 papers. Visualization is represented the least by only 1 paper, followed by Process Evaluation and Improvement with 4 papers and Adoption with 5 papers. A graphical representation of the results is presented in Figure 2. The results for each individually reviewed paper in regards to RQ1.1 can also be viewed by referring to Table 29 in Appendix C. Table 8: Targeted Problem Areas Problem Area Code Used For This Problem Area Number Percentage Feature Modeling MOD