Multi-objective Approaches to Optimal Testing Resource Allocation in Modular Software Systems

Multi-objective Approaches to Optimal Testing Resource Allocation in Modular Software Systems Zai Wang 1, Ke Tang 1 and Xin Yao 1,2 1 Nature Inspired Computation and Applications Laboratory (NICAL), School of Computer Science and Technology, University of Science and Technology of China, Hefei 2327, China Email: wangzai@mail.ustc.edu.cn, ketang@ustc.edu.cn http://nical.ustc.edu.cn 2 The Center of Excellence for Research in Computational Intelligence and Applications (CERCIA), School of Computer Science, University of Birmingham, Birmingham B15 2TT, U.K. Email: x.yao@cs.bham.ac.uk Abstract. Software testing is an important issue in software engineering. Nowadays, as the software systems become increasingly large and complex, the problem of how to optimally allocate the limited testing resource during the testing phase has become more and more important and difficult. Traditional Optimal Testing Resource Allocation Problems (OTRAPs) involve seeking an optimal allocation of a limited amount of testing resource to a number of activities with respect to some objectives (e.g., reliability or cost). In this paper, we suggest solving OTRAPs with Multi-Objective Evolutionary Algorithms (MOEAs). Specifically, we formulate OTRAPs as two types of multi-objective problems. First, we consider the reliability of the system and the testing cost as two objectives. Second, the total testing resource consumed is also taken into account as the third objective. The advantages of MOEAs over state-of-the-art single objective approaches to OTRAPs will be shown by empirical studies. Our study has revealed that a well-known MOEA, namely Nondominated Sorting Genetic Algorithm II (NSGA-II) performs well on the first problem formulation, but fails on the second one. Hence, a Harmonic Distance Based Multi-Objective Evolutionary Algorithm (HaD-MOEA) is proposed and evaluated in this paper. Comprehensive experimental studies on both parallel-series and star-structure modular software systems have shown the superiority of HaD-MOEA over NSGA-II for OTRAPs. keywords: Software Engineering; Software ; Multi-Objective Evolutionary Algorithm; Parallel-Series Modular Software System; Star-Structure Modular Software System; Software Testing. 1 Introduction A software development process typically consists of four phases [1]: specification, designing, coding, and testing. The requirements of consumers are defined in the specification phase. After that, the structures and details are designed and implemented during the next two phases. Finally, the software systems are tested to detect and correct latent software errors during the testing phase. The testing phase, which aims to improve the reliability of a software system, is the most costly and time-consuming one among the four phases [1]. It has been estimated that about half of the resource consumed during the software development cycle is testing resource [1]. Moreover, since the sizes of software systems have increased significantly during the past decades, effective utilization of limited testing resource has become even more important than before. A software system is typically comprised of a number of modules. Each of them needs to be assigned with some testing resource before the testing phase. Hence, a natural question is how to allocate the testing resource to modules so that the reliability of a software system is maximized. Such a problem was formally defined by Ohtera and Yamada as the Optimal Testing Resource Allocation Problems (OTRAPs) [1]. Although testing resource can be allocated in rather simple ways (e.g. average allocation, random allocation and proportional allocation), Huo et al. [2] proved that an optimal allocation scheme may lead to significant improvement in terms of the reliability of a software system. In other words, it is well worth optimizing the allocation scheme. Solving OTRAPs is a non-trivial task. Much effort has been devoted to this topic since 199s [1 18], and progress has been made in the way of either proposing more precise/practical formulations of OTRAPs, or utilizing novel problem-solving techniques. We start revisiting the literature from the former type of work. An OTRAP is typically concerned with three factors, namely reliability, cost and testing resource. To explicitly formulate an OTRAP, the relationship between these factors needs to be precisely defined. In the literature, the relationship between reliability and testing resource was usually formulated by Corresponding author

2 Zai Wang, Ke Tang and Xin Yao the Software Growth Models (SRGMs), where the reliability is usually some metric of the failure data, such as the number of failures, time of occurrence, failure severity, or the interval between two consecutive failures [13]. The SRGMs describe reliability growth during software development processes, and can be viewed as formulating the reliability of a software system as a function of the testing resource allocation scheme. Unsurprisingly, early work exclusively aimed at maximizing the reliability with a given budget of resource, by means of minimizing the remaining errors [1, 4, 5], minimizing the number of software faults detected [2] or directly maximizing a function that quantifies system reliability [6 8, 1]. More recently, testing cost is attracting more attentions. Intuitively speaking, testing cost measures the cost required for attaining a given level of reliability. That means, it is essentially a function of reliability, and thus also a function of testing resource [11 13, 15 18]. Recent work on OTRAPs either proposed minimizing testing cost instead of maximizing reliability, or incorporated additional constraints on the testing cost (e.g., the reliability is maximized subject to given bounds of the testing cost and resource) [12, 13, 15 18]. From the perspective of problem-solving techniques, a lot of traditional optimization approaches have been employed to solve OTRAPs, such as nonlinear programming [2 5, 13, 16], gradient projection method [6] and dynamic programming [5]. However, these approaches only guarantee local convergence. They may be easily trapped in local optima if the solution space of an OTRAP is multimodal. Although such disadvantage can be alleviated by running the algorithms multiple times from different initial solutions, choosing appropriate initial solutions requires one to be familiar with both the algorithm and the characteristics of OTRAPs, which is usually very difficult for ordinary users. Alternatively, evolutionary algorithms are metaheuristics that possess strong capability of global search and are usually not very sensitive to initial solutions. Their effectiveness have been demonstrated on a large spectrum of problems in the reliability optimization field, such as resource management and task partition in grid systems [19 21], redundancy allocation [22, 23], reliability optimization of weighted voting systems [24] and OTRAPs [7 12, 15, 17, 18]. In particular, evolutionary algorithms have been reported to perform better than some other techniques on OTRAPs [7, 9 12, 15, 17, 18]. Being intensively investigated in the past decade, OTRAPs were mostly handled as optimization problems with a single objective, i.e., the testing resource was allocated with the only purpose to maximize the system reliability or minimize the testing cost. However, both reliability and testing cost are important for software development and it is unrealistic to overlook either of them. Unfortunately, given a budget of testing resource, more testing cost is usually inevitable if we want to improve the reliability of a software system. Hence, it is impossible that a single solution is optimal in terms of both reliability and cost. Instead, we may resort to balancing between reliability and cost and seeking a good trade-off. Though some previous researchers did attempt to do so [12], they still solved the problem in a single-objective optimization manner, where the resource allocation scheme is optimized with respect to a weighted sum of reliability and testing cost. This might be inappropriate since reliability and cost are generally incommensurable, i.e., they may have different scales. Summing them up does not really provide meaningful information about the quality of solutions, and might cause difficulties in practice since it is hard to determine the appropriate values of weights. In a preliminary work [25], we proposed employing Multi-Objective Evolutionary Algorithms (MOEAs) to solve OTRAPs. Specifically, we formulated OTRAPs as two types of multi-objective problems. First, we considered the reliability of the system and the testing cost as two objectives. Second, the total testing resource consumed was taken into account as the third goal. The advantages of MOEAs over existing approaches were then evaluated by applying two MOEAs, namely Nondominated Sorting Genetic Algorithm II (NSGA-II) [26] and multi-objective differential evolution, on two simple parallel-series modular software systems. This paper substantially extends our previous work, and can be distinguished from it from three aspects: 1. In [25], we found that NSGA-II performs well on the first problem formulation, but fails on the second one. Multi-objective differential evolution performs satisfactorily on both problem formulations with fine-tuned control parameters. However, multi-objective differential evolution involves too many control parameters, which are usually problem-dependent and difficult to fine-tune. As a result, it is hard for a practitioner to utilize multi-objective differential evolution effectively. For this reason, we propose in this paper a novel MOEA called Harmonic Distance Based Multi-Objective Evolutionary Algorithm (HaD-MOEA). HaD-MOEA not only performs well on both types of multi-objective formulations of OTRAPs, but also involves much less control parameters than multi-objective differential evolution. Hence, it fills the gap between the two algorithms we adopted in the previous study. 2. In [25], the efficacy of MOEAs on OTRAPs was only verified on small-scale parallel-series modular software systems, which might not be sufficient to draw a more general conclusion. In this work, we have conducted extensive empirical studies on two additional software systems. One is a parallel-series modular software system with a larger scale than those investigated before. The other is a star-structure modular software system. In this way, we can evaluate the efficacy of MOEAs (in particular, HaD-MOEA) more comprehensively on different problem sizes and system structures. 3. A sensitivity analysis of the proposed HaD-MOEA with respect to the model parameters (i.e., parameters of the software system) is presented. In addition, computational cost of both NSGA-II and HaD-MOEA is also investigated. These issues are indispensable for evaluating the utility of our approach in the real world, but were absent from the previous work.

Multi-objective Approaches to Optimal Testing Resource Allocation in Modular Software Systems 3 The rest of this paper is organized as follows: Section 2 presents the problem formulations of two multi-objective OTRAPs based on two types of software systems. Section 3 introduces the related work on evolutionary multi-objective optimization and proposes HaD-MOEA. In Section 4, NSGA-II and HaD-MOEA are experimentally compared with each other and with some state-of-the-art single-objective approaches on four case studies including three parallel-series modular software systems and a mixture of star-structure modular software system. Finally, discussions and conclusions are presented in Section 5. 2 Problem Formulations Since a software system typically consists of multiple modules, two issues need to be addressed in the problem formulation. That is, the model that formulates the relationship between testing resource and reliability (or cost) on a single module, and the structure of the system (i.e., the way that modules are organized to form the system). Following previous work, we address the former issue with SRGMs, which are defined based on four assumptions [13]: 1. The process of the fault removing can be described by a nonhomogeneous Poisson process (NHPP). 2. The software application is subject to failures at random time caused by the remaining faults in the system. 3. Each time a failure occurs, the corresponding fault is immediately removed and no new faults are introduced. 4. The mean number of faults detected in the time interval (t, t + t) by the current testing resource expenditures is proportional to the mean number of remaining errors. For the latter issue, two types of system structures are considered, including parallel-series structure and star structure. Table 1 summarizes the notations and main acronyms used in this paper. Table 1. The notations and main acronyms used in this paper Notations T total testing resource (or time) T i testing resource allocated to module i Ti optimal allocation resource to module i C the total testing cost R system reliability C i(r i) cost function of module i according to the reliability of module i (R i) m(t) mean value function in NHPP R i(x T i) reliability of module i after a testing period T i. x is a constant number associated with the studied system T li testing resource allocated to the ith modular in the lth parallel group T j testing resource allocated to the jth serial module R li reliability of the ith module in the lth parallel group R j reliability of the jth serial modular T C testing resource allocated to the central unit in star-structure modular software system Acronyms OTRAP optimal testing resource allocation problem SRGM software reliability growth model MOEA multi-objective evolutionary algorithm NSGA-II nondominated sorting genetic algorithm II HaD-MOEA harmonic distance based multi-objective evolutionary algorithm Y-X a single-objective algorithm proposed by Yang and Xie [9] D-X a genetic algorithm proposed by Dai and Xie [12] T-M a single-objective algorithm proposed by Tom and Murthy [27] 2.1 Problem Formulations on Parallel-series Modular Software Systems Parallel-series modular software systems are the most popular simulated systems to describe real-world software systems [28]. They are comprised of some groups of parallel modules and some serial modules. The basic structure of a parallel-series modular software system with n groups of parallel modules (the group i consists of k i modules) and m serial modules is demonstrated in Fig. 1. Based on SRGMs, the reliability of the a parallel-series modular software system and testing cost can be quantified using Eqs. (1)-(5).

4 Zai Wang, Ke Tang and Xin Yao P 11 P n1 Input P 12 P n2 S 1 S m Output Fig. 1. The basic structure of a parallel-series modular software system The failure intensity of module i in a system is denoted as λ i (T i ), which can be calculated by: λ i (T i ) = a i b i exp{ b i T i }, (1) where a i and b i are constants. a i is the mean value of the total errors in module i and b i is the rate of detected errors in module i. T i is the testing resource (testing time) assigned to module i. The reliability of module i is calculated by R i (x T i ) = exp { λ i (T i )x}, x, (2) Based on the above two equations, the reliability of a parallel-series modular software system can be calculated with the following equation: R(x T ) = n k l m (1 [1 R li (x T li )]) R j (x T j ), (3) l=1 where (1 k l [1 R li (x T li )]) is the reliability of the lth parallel group. There are n groups of parallel modules, thus the total reliability of these n groups of parallel modules is n (1 k l m [1 R li (x T li )]). R j (x T j ) is the total reliability of the m serial modules. The testing cost for each module is defined as [12]: l=1 C i (R i ) = H i exp{b i R i D i }, (4) where H i, B i and D i are constants that control the increment speed of the testing cost corresponding to the reliability in module i. Correspondingly, the testing cost for the whole parallel-series modular software system is: C = n k l C li (R li ) + l=1 m C j (R j ), (5) where n k l C li (R li ) is the total testing cost for the n groups of parallel modules, and m C j (R j ) is the total testing cost for l=1 the m series modules. Given an OTRAP, we aim at maximizing the reliability, while minimizing the testing cost. Meanwhile, the consumed testing resource should not exceed a predefined budget or (ideally) be minimized. However, Eq. (4) shows that the testing cost exponentially increases with reliability. Hence, the objectives of maximizing reliability and minimizing testing cost conflict with each other. Such a scenario is not uncommon in the real world, and has been investigated under the name of multi-objective optimization problem. Mathematically, a multi-objective problem with m conflicting minimizing objectives can be formulated as Eq. 6: Minimize y = f(x) = (f 1 (x), f 2 (x),..., f m (x)) subject to x = (x 1, x 2,..., x n ) X y = (y 1, y 2,..., y m ) Y, (6)

Multi-objective Approaches to Optimal Testing Resource Allocation in Modular Software Systems 5 where x, X, y and Y are called decision vector, decision space, objective vector and objective space, respectively. f i (x) is the ith objective function of the problem. Since different objective functions usually conflict with each other, there seldom exist a unique solution that is optimal in terms of all objective functions. For this reason, the common approach to a multi-objective optimization problem is to seek a set of Pareto optimal solutions. In other words, each solution should not be inferior to any other solution on all objectives. Such solutions are referred to as nondominated solutions. When formulating OTRAPs as multi-objective optimization problems, we first simultaneously consider reliability and cost as two objectives and formulate OTRAPs as bi-objective problems. Despite of the increase of the size and complexity of software systems, the cycle of a software development process has become shorter and shorter. Under such a circumstance, software testers might want to shorten the testing phase, even with the price of slight decrease in reliability. So, we further consider the total testing resource expenditure as a new target, and a tri-objective problem can be formulated. Table 2 presents the mathematical formulation of the two multi-objective OTRAPs on parallel-series modular software systems. Table 2. The formulations of two multi-objective problems on parallel-series modular software systems (maximum testing resource to be allocated is T) (1) Bi-objective problem Maximize R = n k l (1 [1 R li (x T li )]) m R j(x T j) l=1 Minimize C = n k l C li (R li ) + m C j(r j) s.t. n k l l=1 l=1 T li + m T j T, T li, T j (2) Tri-objective problem Maximize R = n (1 Minimize Minimize s.t. n k l l=1 l=1 k l k l [1 R li (x T li )]) m R j(x T j) C = n C li (R li ) + m C j(r j) l=1 k l T c = n T li + m l=1 T j T li + m T j T, T li, T j 2.2 Problem Formulations on Star-structure Modular Software Systems Star-structure modular software systems are comprised of two types of units: The central unit acts as a server and can be comprised of parallel, serial or parallel-series modules. A non-central unit consists of some connected modules, each of which is an input of the whole system. Fig. 2 illustrates a star-structure modular software system with n non-central modules and m central modules. Eqs. (1), (2) and (4) can be readily utilized to quantify the reliability and cost of every single module in a star-structure modular software system. On the basis of these equations, the reliability of a star-structure modular software system can be calculated as follows: R(x T) = n R c i (x Ti c ) R C (x T C ) n, (7) where R c i (x T c i ) is the reliability of the ith module in the non-central unit, and R C(x T C ) is the reliability of the central unit. Similarly, the cost of testing a star-structure modular software system can be calculated by Eq. (8). C = n m C c i (R c i ) + C c j (R c j), (8)

6 Zai Wang, Ke Tang and Xin Yao Input NC 1 Input NC 2 Input NC 3 CM Output Input NC n Non-Central Modules m Central Modules Fig. 2. The basic structure of a star-structure modular software system where C c i (R c i) is the cost of the ith module in the non-central unit, and C c j (R c j) stands for the cost of the jth module of the central unit. Similar to parallel-series modular software systems, the bi-objective and tri-objective OTRAPs on star-structure modular software systems are mathematically formulated in Table 3. Table 3. The formulations of two multi-objective problems on star-structure modular software systems (maximum testing resource to be allocated is T) (1) Bi-objective problem Maximize R = n R c i (x Ti c ) R C(x T C) n Minimize C = n C c i (R c i ) + m C c j (R c j ) s.t. n Ti c + m Tj c T, Ti c, Tj c (2) Tri-objective problem Maximize R = n R c i (x Ti c ) R C(x T C) n Minimize Minimize s.t. n Ti c C = n C c i (R c i ) + m C c j (R c j ) T c = n Ti c + m Tj c + m Tj c T, Ti c, Tj c 3 Harmonic Distance based MOEA for Multi-Objective OTRAPs In the past decades, evolutionary algorithms have emerged as an effective approach to multi-objective optimization problems [29, 3]. A lot of so called MOEAs, such as the Vector Evaluated Genetic Algorithm (VEGA) [31], the Pareto Archived Evolution Strategy (PAES) [32], and the Nondominated Sorting Genetic Algorithm II (NSGA-II), have been proposed. Proposed by Schaffer in 1985, VEGA is known as the first MOEA in literature. After that, the Nondominated Sorting Genetic Algorithm (NSGA) [33], Niched- Pareto Genetic Algorithm (NPGA) [34] and Multi-Objective Genetic Algorithm (MOGA) [35] were developed. Unlike VEGA, which adopts an objective-based fitness assignment strategy, these three methods assign fitness based on the concept of domination. Intuitively speaking, the latter three methods differ from VEGA in the way of selecting good solutions that are preserved in the problem-solving process. More recently, some other novel techniques have been developed for

Multi-objective Approaches to Optimal Testing Resource Allocation in Modular Software Systems 7 new MOEAs. For example, niching technique has been suggested for preserving the diversity of the obtained solutions, based on which NSGA-II and PAES were proposed [26, 32]. Elitism was also proven to be an effective factor for the convergence of an MOEA. Hence, most of state-of-the-art MOEAs, including the Strength Pareto Evolutionary Algorithm (SPEA) [36], SPEA2 [37], PAES and NSGA-II, were designed to preserve the nondominated solutions found during the problem-solving process. Algorithm 1 The Pseudo-Code of NSGA-II [26] 1: Initialize: Set the population size as N, randomly generate the parent population P = {x 1, x 2,..., x N }; 2: Set the generation number t = ; 3: while t < t max (the terminate generation number) do 4: 5: Generate the offspring population Q t from P t with the same population size; Combine the parent and offspring population via R t = P t Qt ; 6: Sort all solutions of R t to get all non-dominated fronts F = fast-non-dominated-sort(r t) where F = (F 1, F 2,...); 7: Set P t+1 = φ, i = 1; 8: while the parent population size P t+1 + F i < N do 9: (a) calculate crowding-distance of F i; 1: (b) add the ith non-dominated front F i to the parent pop P t+1; 11: (c) i = i + 1; 12: end while 13: Sort the F i according to the crowding distance; 14: Fill the parent pop P t+1 with the first N P t+1 elements of F i; 15: Set t = t + 1; 16: end while 17: return P t. Among all the above-mentioned MOEAs, NSGA-II has shown superior performance on not only benchmark problems [38,39], but also real-world application [4], and thus has been an off-the-shelf choice for solving multi-objective optimization problems. For this reason, we first employed NSGA-II to tackle the multi-objective OTRAPs on parallel-series modular software systems in a previous study [25]. However, empirical studies showed that NSGA-II managed to solve the bi-objective problem, but failed on the tri-objective problem. Although some recent studies have revealed that the performance of MOEAs may deteriorate when the number of objectives of a problem increases [41], it is still necessary to investigate why NSGA-II failed in the context of OTRAPs. Therefore, we have investigated this issue in-depth, and found that the failure of NSGA-II may be due to two reasons: 1. The crowding distance measure does not accurately reflect the crowding degree. f2 A B C D X Y Z f1 Fig. 3. Crowding degree estimation Algorithm 1 presents the pseudo-code of NSGA-II. One critical procedure in NSGA-II is the calculation of crowding distance of a solution in the objective space [26]. Given a solution (a point in the objective space), the crowding distance is calculated as the 1-norm distance between the two nearest neighbors of the solution. In a multi-objective optimization problem, it is usually expected that the final solutions cover the whole objective space well. Hence, in the problem-solving

8 Zai Wang, Ke Tang and Xin Yao process, the solutions with larger crowding distances are preferable. The first drawback of NSGA-II is that it employs a crowding distance metric which does not accurately reflect the actual crowing degree of a given solution. Fig. 3 illustrates such a scenario. We would say that solution C is more crowded than solution Y since C is quite close to D, and thus preserving one of them should be enough. However, in NSGA-II, the crowding distance of C is larger than that of Y. Hence, NSGA-II may obtain more solutions in one region of the objective space, while obtain much less solutions in some other regions. 2. The selection strategy based on the crowding distance is unilateral. Since all MOEAs search for the solutions in an iterative way, the most important part of an MOEA is how it operates in a single iteration (usually referred to as a generation in the EA literature). From Algorithm 1, we can observe that, in each generation of NSGA-II, the parent population P t and offspring population Q t are first combined to make an intermediate population R t. Then, R t is sorted into different nondominated fronts through the function fast-non-dominated-sort, and for every nondominated front, the crowding distance of each individual in it is calculated. The parent population for the next generation is finally selected from R t based on their nondominated levels and crowding distances. During this process, when calculating the crowding distance of a solution, only those solutions belonging to the same nondominated front are considered. However, this may lead to an inappropriate selection of solutions. For example, the rectangular and circular points in Fig. 4 belong to two nondominated fronts, respectively. Assume that all of rectangular points have been selected and we need to select some circular points based on the crowding distance. Based on the selection strategy of NSGA-II, point A will be selected in prior to point B, but this is somewhat counter-intuitive since point B is obviously less crowded than point A. f2 X Y A B f1 Fig. 4. Crowding degree estimation Having the above two drawbacks of NSGA-II in mind, we propose two alternative approaches to deal with them. First, instead of calculating the crowding distance using the 1-norm distance between the two nearest points to a given solution, we calculate it with the harmonic average distance. Assume the distances of the k-nearest solutions of a solution are d 1, d 2,..., d k, the harmonic average distance associated with this solution can be calculated by Eq. 9: k d = 1 d 1 + 1 d 2 + + 1 (9) d k To overcome the second drawback of NSGA-II, we propose the following scheme: After sorting the solutions in the intermediate population into a number of fronts, if some solutions are to be selected from a front, the crowding distance will be calculated based on both the solutions belonging to the same front and all the previously selected solutions. In this way, the undesirable scenario illustrated in Fig. 4 will be prevented. Incorporating the above two schemes into the framework of NSGA-II, we obtain a new MOEA, HaD-MOEA. The pseudocode of HaD-MOEA is shown in Algorithm 2. Specifically, lines 12-15 present the detailed steps of the two novel schemes adopted by HaD-MOEA. Taking advantage of the two new schemes, we expect the solutions obtained by HaD-MOEA to spread better in the objective space than those obtained by NSGA-II. When applying HaD-MOEA to a system with n modules, a solution is encoded by an n-dimensional vector (chromosome). Each element of it represents the testing time consumed by a module, and the sum of these elements should not exceed the total testing time T. Besides, HaD-MOEA cannot guarantee always generating solutions that satisfy this constraint during search. Therefore, whenever a solution obtained violates the constraint, it will be repaired by using the procedures presented in Table 4.

Multi-objective Approaches to Optimal Testing Resource Allocation in Modular Software Systems 9 Algorithm 2 The Pseudo-Code of Harmonic Distance Based Multi-Objective Evolutionary Algorithm (HaD-MOEA) 1: Initialize: Set the population size as N, randomly generate the parent population P = {x 1, x 2,..., x N }; 2: Set the generation number t = ; 3: while t < the terminate generation number do 4: 5: Generate the offspring population Q t from P t with the same population size; Combine the parent and offspring population via R t = P t Qt ; 6: Sort all solutions of R t to get all non-dominated fronts F = fast-non-dominated-sort(r t) where F = (F 1, F 2,...); 7: Set P t+1 = φ and i = 1; 8: while the parent population size P t+1 + F i < N do 9: add the ith non-dominated front F i to the parent pop P t+1; 1: i = i + 1; 11: end while 12: Combine F i and P t+1 to a temporary vector T ; 13: Calculate the harmonic crowding distances of individuals of F i in T ; 14: Sort the F i according to the crowding distance; 15: Set T = φ; 16: Fill the parent pop P t+1 with the first N P t+1 elements of F i; 17: Set t = t + 1; 18: end while 19: return P t. Table 4. Repairing an infeasible offspring a = (t 1, t 2,...t n) Assume an infeasible solution a is re-assigned to a new chromosome a : 1: First we calculate the sum of elements in a :S = sum(a ); 2: For the bi-objective problem, the a is: a =(t 1, t 2,...t n) = ((t 1 T/S), (t 2 T/S),...(t n T/S)); 3: For the tri-objective problem, the a is: a =(t 1, t 2,...t n) = ((t 1 T r(, 1)/S), (t 2 T r(, 1)/S),...(t n T r(, 1)/S)); r(, 1) is a random number between and 1. 4 Experimental Studies Experimental studies have been carried out to evaluate the efficacy of our proposed approach. The experimental studies are designed to consist of two parts. First, the MOEAs were compared with three state-of-the-art single-objective approaches to evaluate whether they (in particular HaD-MOEA) are able to provide some advantage over the single-objective approaches. Second, HaD-MOEA was compared with NSGA-II in terms of both solution quality and computational time. 4.1 Comparing MOEAs with Single-Objective Approaches Three single-objective approaches were considered in this experiment. For brevity, these approaches will be denoted using the acronyms of the family names of the researchers who proposed them. The first algorithm, denoted as the D-X algorithm, was proposed by Dai and Xie et al. [12]. It is essentially a genetic algorithm which aims to solve a single-objective problem whose objective function is the weighted sum of the reliability and cost. The second algorithm was proposed by Yang and Xie [9] and thus is denoted as the Y-X algorithm. It considers only the software reliability and ignores the cost. The last method, denoted as the T-M algorithm, was devised by Tom and Murthy [27] who considered the problem of finding an allocation of program modules onto processors of a distributed computing system with the single goal to maximize the reliability while ignored the cost. We compared the MOEAs with the single-objective approaches on two parallel-series modular software systems: A simple parallel-series modular software system with only two parallel modules and a complex parallel-series modular software system with eight modules. The structures of the two systems are illustrated in Fig. 5 and Fig. 6. The Y-X and T-M algorithms have been separately applied to the simple and complex parallel-series modular software systems. The D-X algorithm has been applied to both. Hence, the two systems serve as a good platform for our comparative study. It is noteworthy that all the compared singleobjective algorithms do not consider the total testing time as an objective. Hence, we only consider the bi-objective problem on the two systems in this experiment. All results of the single-objective approaches were obtained from the original publications.

1 Zai Wang, Ke Tang and Xin Yao 1 IN OUT 2 Fig. 5. The structure of the simple parallel-series modular software system with two modules 3 1 4 IN 7 8 OUT 2 5 6 Fig. 6. The structure of the complex parallel-series modular software system with eight modules Following the previous studies, we assume that 1 persons are available for testing the simple system, and 23 persons are available for the complex system. Each person can spend up to 1 hours on the testing task. In addition, all the parameters of the systems are set as the values used in previous studies [9, 12, 27]. We ran the two MOEAs (NSGA-II and HaD-MOEA) on each system for 3 independent runs. Similar results were obtained in the 3 runs. The solutions obtained in the first run on the simple system are illustrated in Fig. 7, together with the solutions obtained by D-X and Y-X algorithms. In addition, the solutions obtained in the first run on the complex system are illustrated in Fig. 8, together with the solutions obtained by D-X and T-M algorithms. From the figures, it can be observed that both NSGA-II and HaD-MOEA managed to provide such a set of solutions that provide a variety of trade-off between the system reliability and the cost. Furthermore, it would be interesting to check whether the solutions achieved by the single-objective approaches can also be obtained by MOEAs. For this reason, we recorded all nondominated solutions obtained by the MOEAs in the 3 runs. Then, the difference between the reliability of these solutions and the solutions obtained by the single-objective approaches were calculated. The nondominated solutions corresponding to the smallest difference were finally identified and presented in Table 5. It can be found that the MOEAs managed to find the same or very similar solutions as that provided by the single-objective approaches. However, MOEAs have find many additional solutions with different trade-off between the reliability and cost, which enables a software tester to ask what-if questions. Such a set of different solutions also enable a software tester to see the exact relationship between the reliability and cost. It is now possible to ask practical questions, such as how much savings one can obtained if we were willing to sacrifice the reliability by certain amount. Therefore, compared to the single-objective approaches, the major advantage of using MOEAs on OTRAP is that it can provide a lot of additional choices to a practitioner, and thus benefit organizing the whole testing process. 12 The results for the bi objective problem on the simple parallel series system 12 The results for the bi objective problem on the simple parallel series system 1 8 NSGA II D X Y X 1 8 HaD MOEA D X Y X 6 6 4 4 2 2.4.5.6.7.8.9 1.4.5.6.7.8.9 1 Reliaility Fig. 7. The results for the bi-objective problem on the simple parallel-series modular software system

Multi-objective Approaches to Optimal Testing Resource Allocation in Modular Software Systems 11 6 5 4 The results for the bi objective problem on the complex parallel series system NSGA II D X T M 6 5 4 The results for the bi objective problem on the complex parallel series system HaD MOEA D X T M 3 3 2 2 1 1.2.4.6.8 1.2.4.6.8 1 Fig. 8. The results for the bi-objective problem on the complex parallel-series modular software system Table 5. The nearest solutions obtained by MOEAs to the solutions obtained by signle-objective algorithms on the simple and complex parallel-series modular software systems cost factor simple system complex system Algorithm (units) Algorithm (units) Consider NSGA-II.8953 7.356 NSGA-II.8957 49.46 Consider HaD-MOEA.8953 7.355 HaD-MOEA.8927 49.4 consider D-X.8953 7.356 D-X.89 49.65 Consider NSGA-II.9344 1.93 NSGA-II.955 51.88 Consider HaD-MOEA.9344 1.93 HaD-MOEA.914 52.33 Without Y-X.9344 1.93 T-M.91 52.64 4.2 Comparing HaD-MOEA with NSGA-II According to the results presented in the above subsection, we can find that NSGA-II and HaD-MOEA performed comparably for the bi-objective problem on the simple and complex parallel-series modular software systems. Hence, a natural question is: Do the two algorithms perform comparably on all OTRAPs? In other words, is NSGA-II alone sufficient for OTRAPs? To answer this question, we have further conducted comprehensive experiments to compare HaD-MOEA and NSGA-II. In this experiment, we considered both types of multi-objective problems that have been formulated in Section 2. The simple and complex parallel-series modular software systems were utilized as the test-bed. In addition, the efficacy of the MOEAs on another two systems was also evaluated. The first system is again a parallel-series modular software system illustrated in Fig. 9. It consists of twelve modules. The purpose of evaluating the MOEAs on this system is to verify the MOEAs scalability with respect to the number of modules. As shown in Fig. 1, the second system is a star-structure modular software system (denoted as the SS-system) with six modules. By applying the MOEAs to this system, we aim to check whether the MOEAs can work well on different types of structures. For the simple and complex parallel-series modular software systems, we assumed the same total testing time as in the previous sub-section. We assumed that 35 and 15 persons are available for the larger parallel-series modular software system and the star-structure modular software system. The testing time for each person was set to 1 hours. In the previous subsection, all algorithms were evaluated using a set of pre-defined parameters of the systems. It is necessary to evaluate the sensitivity of the MOEAs to these system parameters. Therefore, we further randomly generated 3 different parameter settings for each system. For each module of each system, all the 3 parameter settings were generated within a pre-defined interval, as presented in Table 6. Given a system with a randomly generated parameter setting, both HaD-MOEA and NSGA-II have been applied to it for 3 independent times. To make the comparison as fair as possible, the two algorithms adopted exactly the same initial populations and control parameters in each run. Table 7 presents the control parameters used throughout the experiment. To summarize, we altogether compared the two MOEAs on 24 (two types of problems four systems 3 system parameter settings) problem instances, and carried out statistical test on each instance based on the results of the 3 independent runs. Figure 11 illustrates the nondominated solutions obtained on the tri-objective problems in the first run of the first randomly generated parameter settings. It can be observed that the solutions obtained by HaD-MOEA spread better in the objective space than those obtained by NSGA-II. This suggests that HaD-MOEA might be advantageous to NSGA-II. However, the figures are just a simple illustration and no rigorous conclusion can be made yet. To compare the MOEAs in a quantitative and rigorous way, a quantitative measure of the performance of the two algorithms is required. In the multi-objective optimization field, there are a number of metrics for measuring the goodness of the solutions

12 Zai Wang, Ke Tang and Xin Yao 1 5 9 2 6 IN 1 12 OUT 3 7 11 4 8 Fig. 9. The structure of the larger parallel-series modular software system with twelve modules IN 1 4 IN 2 6 OUT 5 IN 3 Central Unit Non-Central Unit Fig. 1. The structure of the SS-system with six modules Table 6. The range of the parameter values of the four software systems (the value of the parameter is a real number which is randomly generated within the corresponding range). a i, b i, H i, B i and D i are parameters of the ith module in software systems a i b i H i B i D i Simple system i = 1 : 2 i = 1 : 2 i = 1 : 2 i = 1 : 2 i = 1 : 2 [15., 25.] [5.5E-4, 6.E-4] [4.5, 5.5] [5.5, 7.] [4.5, 5.5] Complex system i = 1 : 6 i = 7 : 8 i = 1 : 6 i = 7 : 8 i = 1 : 8 i = 1 : 8 i = 1 : 6 i = 7 : 8 [2., 35.] [3., 35.] [3.E-4, 9.E-4] [5.8E-3, 6.2E-3] [3.4, 3.55] [6., 6.2] [4.9, 5.1] [4., 4.1] Larger system i = 1 : 11 i = 12 i = 1 : 11 i = 12 i = 1 : 12 i = 1 : 12 i = 1 : 11 i = 12 [2., 35.] [3., 35.] [3.E-4, 9.E-4] [5.8E-3, 6.2E-3] [3.4, 3.55] [6., 6.2] [4.9, 5.1] [4., 4.1] SS-system i = 1 : 5 i = 6 i = 1 : 5 i = 6 i = 1 : 6 i = 1 : 6 i = 1 : 5 i = 6 [2., 35.] [3., 35.] [3.E-4, 9.E-4] [5.8E-3, 6.2E-3] [3.4, 3.55] [6., 6.2] [4.9, 5.1] [4., 4.1] Table 7. The parameter settings of NSGA-II and HaD-MOEA software systems problem type crossover mutation terminate population probability probability generation size simple system bi-objective problem.9.1 2 2 simple system tri-objective problem.9.1 2 5 complex system bi-objective problem.9.1 2 2 complex system tri-objective problem.9.1 2 5 larger system bi-objective problem.9.1 2 2 larger system tri-objective problem.9.1 2 5 SS-system bi-objective problem.9.1 2 2 SS-system tri-objective problem.9.1 2 5

Multi-objective Approaches to Optimal Testing Resource Allocation in Modular Software Systems 13 Tesiting Resource Consumed 1 8 6 4 2 2 1 The results for the tri objective problem on the simple parallel series system NSGA II.2.4.6.8 1 (a) Total Testing Resource Consumed 1 8 6 4 2 2 The results for the tri objective problem on the simple parallel series system 1 HaD MOEA.2.4.6.8 1 (b) Testing Resource Consumed 25 2 15 1 5 8 The results for the tri objective problem on the complex parallel series system NSGA II Testing Resource Consumed 25 2 15 1 5 8 The results for the tri objective problem on the complex parallel series system HaD MOEA 4.2.4.6.8 1 (c) 4.2.4.6.8 1 (d) The results for the tri objective problem on the larger parallel series system The results for the tri objective problem on the larger parallel series system Testing Resource Consumed 35 3 25 2 15 1 5 6 NSGA II Testing Resource Consumed 35 3 25 2 15 1 5 6 HaD MOEA 3.2.4.6.8 1 (e) 3.2 (f).4.6.8 1 Testing Resource Consumed 15 1 5 8 The results for the tri objective problem on the SS system NSGA II Testing Resource Consumed 15 1 5 8 The results for the tri objective problem on the SS system HaD MOEA 4 4.2.4.6.8 1 (g).2.4.6.8 1 (h) Fig. 11. Pareto sets obtained by NSGA-II and HaD-MOEA for the tri-objective problems on the four employed software systems (one case study)

14 Zai Wang, Ke Tang and Xin Yao Table 8. The values of hypervolume indicator of obtained results on different parameter suites on the simple parallel-series modular software system (the result that is significantly better than the other is emphasized in boldface) parameter bi-objective problem tri-objective problem suite NSGA-II HaD-MOEA NSGA-II HaD-MOEA best worst mean best worst mean best worst mean best worst mean No.1.833.8328.8329.845.8498.8499.3633.3597.3621.3668.3651.3662 No.2.818.7858.7866.798.796.797.3485.345.3473.3519.355.3514 No.3.8631.863.865.874.868.8683.3734.3686.3719.3775.3723.3749 No.4.8838.8832.8833.8593.8591.8592.3824.3796.3813.3842.3813.3835 No.5.8844.882.884.8712.8712.8712.3428.3323.3376.3363.332.3337 No.6.8691.85.829.8615.869.861.3742.3724.3733.3765.3751.3758 No.7.8479.8477.8478.8471.836.8365.3266.3214.3252.3298.329.3294 No.8.8115.8113.8114.785.7849.7849.3717.3667.3695.3736.3721.3728 No.9.7986.7947.7949.7758.772.774.3222.3211.3218.3238.321.3224 No.1.8494.8491.8492.8289.8288.8289.3499.3437.3476.3538.3499.3513 No.11.8562.8553.8553.8644.864.8641.3568.3549.3555.359.3572.3578 No.12.8866.8865.8865.8595.8583.8585.3993.3946.3977.425.412.42 No.13.8476.8467.847.8289.8288.8289.359.3589.359.3588.3575.3584 No.14.7925.7559.7572.784.7839.784.388.3854.3872.3916.391.398 No.15.8665.8663.8665.868.8661.8667.3225.321.3218.3271.3217.325 No.16.8213.8189.8193.8373.8372.8372.3368.337.3342.3433.342.3425 No.17.8796.879.8793.8925.8874.8882.3423.3387.349.346.3448.3455 No.18.8535.8469.8472.8519.8518.8518.351.3497.351.3568.3515.3527 No.19.8283.8278.828.8376.8376.8376.3797.3765.3789.3834.3826.3831 No.2.7691.7556.7563.7769.7769.7769.3648.363.3643.3683.3677.3681 No.21.8496.8494.8495.8174.8173.8174.3291.3249.3271.3313.3293.333 No.22.8172.885.889.8118.8117.8117.3336.3321.3327.3382.3266.3323 No.23.8586.8574.8575.8698.869.8691.3733.3692.3721.3764.375.3758 No.24.8615.8614.8614.8788.8785.8785.3479.344.3467.3512.345.357 No.25.7997.7934.7937.8155.8154.8155.3738.3698.3727.377.3758.3764 No.26.8649.8647.8648.8626.8625.8625.3832.3811.3827.3858.3843.3851 No.27.8559.8551.8552.8458.8426.843.3211.319.322.3251.3237.3245 No.28.8539.8176.819.8419.8417.8418.3575.3417.3431.3476.346.3424 No.29.8273.8269.8271.8141.814.8141.315.366.396.3132.3117.3127 No.3.8728.8722.8723.8587.8584.8584.3422.3414.3417.3479.3437.3445 Table 9. The values of hypervolume indicator of obtained results on different parameter suites on the complex parallel-series modular software system (the result that is significantly better than the other is emphasized in boldface) parameter bi-objective problem tri-objective problem suite NSGA-II HaD-MOEA NSGA-II HaD-MOEA best worst mean best worst mean best worst mean best worst mean No.1.5581.5122.5459.5694.5559.5637.2281.144.265.2221.2113.214 No.2.5519.5183.543.5444.5245.5359.1969.1685.1833.1998.1935.1956 No.3.563.574.547.5678.5431.5591.1917.984.1497.1834.1385.16 No.4.5547.5239.542.5393.5256.5319.2231.1861.216.231.2275.2295 No.5.531.464.5121.5222.4927.5134.1629.816.139.175.1241.1521 No.6.5666.5446.5562.5616.5489.556.2278.1246.214.2279.1992.2151 No.7.5482.518.531.5477.5311.5394.179.1247.1622.1894.1781.1864 No.8.5537.5227.5443.5356.524.5293.2251.1137.216.2329.219.2296 No.9.5248.4723.566.598.4861.4997.1443.73.124.1541.142.151 No.1.5618.4928.553.5483.538.5422.2197.1848.288.2242.219.2182 No.11.5364.4999.5192.5416.5249.5348.1858.1289.161.1966.1365.1486 No.12.5471.543.5295.534.5175.5247.2125.1361.1943.2365.2132.221 No.13.5569.4743.5426.5446.5314.5385.2245.161.268.2578.1833.218 No.14.5716.5457.562.5654.5467.557.2362.1947.2139.2484.1979.2194 No.15.538.4484.5198.5389.5145.5284.1773.179.1577.1924.1489.1623 No.16.5669.5354.5531.578.5653.5723.2391.147.2262.2382.235.2364 No.17.5261.4671.594.5339.516.522.155.541.123.1943.1183.1389 No.18.5723.5473.56.5713.5536.5618.2189.1133.1771.2178.142.185 No.19.526.4889.5123.5264.4671.544.1369.415.1218.1666.1221.143 No.2.553.524.5371.5586.5387.551.215.1924.267.228.1844.1984 No.21.5589.5285.543.5377.527.53.2111.1449.1937.2342.1532.1977 No.22.511.4274.4967.576.4361.4842.1257.766.1117.1321.944.1242 No.23.5533.5127.5413.565.5453.5525.2255.1165.21.2599.182.2366 No.24.514.4226.4925.526.4728.537.1343.389.1168.1665.1198.1444 No.25.5569.5382.5498.568.5528.569.2144.189.265.2495.2.229 No.26.5489.5162.5341.5474.5175.5298.163.1391.1489.1795.131.1473 No.27.5538.525.5377.5473.5251.549.248.183.1983.2199.1924.277 No.28.5437.5188.5318.536.518.5227.188.964.1618.1967.1183.1432 No.29.5387.4656.521.531.54.5174.1589.831.1391.1633.135.1386 No.3.5184.4782.53.511.4566.4787.125.81.114.1463.114.1363

Multi-objective Approaches to Optimal Testing Resource Allocation in Modular Software Systems 15 Table 1. The values of hypervolume indicator of obtained results on different parameter suites on the larger parallel-series modular software system (the result that is significantly better than the other is emphasized in boldface) parameter bi-objective problem tri-objective problem suite NSGA-II HaD-MOEA NSGA-II HaD-MOEA best worst mean best worst mean best worst mean best worst mean No.1.7916.7426.7747.877.7712.7892.277.259.2662.2842.2553.2717 No.2.7949.7422.7771.7841.7377.7664.2835.2563.27.2889.2626.2768 No.3.7813.742.7638.7879.7354.7678.2618.228.2454.265.2296.259 No.4.7617.686.7326.746.6796.783.2458.2119.2326.2512.2293.248 No.5.758.588.7293.7468.6751.7128.2241.1779.274.2282.1851.2124 No.6.7361.4987.6692.7297.6696.73.233.149.1854.258.1828.1999 No.7.7649.6825.7362.7641.7125.7431.2279.1995.2175.2338.25.2219 No.8.845.7597.7875.7782.7418.7642.3235.2913.354.3257.316.3126 No.9.7744.6341.7482.7523.6925.7288.2422.2232.2318.2461.235.2394 No.1.7723.6659.746.7536.6882.7249.234.1865.212.2379.194.2168 No.11.7972.7373.7792.849.7664.792.344.2775.2916.3291.355.3183 No.12.7822.66639.7481.7583.746.7369.2637.2181.2482.2666.2268.2538 No.13.7785.6773.7497.7612.7244.745.253.2125.236.2555.218.2356 No.14.784.651.7585.7756.73.7559.2598.2382.2483.2633.2431.2535 No.15.7632.7178.7322.7646.6823.7282.2311.194.2186.2363.1959.2237 No.16.7789.7163.7493.7941.726.767.2484.288.2297.252.2124.2353 No.17.7826.7229.7653.7941.7417.7683.2547.211.2385.2633.216.2438 No.18.745.5542.6566.732.5869.6395.1496.1171.1394.1534.114.1425 No.19.7763.6278.7498.7851.7221.7541.23.242.2187.2394.292.2243 No.2.821.7652.7855.813.7689.7932.2934.2526.2739.296.2552.2795 No.21.766.6966.7439.737.6743.719.2424.219.22.257.268.2248 No.22.7695.6816.742.7644.6812.731.2232.187.289.229.1944.214 No.23.7668.6897.7476.7768.796.7459.2333.1893.2153.2426.1965.221 No.24.7578.6556.718.773.6918.7368.292.1632.1853.214.1651.1895 No.25.785.7295.7648.86.7559.7821.2751.248.2599.2769.2487.2651 No.26.7968.7399.772.7947.7579.7788.2694.234.2521.2747.2426.2584 No.27.7877.7342.7645.7785.7273.7667.2739.279.2445.2761.2232.2698 No.28.7945.7541.7796.7834.7396.7697.2956.2624.2845.357.2696.2922 No.29.7762.6759.7416.7639.726.7559.2548.211.2329.2566.2148.2382 No.3.787.684.7528.7743.737.755.254.2267.2419.2599.2343.2477 Table 11. The values of hypervolume indicator of obtained results on different parameter suites on the SS-system (the result that is significantly better than the other is emphasized in boldface) parameter bi-objective problem tri-objective problem suite NSGA-II HaD-MOEA NSGA-II HaD-MOEA best worst mean best worst mean best worst mean best worst mean No.1.4937.4873.4927.537.4647.4851.337.3274.3293.338.3222.331 No.2.4877.4856.4865.4811.4579.47.3322.3292.338.3423.3322.3367 No.3.4918.4889.491.4959.472.4859.3332.335.3317.3362.3216.3291 No.4.474.4452.4636.4574.4368.4457.3376.3312.3343.3585.3491.353 No.5.5111.585.594.535.4817.495.3441.344.3422.3449.3356.347 No.6.4816.4583.4791.4774.4553.4678.3353.3319.3338.3453.3356.343 No.7.4977.4948.4963.4972.4742.4889.3342.3314.3332.3336.324.3297 No.8.4756.4693.4748.46.4358.456.3229.321.322.339.3242.3278 No.9.4835.4587.4761.4697.4452.4572.3456.3432.3445.345.3375.3418 No.1.464.4442.4587.4528.4331.4434.3353.3311.3328.3343.3245.3294 No.11.4783.4759.4773.4829.4615.4728.3261.3228.3248.3283.3153.3226 No.12.55.465.4949.4896.4699.4791.351.3465.3485.3489.3395.346 No.13.4729.454.4668.4625.4365.454.334.3312.3326.3432.334.3393 No.14.481.423.4664.4758.46.478.3393.3363.3377.3491.335.3446 No.15.4928.465.4889.4937.4732.4836.358.3477.3498.3485.3426.3459 No.16.482.478.4794.4896.4653.4777.349.3372.3394.343.343.3412 No.17.4689.4324.4599.4758.4536.4659.3272.324.326.3262.315.3224 No.18.478.4418.474.4771.4573.4683.3377.3348.3364.3472.3388.343 No.19.476.4448.4697.4814.458.4633.3384.3358.3375.3388.338.3343 No.2.4725.4658.4684.4773.446.4669.3338.334.3319.3349.3325.3339 No.21.497.4513.4795.4722.4495.4631.3451.3421.3439.3539.3462.351 No.22.4636.447.4576.465.4388.4522.3313.3282.3299.343.3226.336 No.23.463.4299.4324.4663.448.4544.3324.3267.338.3332.3233.3291 No.24.53.4575.4823.514.4894.523.3436.345.3425.3524.3444.3492 No.25.475.4734.4744.4844.4679.476.3222.3184.3199.3224.3114.3178 No.26.4854.4527.4761.4841.4571.4698.3383.3357.3372.3472.3297.339 No.27.4698.4495.4671.4642.4363.4516.3349.3316.3334.335.3259.338 No.28.4987.4963.4976.4916.464.4785.3367.3343.3357.3375.3348.3351 No.29.479.475.4768.4714.446.4591.338.3349.3368.341.3291.3385 No.3.4577.4322.445.453.4245.4378.3289.3236.3267.3325.321.3248

16 Zai Wang, Ke Tang and Xin Yao Table 12. The overall comparison between HaD-MOEA and NSGA-II for the two multi-objective problems on four software systems (Twosided Wilcoxon rank-sum test with a significance level.5 was used). S, C, L and SS stand for simple, complex, larger and SS software systems, respectively. Results are presented in form of win-draw-lose, indicating the numbers of test cases on which the corresponding HaD-MOEA is superior, comparable (i.e., statistical insignificant), and inferior to the NSGA-II. HaD-MOEA vs NSGA-II bi-objective problem tri-objective problem S C L SS S C L SS 1-9-11 7-2-3 6-2-4 6-18-6 22-6-2 14-13-3 22-8- 15-13-2 obtained by an algorithm. Among these metrics, we adopted the hypervolume indicator in our study because it holds two important properties: 1. It is sensitive to any type of improvements, i.e., whenever an approximation set A dominates another approximation set B, the measure yields a strictly better quality value for the former than for the latter set [42]. 2. As a result of the first property, the hypervolume measure guarantees that any approximation set A that achieves the maximally possible quality value for a particular problem contains all the Pareto-optimal objective vectors [43]. So far, the hypervolume indicator is the only metric that possesses the above two properties. Hence, it serves as the most appropriate metric for comparing NSGA-II and HaD-MOEA. For space considerations, we refrain from providing the detailed information of hypervolume indicator but direct interested readers to [42]. Tables 8 to 11 present the results (in terms of the values of hypervolume indicator) obtained by the MOEAs algorithms in 3 independent runs. For each algorithm, the best, median and worst results are given. Two-sided Wilcoxon rank-sum tests with a significance level.5 have been conducted to compare the performance of NSGA-II and HaD-MOEA. Table 12 summarizes the results of the Wilcoxon test. When considering the bi-objective problems, it can be observed that HaD-MOEA performed comparably with NSGA-II on all four systems. This is consistent with the results presented in the previous subsection. However, in case of the tri-objective problems, HaD-MOEA is obviously superior to NSGA-II. For all 12 testing instances, HaD-MOEA performed significantly better than NSGA-II on 73 instances, while was inferior on only seven. To investigate the computational efficiency of the MOEAs, we recorded their runtime on all test instances. Table 13 presents the average runtime for each problem on the four systems. In comparison with NSGA-II, HaD-MOEA merely involved additional computational overhead of.4 seconds in the worst case, which was about 2% of the overall runtime of the whole algorithm. In essence, the computational efficiency of the two algorithms is comparable. Moreover, we may find that the runtime of both MOEAs are about 2 seconds for the bi-objective problems and 13 seconds for the tri-objective problems, which are negligible compared to the total time that we allocated to the testing phase. This further justifies the utility of MOEAs in practice. Table 13. Average runtime (in CPU seconds) of MOEAs on the four case studies bi-objective problem tri-objective problem NSGA-II HaD-MOEA NSGA-II HaD-MOEA simple system 2.7938 2.92 113.1183 113.5321 complex system 22.629 22.1821 124.621 124.681 larger system 22.993 23.1542 136.212 136.332 SS-system 24.784 24.3928 121.8521 121.968 5 Conclusions and Discussions In this paper, we formulated two types of OTRAPs as multi-objective optimization problems, one with two objectives and the other with three. Then, the two problems solved using MOEAs. Specifically, a novel MOEA called HaD-MOEA has been proposed since the state-of-the-art MOEA, namely NSGA-II does not perform satisfactorily on the second type of OTRAP. Empirical studies were first carried out to evaluate the utility of MOEAs in comparison with existing single-objective approaches. It was shown that MOEAs not only managed to achieve almost the same solution as that can be attained by single-objective approaches, but also found simultaneously a set of alternative solutions. These solutions showed different trade-off between the reliability of a system and the testing cost, and hence can facilitate informed planning of a testing phase. Later, a comprehensive experimental study was conducted to compare HaD-MOEA with NSGA-II on three parallel-series modular software systems and a star-structure modular software system, each with a large variety of randomly generated parameter settings. The

Multi-objective Approaches to Optimal Testing Resource Allocation in Modular Software Systems 17 purpose was to evaluate the algorithms on systems with different sizes, structures and parameters. The results showed that both algorithms performed well on the bi-objective problems, while HaD-MOEA performed significantly better than NSGA-II on the tri-objective problems, for which the total testing resource expenditure is not determined in advance. The superiority of HaD-MOEA consistently holds for all four tested systems. In addition to the above advantages, MOEAs may also be applied to real-world problems with little effort. Given a software testing task, a software tester only needs to formulate the reliability and cost as the functions of testing resource assigned to each modules of the system. Then, control parameters of the MOEAs can be set to those values used in this work, since they generally performed well in our empirical study. After that, the algorithm can obtain a set of solutions (i.e., plans of testing resource assignment) without human involvement in the search process. The very modest computational cost of the MOEAs (in comparison with the total testing resource) also supports its potential utility in real-world scenarios. Two directions may be further pursued in the future. First, the efficacy of MOEAs was only evaluated on parallel-series and star-structure modular software systems, and the objective functions were formulated based on the assumption of independence between modules and components. However, in the reliability literature, some software systems are modeled under the assumption that the modules and components are dependent with each other [28]. In this case, the objective functions (e.g., the reliability function) will be substantially different from those investigated in this work. It would be interesting to verify whether MOEAs can still work for such scenarios. Second, although HaD-MOEA was specifically designed to address the drawback of NSGA-II on the tri-objective problems, the improvement essentially arose from the consideration of multi-objective optimization. No domain-specific knowledge about OTRAPs was taken into account. However, it is commonly thought that incorporating domain-knowledge into an algorithm will further boost its performance on the specific problem. Hence, we will investigate designing novel search operators that are specifically suitable for OTRAPs in the future. Acknowledgment This work is partially supported by National Natural Science Foundation of China grants (No. 68236 and No. U8352), the Engineering and Physical Sciences Research Council (EPSRC Grant No. EP/D52785/1) and the Fund for Foreign Scholars in University Research and Teaching Programs (111 project) in China (No. B733). References 1. H. Ohtera and S. Yamada, Optimal Allocation and Control Problems for Software-Testing Resource, IEEE Transcations on Reliablity, vol. 39, no. 2, pp. 171 176, 199. 2. R. H. Huo, S. Y. Kuo, and Y. P. Chang, Efficient Allocation of Testing Reaources for Software Module Testing Based on the Hyper- Geometric Distribution Software Growth Model, IEEE Transcations on Reliablity, vol. 45, no. 4, pp. 541 549, 1996. 3. O. Berman and N. Ashrafi, Optimization Models for of Modular Software Systems, IEEE Transcations on Software Engineering, vol. 19, no. 11, pp. 1119 1123, 1993. 4. S. Yamada, T. Lehimori, and M. Nishiwaki, Optimal Allocation Policies for Testing Resource Based On a Software Growth Model, Mathemarical and Computer Modelling, vol. 22, pp. 295 31, 1995. 5. Y. W. Leung, Dynamic Resource Allocation for Software Module Testing, The Journal of the Systems and Software, vol. 37, no. 2, pp. 129 139, 1997. 6. D. W. Coit, Ecomonic Allocation of Test Times for Subsystem-Level Growth Testing, IIE Transcations on Business, vol. 3, no. 12, pp. 1143 1151, 1998. 7. D. W. Coit and A. Smith, Optimization of Series-parallel Systems Using a Genetic Algorithm, IEEE Transactions on, vol. 45, no. 2, pp. 254 26, 1996. 8. B. Yang and M. Xie, Testing-resource Allocation for Redundant Software Systems, in Proceedings of the 1999 Pacific Rim International Symposium on Dependable Computing (PRDC 99), Hong Kong, China, 1999, pp. 73 83. 9., A Study of Operational and Testing in Software Analysis, Engineering and System Safety, vol. 7, pp. 323 329, 2. 1., Optimal Testing-Time Allocation for Modular Systems, International Journal of Quality and Management, vol. 18, no. 8, pp. 854 863, 21. 11. C. Y. Huang and M. R. Lyu, Optimal Testing Resource Allocation, and Sensitivity Analysis in Software Development, IEEE Transactions on, vol. 54, no. 4, pp. 592 63, 25. 12. Y. S. Dai, M. Xie, K. L. Poh, and B. Yang, Optimal Testing-Resource Allocation with Genetic Algorithm for Modular software Systems, The Journal of Systems and Software, vol. 66, pp. 47 55, 23. 13. M. R. Lyu, S. Rangarajan, and A. P. A. V. Moorsel, Optimal Allocation of Testing Resources for Software Growth Modeling in Component-Based Software Development, IEEE Transactions on, vol. 51, no. 2, pp. 183 192, 22. 14. D. W. Coit, T. Jin, and N. Wattanapongsakorn, System Optimization with Component Estimation Uncertainty: A Multicriteria Approach, IEEE Transactions on, vol. 53, no. 3, pp. 369 38, 24.

18 Zai Wang, Ke Tang and Xin Yao 15. Y. S. Dai and X. L. Wang, Optimal Resource Allocation on Grid Systems for Maximizing Service Using A Genetic Algorithms, Engineering and System Safety, vol. 91, no. 9, pp. 171 182, 26. 16. C. Y. Huang and J. H. Lo, Optimal Resource Allocation for and of Modular Software Systems in the Testing Phase, The Journals of Systems and Software, vol. 79, pp. 653 664, 26. 17. P. Y. Yin and J. Y. Wang, Ant Colony Optimization for the Nonlinear Resource Allocation Problem, Applied Mathematics and Computation, vol. 174, pp. 1438 1453, 26. 18., A Particle Swarm Optimization Approach to the Nonlinear Resource Allocation Problem, Applied Mathematics and Computation, vol. 183, pp. 232 242, 26. 19. Y. S. Dai and G. Levitin, Optimal Resource Allocation for Maximizing Performance and in Tree-structured Grid Services, IEEE Transactions on, vol. 56, no. 3, pp. 444 453, 27. 2. Y. S. Dai, G. Levitin, and X. Wang, Optimal Task Partition and Distribution in Grid Service System with Common Cause Failures, Future Generation Computer Systems, vol. 23, no. 2, pp. 29 218, 27. 21. G. Levitin and Y. S. Dai, Optimal Service Task Partition and Distribution in Grid System with Star Topology, Engineering and System Safety, vol. 93, no. 1, pp. 153 16, 28. 22. S. J. Sadjadi and R. Soltani, An Efficient Heuristic Versus a Robust Hybrid Meta-heuristic for General Framework of Serial parallel Redundancy Problem, Engineering and System Safety, vol. 94, no. 11, pp. 173 171, 29. 23. Q. Long, M. Xie, S. Ng, and G. Levitin, Analysis and Optimization of Weighted Voting Systems with Continuous States Input, European Journal of Operational Research, vol. 191, no. 1, pp. 238 25, 28. 24. R. Tavakkoli-Moghaddam, J. Safari, and F. Sassani, Optimization of Series-parallel Systems with a Choice of Redundancy Strategies Using a Genetic Algorithm, Engineering and System Safety, vol. 93, no. 4, pp. 55 556, 28. 25. Z. Wang, K. Tang, and X. Yao, A Multi-objective Approach to Testing Reosurce Allocation in Modular Software Systems, in Proceedings of the 28 IEEE Congress on Evolutionary Computation (CEC28), Hongkong, China, 28, pp. 1148 1153. 26. K. Deb, A. Pratap, S. Agarwal, and T. Meyarivan, A Fast and Elitist Multiobjective Genetic Algorithm: NSGA-II, IEEE Transactions on Evolutionary Computation, vol. 6, no. 2, pp. 182 197, 22. 27. P. A. Tom and C. S. R. Murthy, Algorithms for -Oriented Module Allocation in Distributed Computing Systems, The Journal of System and Software, vol. 4, no. 2, pp. 125 138, 1998. 28. W. Kuo and R. Wan, Recent Advances in Optimal Allocation, IEEE Transcations on Systems, Man, and Cybernetics, Part A: Systems and Humans, vol. 37, pp. 143 156, 27. 29. C. A. C. Coello, Evolutionary Multi-objective Optimisation: An Historical View of the Field, IEEE Computational Intelligence Magazine, no. 1, pp. 28 36, 26. 3. A. Konak, D. W. Coit, and A. E. Smit, Multi-objective Optimization Using Genetic Algorithms: A Tutorial, Engineering and System Safety, vol. 91, no. 9, pp. 992 17, 26. 31. J. D. Schaffer, Multiple Objective Optimization with Vector Evaluated Genetic Algorithms, in Genetic Algorithms and their Applications: Proceedings of the First International Conference on Genetic Algorithms, Lawrence Erlbaum, 1985, pp. 93 1. 32. J. D. Knowles and D. W. Corne, Approximating the Nondominated Front Using the Pareto Archived Evolution Strategy, Evolutionary Computation, vol. 8, no. 2, pp. 149 172, 2. 33. N. Srinivas and K. Deb, Muiltiobjective Optimization Using Nondominated Sorting in Genetic Algorithms, Evolutionary Computation, vol. 2, no. 3, pp. 221 248, 1994. 34. J. Horn, N. Nafpliotis, and D. E. Goldberg, A Niched Pareto Genetic Algorithm for Multiobjective Optimization, in Proceedings of the First IEEE Conference on Evolutionary Computation, IEEE World Congress on Computational Intelligence, vol. 1, 27 29 Jun, Orlando, FL, USA, 1994, pp. 82 87. 35. C. M. Fonseca and P. J. Fleming, Genetic Algorithms for Multiobjective Optimization: Formulation, Discussion and Generalization, in Proceedings of the Fifth International Conference on Genetic Algorithms, San Mateo, California, 1993, pp. 416 423. 36. E. Zitzler and L. Thiele, Multiobjective Evolutionary Algorithms: A Comparative Case Study and the Strength Pareto Approach, IEEE Transactions on Evolutionary Computation, vol. 3, no. 4, pp. 257 271, 1999. 37. E. Zitzler, M. Laumanns, and L. Thiele, SPEA2: Improving the Strength Pareto Evolutionary Algorithm, in Evolutionary Methods for Design, Optimization and Control with Applications to Industrial Problems, K. Giannakoglou et al., Ed., Athens, Greece, 22, pp. 95 1. 38. E. Zitzler and L. Thiele, Comparison of Multiobjective Evolutionary Algorithms: Empirical Results, Evolutionary Computation, vol. 8, no. 2, pp. 173 195, 2. 39. S. Huband, P. Hingston, L. Barone, and L. While, A Review of Multiobjective Test Problems and a Scalable Test Problem Toolkit, IEEE Transactions on Evolutionary Computation, vol. 1, no. 5, pp. 477 56, 26. 4. C. M. Lin and M. Gen, Multiobjective Resource Allocation Problem by Multistage Decision-based Hybrid Genetic Algorithm, Applied Mathematics and Computation, vol. 187, no. 2, pp. 574 583, 27. 41. R. C. Purshouse and P. J. Fleming, On the Evolutionary Optimization of Many Conflicting Objectives, IEEE Transactions on Evolutionary Computation, vol. 11, no. 6, pp. 77 784, 27. 42. E. Zitler, L. Thiele, M. Laumanns, C. M. Foneseca, and V. G. da Fonseca, Performance Assessment of Multiobjective Optimizers: An Analysis and Review, IEEE Transactions on Evolutionary Computation, vol. 7, no. 2, pp. 117 132, 23. 43. M. Fleischer, The Measure of Pareto Optima: Applications to Multi-objective Metaheuristics, Inst Systems Research, Univ. Maryland, College Park, MD, Tech. Rep. ISR TR 22 32, 22.