4th International Conference on Software Methodologies, Tools and Techniques Approaches to Qualitative Evaluation of the Software Quality Attributes: Overview Presented by: Denis Kozlov Department of Computer Science and Information Systems, University of Jyväskylä, Finland September 29, 2005
Goal of the paper (I( I / II) The primarily goal of the software measurement is to measure software quality attributes as precisely as possible. There are only few software attributes that can be evaluated using the interval scale. The majority of the attributes is to be measured in the ordinal scale. In this case the measurement is usually performed based on the experts judgment.
Goal of the paper (II / II) The most frequently used approach to the qualitative measurement of complex entities and systems is considered to be Overall Integral Index. There are also some other methods for qualitative evaluation of entities that can be applied for the case of the software measurement, for instance, Fuzzy Analytic Hierarchy Process and Consensus Relation. The aim of the paper is to highlight the main points of the above approaches and to make a tentative comparative analysis of them. Some issues of the application of the approaches for the software measurement are discussed.
i i Overall Integral Index (I / II) The main principles were developed in 1980s by Russian scientist G.G. Azgaldov Implemented for different fields The entity or system to be evaluated should be characterized by a number of attributes, each of which is to be evaluated by experts. The tree of criteria (attributes) is to be developed. The tree of attributes should reflect all the attributes and subattributes to be evaluated.
Overall Integral Index (II / II) Weights of each attribute are to be defined. Each attribute should be evaluated from the viewpoint of its importance for the end-user. The sum of the weights for sub-attributes of an attribute should be 100% or 1. The generalized criteria is defined according to the formula If W1 > W2 then the entity (software) corresponding to the generalized criteria W1 is of higher quality than the second one.
Example (I / III) Let 4 software products be evaluated by 5 criteria: understandability, learnability, operability, attractiveness and compliance. For this purpose a group of 3 experts was organized. Each of 3 experts sets weights for attributes with the help of [n x n] matrix of attributes pair comparison. For instance, the matrix for group of the above attributes (let define them as x1, x2, x3, x4 and x5) has been made by the first expert: X 1 X 2 X 3 X 4 X 5 X 1 10 8 7 6 5 X 2 10 9 7 6 X 3 10 8 6 X 4 10 7 X 5 10
Example (II / III) Then, relative estimation of weight for each attribute are usually calculated by means of the matrix of pair comparisons in several ways: X 2 / X 1 X 3 / X 2 X 4 / X 3 X 5 / X 4 X 1 0,800 0,875 0,857 0,833 X 2 0,900 0,778 0,857 X 3 0,800 0,750 X 4 0,700 Average value 0,800 0,888 0,812 0,785 All the average values presented in the bottom row of the table are used to calculate normalized weight factors. For instance, mx3/x1 = 0.7; mx3/x2/x1= 0.888*0.8=0.71, etc. The average relative weights: Mx3/x1 = (mx3/x1+ mx3/x2/x1)/2 = 0.705 Mx4/x1= (0.6+0.577)/2 = 0.589; Mx5/x1=(0.5+0.453)/2=0.477. The obtained values are normalized.
Example (III / III) Further procedure is repeated for each of the quality attributes. Other experts make the same procedure in parallel. Average estimations of weights of indexes are normalized by division of each into the sum of estimations. For every attribute the average value is calculated by three experts. Normalized final weights of indexes made by three experts are as follows: mx1 = 0.27; ˆmx 2 = 0.24; ˆmx 3 = 0.19; ˆmx 4 = 0.16; ˆmx 5 = 0.13. A value of the quality attribute W1 for the first software is W1 = 10 0.27+8 0.24 + 7 0.19+6 0.16+ 5 0.13 = 7.56 The same procedure should be repeated also for the other quality attributes and software products.
Analytic Hierarchy Process (I / II) The method is a regular procedure for hierarchical representation elements (a tree of criteria), determining features of software In order to establish relative importance of elements in hierarchy, the scale of relations is used. Value 1 is given to the relation of the objects, which have identical importance when compared in pairs. Value 9 is given to the relation of the objects when one object is superior to the other. Values 3, 5, 7 are used in interim situation when compared in pairs. In case when the compromise is necessary, values 2, 4, 6, 8 can be used.
1 Analytic Hierarchy Process (II / II) The given scale enables a Decision Maker to bring some numbers to conformity with the degrees of preference of one object under comparison to another. The method provides estimation of rejection degree from consistency. When such deviations exceed the established limits, the decision maker should recheck them in a matrix of pair comparisons. Criticism of Analytic Hierarchy Process. It is very difficult for the decision maker to express the strength of his preferences and to provide exact pairwise comparison judgment.
1 Fuzzy Analytic Hierarchy Process In case of the Fuzzy Analytic Hierarchy process the comparison ratios are expressed as fuzzy sets or fuzzy numbers, which incorporate the vagueness of the human thinking. When comparing any two objects Ei and Ej at the same level of the decision hierarchy, the uncertain comparison judgment can be represented by the fuzzy number ãij ãij = (lij; mij; uij), where lij, mij and uij the lower, mean and upper bounds, respectively; It is assumed that lij < mij < uij.
1 Fuzzy Analytic Hierarchy Process The membership function that represents the decisionmaker s satisfaction with different crisp solution ratios ŵi / ŵj: The maximum prioritization problem can be represented in the following way:
1 Example (I / II) Let s consider that three quality attributes are measured. The fuzzy comparison judgments are shown in the table: Understandability Learnability Operability Understandability 1 (2, 3, 4) (1, 2, 3) Learnability (1/4, 1/3, ½) 1 (1/3, 1/2, 1) Operability (1/3, 1/2, 1) (1, 2, 3) 1 From the table it is seen that Understandability is considered as the most important attribute, since all the fuzzy numbers in the first row are greater than one. For example, Understandability is assessed as being about three times more important than Learnability and about two times more important than Operability.
1 Example (II / II) Since the fuzzy pairwise comparison matrix is reciprocal, only the elements of the upper right part are used for the calculation of weights. The weights can be obtained by resolving a non-linear equation
1 Consensus Relation (I / III) Let some n software products will be characterized by m criterion (attributes). Let a set A = {a1, a2,..., an}, A = n, be a set of software products. By every criterion (attribute), the software product can be ranked in order of preference. Every ranking (preference relation ) = {a1 a2...~ as ~...~ an} includes, a strict preference relation, and ~, an indifference relation. Such the relation is known to be called preorder. A single preference relation can be determined that would give an integrative characterization of the software products. Let a space be a set of all n! strict (linear) order relations on A. Each linear order corresponds to one of permutations of first n natural numbers Nn. We will consider a permutation of the software products a1,..., an to represent the preference profile and will call it consensus ranking. It is desirable that, in some sense, would be nearest to the every of rankings
1 Consensus Relation (II / III) Finding the consensus ranking is possible due to measure of distance between pairs of rankings firstly introduced by Kemeny The ranking can be represented by an (n n) relation matrix [aij] whose rows and columns are labeled by the software products and properties of the relation matrix are connected to those of the corresponding relation The Kemeny distance function d( k, l) between two rankings k and l is defined by formula
1 Consensus Relation (III / III) Like in the case of the relation matrix, we can define an (n n) profile matrix P = pij which can represent in compact form all the rankings
1 Comparison of approaches Characteristic of the method Analytic Hierarchy Process Fuzzy Analytic Hierarchy Process Overall Integral Index Consensus Relation Scaling Ratio; priorities Ratio; fuzzy defined priorities Order; priorities Order Preference elicitation Pairwise comparison Pairwise comparison of fuzzy numbers Pairwise comparison Distance between rankings Weighting of particular attributes Normalized ratio via eigenvalues Normalized ratio via fuzzy eigenvectors Normalized relative attributes No Synthesis of resulting estimation Additive, eigenvectors Additive, fuzzy eigenvector Additive, multiplicative Linear order relation as consensus Description of structure of objects in question Hierarchic Hierarchic Hierarchic Not important
1 Conclusions The most accurate approach from the above ones is the Fuzzy Analytic Hierarchy Process; however, this is also the most complex method. In many cases the sub-optimal approach to be chosen for the qualitative evaluation seems to be the Consensus Relation, since it gives an acceptable level of accuracy. Finally, we would like to stress that one of the considerable disadvantages of the above approaches is that they all require at least two software products to be evaluated. If there is only one software product available for evaluation, these approaches can not be implemented. In this case it is reasonable to use other methods like Heuristic Analysis etc.
2 Future work Evaluation of a real software instead of simulation Politec Brazilian company Korppi University of Jyväskylä Taking into account possible errors errors associated with the process of picking up experts for evaluation (the issue of the competence measuring); inconsistencies associated with the sufficient number of experts; errors related to the process of organization of the experts work; errors associated with the validation of the evaluation results; errors occurred due to incomplete data used by the evaluation