Ranking Irregularities When Evaluating Alternatives by Using Some ELECTRE Methods

Transcription

1 Please use the followig referece regardig this paper: Wag, X., ad E. Triataphyllou, Rakig Irregularities Whe Evaluatig Alteratives by Usig Some ELECTRE Methods, Omega, Vol. 36, No. 1, pp , February Rakig Irregularities Whe Evaluatig Alteratives by Usig Some ELECTRE Methods Xiaotig Wag Departmet of Idustrial Egieerig 3128 CEBA Buildig, Louisiaa State Uiversity Bato Rouge, LA 70803, U.S.A. Webpage: Evagelos Triataphyllou Departmet of Computer Sciece 298 Coates Hall, Louisiaa State Uiversity Bato Rouge, LA 70803, U.S.A. Webpage: Abstract: The ELECTRE II ad III methods ejoy a wide acceptace i solvig multi-criteria decisio-makig (MCDM) problems. Research results i this paper reveal that there are some compellig reasos to doubt the correctess of the proposed rakigs whe the ELECTRE II ad III methods are used. I a typical test we first used these methods to determie the best alterative for a give MCDM problem. Next we radomly replaced a o-optimal alterative by a worse oe ad repeated the calculatios without chagig ay of the other data. Our computatioal tests revealed that sometimes the ELECTRE II ad III methods might chage the idicatio of the best alterative. We treat such pheomea as rak reversals. Although such rakig irregularities are well kow for the additive variats of the AHP method, it is the very first time that they are reported to occur whe the ELECTRE methods are used. These two methods are also evaluated i terms of two other rakig tests ad they failed them as well. Two real-life cases are described to demostrate the occurrece of rak reversals with the ELECTRE II ad III methods. Based o the three test criteria preseted i this paper, some computatioal experimets o radomly geerated decisio problems were executed to test the performace of the ELECTRE II ad III methods ad a examiatio of some real-life case studies are also discussed. The results of these examiatios show that the rates of the three types of rakig irregularities were rather sigificat i both the simulated decisio problems ad the real-life cases studied i this paper. Keywords: Multi-criteria decisio-makig, rakig irregularities, ELECTRE methods, the Aalytic Hierarchy Process (AHP), multiplicative AHP. 1. Itroductio Multi-criteria decisio-makig (MCDM) is oe of the most widely used decisio methodologies i the scieces, busiess, govermet ad egieerig worlds. MCDM methods ca help to improve the quality of decisios by makig the decisio-makig process more explicit, ratioal, ad efficiet. It is ot a coicidece that a simple search (for istace, by usig google.com) o the web uder the key words multi criteria decisio 1

2 makig returs more tha oe millio hits. Some applicatios of MCDM i egieerig iclude the use o flexible maufacturig systems [Wabalickis, 1988], layout desig [Cambro ad Evas, 1991], itegrated maufacturig systems [Putrus, 1990], ad the evaluatio of techology ivestmet decisios [Boucher ad Mcstravic, 1991]. The typical MCDM problem is cocered with the task of rakig a fiite umber of decisio alteratives, each of which is explicitly described i terms of differet characteristics (also ofte called attributes, decisio criteria, or objectives) which have to be take ito accout simultaeously. Usually, the performace values a ij ad the criteria weights w j are viewed as the etries of a decisio matrix defied as i Figure 1. The a ij elemet of the decisio matrix represets the performace value of the i-th alterative i terms of the j-th criterio. The w j represets the weight of the j-th criterio. Data for MCDM problems ca be determied by direct observatio (if they are easily quatifiable) or by idirect meas if they are qualitative [Triataphyllou, et al., 1994]. C r i t e r i a C 1 C 2... C (w 1 w 2... w ) Alteratives a 11 a a 1 a 21 a a A m a m1 a m2... a m Figure 1. Structure of a Typical Decisio Matrix. Aother term that is also used frequetly to mea the same type of decisio models is multi-criteria decisio aalysis (MCDA). There is a subtle differece betwee these two terms. The term MCDM is ofte used to mea fidig the best alterative i a cotiuous eviromet. However, i the settig of MCDA, the alteratives are ot kow a priori but they ca be determied by calculatig the values of a umber of discrete ad/or cotiuous variables. Usually, a MCDA method aims at oe of the followig four goals, or problematics [Roy, 1985], [Jacquet-Lagreze ad Siskos, 2001]: Problematic 1: Fid the best alterative. Problematic 2: Group the alteratives ito well-defied classes. Problematic 3: Rak the alteratives i order of total preferece. Problematic 4: Describe how well each alterative meets all the criteria simultaeously. May iterestig aspects of MCDA theory ad practice are discussed i [Hobbs, 2000; ad 1986], [Hobs, et al., 1992], [Stewart, 1992], [Triataphyllou, 2000], [Zaakis, et al., 1995], ad [Zaakis, et al., 1998]. However, the terms MCDM ad MCDA may also be used to deote the same class of models. A promiet role i MCDM methods is played by the Aalytic Hierarchy Process (AHP) method which is based o pairwise comparisos as it was proposed by Saaty [1980 ad 1994]. Accordig to that method the decisio maker compares two decisio etities (pair of alteratives cosidered i terms of a sigle criterio or a pair of criteria) at a time ad elicits his/her judgmet with the help of a scale. Such a scale assigs umerical values to liguistic expressios ad later these umerical values are aalyzed mathematically ad the a ij values are determied. May methods have bee proposed to aalyze the data of a decisio matrix ad rak the alteratives. Ofte times differet MCDM methods may yield differet aswers to exactly the same problem [Triataphyllou, 2000]! Some of the methods use additive formulas to compute the fial priorities of the alteratives. Represetatives of such methods are the weighted sum model (WSM) [Fishbur, 1967], ad the Aalytic Hierarchy Process (AHP) [Saaty, 1980 ad 1994] ad its variats (such as the Revised or Ideal Mode AHP [Belto ad Gear, 1983]). Some multiplicative versios of these methods have also bee developed. Examples are the weighted product 2

3 model (WPM) [Bridgma, 1922; Miller ad Starr, 1969] ad a later versio of it; the multiplicative AHP [Barzilai ad Lootsma, 1994; Lootsma, 1999]. I some earlier research it was foud that the previous additive models might ofte exhibit cases of irregular rakig reversals uder certai tests [Triataphyllou, 2000 ad 2001; Triataphyllou ad Ma, 1989; Belto ad Gear, 1983]. However, the previous multiplicative models are immue to most of these rakig irregularities. Aother family of MCDM models uses what is kow as outrakig relatios to rak a set of alteratives. A promiet role i this group is played by the ELECTRE method ad its derivatives. The ELECTRE approach was first itroduced i [Beayou, et al., 1966]. It is a comprehesive evaluatio approach i that it also tries to rak a umber of alteratives each oe of which is described i terms of a umber of criteria. The mai idea is the proper utilizatio of what is called outrakig relatios. Soo after the itroductio of the first versio kow as ELECTRE I [Roy, 1968], this approach has evolved ito a umber of other variats. Today the most widely used versios are kow as ELECTRE II [Roy ad Bertier, 1971, 1973] ad ELECTRE III [Roy, 1978]. Aother variat of the ELECTRE approach is the TOPSIS method [Hwag ad Yoo, 1981]. The TOPSIS method was also foud to suffer of the rakig irregularities related to the AHP ad its additive variats (accordig to some upublished results by the authors). I cotrast to the above approaches, there is a differet type of aalysis based o value fuctios. These methods use a umber of trade-off determiatios which form what are kow as value fuctios [Kirkwood, 1997]. A value fuctio attempts to map chages of values of performace of the alteratives i terms of a give criterio ito a dimesioless value. Some key assumptios are made i the process for trasferrig chages i values ito these dimesioless quatities [Kirkwood, 1997]. The roots to this type of aalysis ca be foud i [Edwards, 1977], [Edwards ad Barro, 1994], [Edwards ad Newma, 1986], ad [Dyer ad Sari, 1979]. Of the above MCDM methods a few stad out as beig the most widely used, for example, the AHP ad the ELECTRE approaches. Cases of rakig irregularities whe the AHP is used have bee reported by may researchers for a umber of years [Belto ad Gear, 1983; Dyer ad Wedell, 1985; Triataphyllou, 2000 ad 2001]. The ELECTRE method is a well kow method, especially i Europe. It has bee widely used i civil ad evirometal egieerig [Hobbs ad Meier, 2000]. Applicatios iclude the assessmet of complex civil egieerig projects, selectio of highway desigs, site selectio for the disposal of uclear waste, water resources plaig [Raj, 1995] ad waste water [Roger, et al., 1999] or solid waste maagemet [Hokkae ad Salmie, 1997a] etc. However, to the best of our kowledge, this is the first time that rakig irregularities are reported for the ELECTRE approach. This paper is orgaized as follows. The ext sectio discusses the three test criteria that have bee used i this study to test the performace of the ELECTRE II ad III methods. The third sectio describes two examples that are based o two real-life decisio problems for which rak reversals occurred uder test criterio #1 by usig the ELECTRE II ad III methods. The fourth sectio presets some empirical results o radomly geerated decisio problems accordig to the three test criteria described i sectio 2. The fifth sectio discusses the results based o the test criteria of some real-life case studies. Some cocludig commets are preseted i the last sectio. 2. Some Test Criteria A itriguig problem with decisio-makig methods which rak a set of alteratives i terms of a umber of competig criteria is that oftetimes differet methods may yield differet aswers (rakigs) whe they are fed with exactly the same umerical data. Thus, the issue of evaluatig the relative performace of such methods is aturally raised. This, i tur, raises the questio how ca oe evaluate the performace of such methods? Sice it is practically impossible to kow which oe is the best alterative for a give decisio problem, some kid of testig procedures eed to be determied. The above subject, alog with some rak irregularity issues, has bee discussed i detail i [Triataphyllou, 2000 ad 2001]. I those studies, three test criteria were established to test the relative performace of various MCDM methods. These test criteria are as follows: 3

4 Test Criterio #1: A effective MCDM method should ot chage the idicatio of the best alterative whe a o-optimal alterative is replaced by aother worse alterative (give that the relative importace of each decisio criterio remais uchaged). Suppose that a MCDM method has raked a set of alteratives i some way. Next, suppose that a o-optimal alterative, say A k, is replaced by aother alterative, say A k /, which is less desirable tha A k. The, accordig to the test criterio #1 the idicatio of the best alterative should ot chage whe the alteratives are raked agai by the same method. The same should also be true for the relative rakigs of the rest of the uchaged alteratives. Test Criterio #2: The rakigs of alteratives by a effective MCDM method should follow the trasitivity property. Suppose that a MCDM method has raked a set of alteratives of a decisio problem i some way. Next, suppose that this problem is decomposed ito a set of smaller problems, each defied o two alteratives at a time ad the same umber of criteria as i the origial problem. The, accordig to this test criterio all the rakigs which are derived from the smaller problems should satisfy the trasitivity property. That is, if alterative is better tha alterative, ad alterative is better tha alterative A 3, the oe should also expect that alterative is better tha alterative A 3. The third test criterio is similar to the previous oe but ow oe tests for the agreemet betwee the smaller problems ad the origial u-decomposed problem. Test Criterio #3: For the same decisio problem ad whe usig the same MCDM method, after combiig the rakigs of the smaller problems that a MCDM problem is decomposed ito, the ew overall rakig of the alteratives should be idetical to the origial overall rakig of the u-decomposed problem. As before, suppose that a MCDM problem is decomposed ito a set of smaller problems, each defied o two alteratives ad the origial decisio criteria. Next suppose that the rakigs of the smaller problems follow the trasitivity property. The, accordig to this test criterio whe the rakigs of the smaller problems are all combied together, the ew overall rakig of the alteratives should be idetical to the origial overall rakig before the problem decompositio. We used these three test criteria to evaluate the performace of the ELECTRE II ad the ELECTRE III methods. Both of them failed i terms of each oe of these three test criteria. Next we demostrate two rak reversal examples which occurred with the ELECTRE II ad III methods uder the first test criterio. The other two test criteria were also applied as they had bee stated above. 3. Illustratio of Rak Reversals with ELECTRE II ad III For most ELECTRE methods, there are two mai stages. These are the costructio of the outrakig relatios ad the exploitatio of these relatios to get the fial rakig of the alteratives. Differet ELECTRE methods may be differet i how they defie the outrakig relatios betwee alteratives ad how they apply these relatios to get the fial rakig of the alteratives. This is true with the ELECTRE II ad III methods. However, the essetial differece betwee these two methods is that they use differet types of criteria. ELECTRE II uses the true criteria where o thresholds exist ad the differeces betwee criteria scores are used to determie which alterative is preferred. I this preferece structure, the idifferece relatio is trasitive [Rogers, et al., 1999]. The criteria used by ELECTRE III are pseudo-criteria which ivolve the use of two-tiered thresholds. Oe is the idifferece threshold q, beeath which the decisio maker shows clear idifferece, ad the other oe is the preferece threshold p, above which the decisio maker is certai of strict preferece [Rogers, 4

5 et al., 1999]. The situatio betwee the above two is regarded as weak preferece for alterative a over alterative b which idicates the decisio maker s hesitatio betwee idifferece ad strict preferece [Rogers, et al., 1999]. The followig two rak reversal examples demostrate how both of the two methods work ad how the rak reversals may happe whe usig them to rak a set of decisio alteratives. 3.1 A Example of Rak Reversal with the ELECTRE II Method This example is based o a real-life case study where the ELECTRE II method was used to help fid the best locatio for a wastewater treatmet plat i Irelad [Rogers, et al., 1999]. The decisio problem is defied o 5 alteratives ad 7 criteria. Note that here all the criteria are beefit criteria, that is, the higher the score the better the performace is. The decisio matrix, that is, the performaces of the alteratives A i i terms of the criteria C j, is as follows: C 1 C 2 C 3 C 4 C 5 C 6 C 7 A The weights of the criteria are: C 1 C 2 C 3 C 4 C 5 C 6 C 7 Weight The ELECTRE methods are based o the evaluatio of two idices, the cocordace idex ad the discordace idex, defied for each pair of alteratives. The cocordace idex for a pair of alteratives a ad b measures the stregth of the hypothesis that alterative a is at least as good as alterative b. The discordace idex measures the stregth of evidece agaist this hypothesis [Belto ad Stewart, 2001]. There are o uique measures of cocordace ad discordace idices. I ELECTRE II, the cocordace idex C(a, b) for each pair of alteratives (a, b) is defied as follows: w i Q( a, b) i Cab (, ), m w = = where Q (a, b) is the set of criteria for which a is equal or preferred to (i.e., at least as good as) b, ad w i is the weight of the i-th criterio. For istace, the cocordace idices for this example are as follows: A 3 A 3 i The discordace idex D (a, b) for each pair of alteratives (a, b) is defied as follows: max[ gi( b) gi( a)] i Dab (, ) =, δ gi ( a) represets the performace of alterative a i terms of criterio Ci, gi ( b) represets the δ = max g ( b) g ( a) (i.e., the maximum Where performace of alterative b i terms of criterio Ci, ad i i i differece o ay criterio). This formula ca oly be used whe the scores for differet criteria are comparable. Whe the above formula is used, it turs out that the discordace idices for this example are as follows: i 5

6 A 3 A After computig the cocordace ad discordace idices for each pair of alteratives, two types of outrakig relatios are built by comparig these idices with two pairs of threshold values: (C *, D * ) ad (C,D ). The pair (C *, D * ) is defied as the cocordace ad discordace thresholds for the strog outrakig relatio ad the pair (C, D ) is defied as the thresholds for the weak outrakig relatio where C * > C ad D * < D. Next the outrakig relatios are built accordig to the followig two rules: (1) If C(a, b) C *, D(a, b) D * ad C(a, b) C(b, a), the alterative a is regarded as strogly outrakig alterative b. (2) If C(a, b) C, D(a, b) D ad C(a, b) C(b, a), the alterative a is regarded as weakly outrakig alterative b. The values of (C *, D * ) ad (C, D ) are decided by the decisio makers for a particular outrakig relatio. They may be varied to yield more or less severe outrakig relatios. The higher the value of C * ad the lower the value of D *, the more severe the outrakig relatio becomes. That is, the more difficult it is for oe alterative to outrak aother [Belto ad Stewart, 2001]. For this example, two pairs of thresholds for the strog outrakig relatio ad oe pair of thresholds for the weak outrakig relatio were chose to be as follows: C * 1 =0.85, D * 1 =0.50; C * 2 =0.75, D * 2 =0.25; ad C =0.65, D =0.25. Accordig to the above rules ad the three pairs of thresholds, the outrakig relatios for this example were derived to be as follows: A 3 A 3 S F S F S F S F S F S F S F S F I the above otatio S F stads for the strog outrakig relatio. For example, S F meas that alterative strogly outraks alterative. We use S f (i.e., the superscript ow is low case f ; ot preset o the above table) to stad for the weak outrakig relatio. The weak outrakig relatio would happe later i this example. O the basis of the outrakig relatios, ext the descedig ad ascedig distillatio processes are applied to obtai two complete pre-orders of the alteratives. The details of the distillatio processes ca be foud i [Belto ad Stewart, 2001] ad [Rogers, et al., 1999]. The descedig pre-order is built up by startig with the set of best alteratives (those which outrak other alteratives) ad goig dowward to the worse oe. O the cotrary, the ascedig pre-order is built up by startig with the set of worst alteratives (those which are outraked by other alteratives) ad goig upward to the best oe. The distillatio results for this example are as follows: the pre-order from the descedig distillatio is = > A 3 > > ; the pre-order from the ascedig distillatio is > = A 3 > >. The last step is to combie the two complete pre-orders to get either a partial or a complete fial pre-order. Whether the fial product is a partial pre-order (ot cotaiig a relative rakig of all of the alteratives) rather tha a complete pre-order depeds o the level of cosistecy betwee the rakigs from the two distillatio procedures [Rogers, et al., 1999]. The partial pre-order allows two alteratives to remai icomparable without affectig the validity of the overall rakig, which differetiates from the complete pre-order. A commoly used method for determiig the fial pre-order is to take the itersectio of the descedig ad ascedig pre-orders. The itersectio of the two pre-orders is defied such that alterative a outraks alterative b if ad oly if a outraks or is i the same class as b accordig to the two pre-orders. If alterative a is preferred to alterative b 6

7 i oe pre-order but b is preferred to a i the other oe, the the two alteratives are icomparable i the fial pre-order [Rogers, et al., 1999]. By followig the above rules, the itersectio of the two pre-orders for this example resulted i the followig complete pre-order of the alteratives: > > A 3 > > ad obviously is the optimal alterative at this poit. Next, alterative A 3 was radomly selected to be replaced by a worse oe, say A / 3, i order to test the stability of the alteratives rakig uder the first test criterio. The ew decisio matrix ow is as follows: C 1 C 2 C 3 C 4 C 5 C 6 C / A / Please ote that alterative A 3 is replaced by a less desirable oe deoted as A 3 which is determied by subtractig the value 1 from the performace values of the origial alterative A 3 (the subtracted value was selected radomly by a computer program to make sure it will cause the chose alterative to become worse tha the oe it replaces). The rest of the data are kept the same as before. The itermediate results durig the rakig process are as follows: The cocordace idices are: A 3 / The discordace idices are: A 3 / / A / A The outrakig relatios are: A 3 / A 3 / S F S F S F S F S f S F S F where S f (i locatio (3, 4)) stads for the weak outrakig relatio. Whe the descedig ad ascedig distillatio processes are applied agai, the descedig pre-order ow is = > A 3 = > while the ascedig pre-order is = >A 3 = >. After combiig the two pre-orders together, a ew complete pre-order is got as follows: = >A 3 = >. Now the best raked alteratives are ad together, a cotradictio from the previous result that had as the oly optimal alterative. 7

8 3.2 A Example of Rak Reversal with the ELECTRE III Method This illustrative example is also based o a real-life decisio problem which was defied o 11 alteratives ad 11 decisio criteria. The goal of this case is to choose the best waste icieratio strategy for the easter Switzerlad regio [Rogers, et al., 1999]. I this example, the first test criterio reveals a case of rak reversal whe the ELECTRE III method is used. The mai data are as follows: C 1 C 2 C 3 C 4 C 5 C 6 C 7 C 8 C 9 C 10 C 11 A 3 A 6 A 7 A 8 A , , , , , , , , , Please ote that i this example, criteria C 2, C 6 ad C 7 are beefit criteria, which meas the higher the score of a give criterio is, the more preferable it is. The other criteria are cost criteria, which meas the lower the score of a give criterio is, the more preferable it is. The weights W, the idifferece thresholds Q ad the preferece thresholds P of the criteria are as follows: C 1 C 2 C 3 C 4 C 5 C 6 C 7 C 8 C 9 C 10 C 11 W Q ± 1,000 10% 10% ± 5 10% 10% 10% ± ± 0 ± 0 P ± 2,000 20% 20% ± 10 20% 20% 20% ± ± 1 ± 1 *Please ote that i this example, o veto threshold is specified as it is the case i the refereced paper. Next the cocordace idex C i (a, b) calculated for each pair of alteratives (a, b) i terms of each oe of the decisio criteria accordig to the followig formula: 1, if zi( a) + qi( zi( a )) zi( b) Ci ( a, b) = 0, if zi( a) + pi( zi( a )) zi( b) (3-1) or by liear iterpolatio betwee 0 ad 1 whe z i (a)+q i (z i (a))<z i (b)<z i (a)+p i (z i (a)), where q i (.) ad p i (.) are the idifferece ad preferece threshold values for criterio C i [Belto ad Stewart, 2001]. For istace, the cocordace idices i terms of the first decisio criterio, which is C 1 (a, b), are as follows: A 3 A 6 A 7 A 8 A A A A A A The ext step is to calculate the discordace idex D i (a, b) for all the alteratives i terms of each oe of the decisio criteria accordig to the followig formula: 8

9 0, if zi( b) zi( a) + pi( zi( a )) Di ( a, b) = (3-2) 1, if zi( b) zi( a) + ti( zi( a )) or by liear iterpolatio betwee 0 ad 1 whe z i (a)+p i (z i (a))<z i (b)<z i (a)+t i (z i (a)), where t i (.) is the veto threshold for criterio C i [Belto ad Stewart, 2001]. If o veto threshold is specified, the D i (a, b)=0 for all pairs of alteratives. For istace, i this example, sice o veto thresholds are specified, the discordace idices i terms of each decisio criterio are all equal to zero. The ext step is to calculate the overall cocordace idex C (a, b) of all the alteratives by applyig the followig formula: m wc ( a, b) i= 1 i i (, ). m Cab = = i 1 w i Fially, the credibility matrix S (a, b) of all the alteratives is calculated by applyig the followig formula: Cab (, ), if Di ( ab, ) Cab (, ) i Sab (, ) = (1 D(, )), i a b (3-4) Cab (, ), otherwise i J( a, b) (1 Ci ( a, b)) where J (a, b) is the set of criteria for which D i (a, b)>c (a, b). The credibility matrix is a measure of the stregth of the claim that alterative a is at least as good as alterative b. For this case, the credibility matrix is equal to the cocordace matrix sice o veto thresholds are assiged so the discordace matrices are all zero matrices, which results to S (a, b) = C (a, b) ad both are as follows: A 3 A 6 A 7 A 8 A A 3 A 6 A 7 A 8 A (3-3) Next the descedig ad ascedig distillatios [Belto ad Stewart, 2001; Rogers, et al., 1999] based o the credibility matrix are applied to costruct two pre-orders for the alteratives. The pre-order obtaied from the descedig distillatio is as follows: A 9 > > A 7 > 0 > A 3 = = A 8 = 1 > > > A 6. The pre-order obtaied from the ascedig distillatio is as follows: = A 7 > A 9 > > 0 > = > A 3 = 1 > A 8 > A 6. The the two pre-orders are combied to get the fial overall rakig of the alteratives as show i Figure 2. The way to combie the two pre-orders is the same as that of ELECTRE II, which has bee described i the first example. The arrow lie i Figure 2 meas outrak. For example, A 9 outraks. Two alteratives are icomparable if there is o direct or idirect arrow lie to lik them together. For example, A 7 ad A 9 are icomparable. Alteratives are idifferet if they are at the same level. For example, A 3 ad 1 are idifferet with each other. We ca see ow that A 7 ad A 9 are both located at the top level ad they are icomparable with each other. Icomparability may be caused by the lack of the criterio iformatio of the alteratives. This meas that there is o clear evidece i favor of either A 7 or A 9. I real-life applicatios of ELECTRE II ad III methods, the decisio aalysts will eed to fid more iformatio about such alteratives ad do a further study to decide which oe is the best oe. However, for the simplicity of the curret test, A 7 ad A 9 are both regarded as the best-raked alteratives because both of them are raked first i the fial partial pre-order. As a result, the rest of the alteratives were regarded as the o-optimal oes. 9

10 Next, accordig to the first test criterio, we radomly selected oe of the o-optimum alteratives; say alterative to be replaced by a worse oe A / 1 to test the reliability of the alteratives rakig. Sice the performace value of i terms of each criterio was [ ], we subtracted [-2, ] (the subtracted value was selected radomly by a computer program to make sure it will make the chose alterative to become worse tha before) from to get A / 1 which is less desirable tha. Please recall that the first criterio is a cost criterio which meas the bigger a score the less desirable it is. So the performace values of A / 1 are [2, ]. The ew decisio matrix with the a ij values of the alteratives (after alterative is replaced by A / 1 ) is as follows: C 1 C 2 C 3 C 4 C 5 C 6 C 7 C 8 C 9 C 10 C 11 / A 3 A 6 A 7 A 8 A , , , , , , , , , , The rest of the data are kept the same. Whe the previous steps are applied o the modified problem agai, we get that the descedig pre-order is A 7 > A 9 > > 0 > A 3 = = A 8 = 1 > > > A 6 ad the ascedig pre-order is A 7 > =A 9 > > 0 > = > A 3 = 1 > A 8 > A 6. The overall rakig of the alteratives is as show i Figure 3. This time it turs out that the best-raked alterative ow is oly A 7 which is differet from the origial coclusio which had A 7 ad A 9 as the best-raked alteratives. A 7 A 7 A 9 A A 3, 1 A 3, 1 A 8 A 8 A 6 Figure 2. Rakig for the origial example. A 6 Figure 3. Rakig for the chaged example. Why did the above cotradictios occur? Whe was replaced by a worse oe, it is reasoable to assume that some alteratives which origially are raked lower tha may become more preferable tha it. However, there is o legitimate reaso why the optimal alterative should also be chaged ad why the origial icomparable relatio betwee two equally raked alteratives should also be chaged. 10

11 4. A Aalysis of the Causes of the Rak Reversals uder the ELECTRE II ad III Methods After aalyzig the rakig processes of the ELECTRE II ad III methods ad some rak reversal cases which occurred whe these methods were used, it was foud that the mai reaso for the above rak reversals lies i the exploitatio of the pairwise outrakig relatios. That is, the upward ad dowward distillatio processes of ELECTRE II ad ELECTRE III. The basic idea behid the distillatio processes is to decide the rak of each alterative by the degree of how this alterative outraks all the other alteratives. Whe a o-optimal alterative i a alterative set is replaced by a worse oe, the pairwise outrakig relatios related to it may be chaged accordigly ad the overall rakig of the whole alterative set, which depeds o those pairwise outrakig relatios, may also be chaged. The first chage is reasoable whe cosiderig the fact that a o-optimal alterative has bee replaced by a worse oe. However, the secod chage is ureasoable ad may cause udesirable rak reversals as i the examples preseted i sectio 3. As it is demostrated i the ext sectio whe oe decomposes a decisio problem ito smaller problems ad aalyzes them by usig the ELECTRE II or III method, the rakigs of the smaller problems may ot follow the trasitivity property. This fact alog with the above rak reversal examples reveals that there is ot a priori rakig of the alteratives whe they are raked by the ELECTRE II or III methods because the rakig of a idividual alterative derived by these methods depeds o the performace of all the other alteratives curretly uder cosideratio. This causes the rakig of the alteratives to deped o each other. Thus, it is likely that the optimal alterative may be differet ad the rakig of the alteratives may be distorted to some extet if oe of the o-optimal alteratives i the alterative set is replaced by a worse oe. This ca be further explaied by meas of a simple example. Give three alteratives:,, ad A 3, suppose that origially strogly outraks A 3, weakly outraks A 3 ad ad are idifferet with each other. The rakig of these three alteratives will be > > A 3 whe usig the ELECTRE II method. Next, if the o-optimal alterative A 3 is replaced by a worse oe, the may strogly outrak A 3 while is still strogly outrakig A 3 ad is still idifferet with. Nothig is wrog so far. But ow the rakig of the three alteratives will be = > A 3 by usig the same method sice both ad ow strogly outrak A 3 ad they are idifferet with each other. It ca be see that ad are raked equally ow because A 3 becomes less desirable. This is exactly what happeed i the first example: ad are raked equally after A 3 has bee replaced by a less desirable alterative. This kid of irregular situatio is udesirable for a practical decisio-makig problem though it is reasoable i terms of the logic of the ELECTRE II method. It could leave the rakig of a set of alteratives to be maipulated to some extet. The rakig irregularity i the above example is very likely to occur whe usig the ELECTRE II or III method to rak a set of alteratives. If the umber of alteratives of a decisio problem is more tha 3, there will be more tha C 3 2 (=6) pairwise outrakig relatios betwee them. The the situatio may become worse by totally chagig the idicatio of the best raked alterative. It was oce poited out i [Belto ad Stewart, 2001] that the results of the distillatios are depedet o the whole alterative set, so that the additio or removal of a alterative ca alter some of the prefereces betwee the remaiig alteratives. A similar situatio occurs with the PROMETHEE method which is aother variat of the outrakig method. I [Keyser ad Peeters, 1996], it was poited out that the complete pre-orders from the PROMETHEE method are based o a all-to-all compariso betwee the alteratives; addig or deletig a alterative ca put the previous pre-orders upside dow. From the study reported i this paper, ow it ca be see that a similar situatio also occurs with the ELECTRE II ad III methods. That is, eve without additio or removal of alteratives, the best raked alterative might be altered ad the previous pre-order betwee the remaiig alteratives might be chaged to some degree by just replacig a o-optimal alterative by a worse oe. It must be poited out here that there is aother factor that may cotribute to rak reversals. Durig the costructio of the pairwise outrakig relatios, both ELECTRE II ad III eed to use a value or a threshold 11

12 which is also depedet o the performace values of all the curretly cosidered alteratives. For ELECTRE II, it is the parameter δ (i.e., the maximum differece of ay criterio) i the discordace idex formula. For ELECTRE III, it is the parameter λ used to decide the λ preferece relatios betwee the alteratives durig the distillatios. These δ ad λ values may be altered whe a o-optimal alterative is replaced by a worse oe. The the previous outrakig relatios betwee the other uchaged alteratives may be distorted to some degree, which fially may alter the idicatio of the best raked alterative or the overall rakig of the alteratives. Accordig to some experimetal aalysis, the above two factors may fuctio together or separately to cause rak reversals. From the above aalysis, it ca be see that the rakig processes of the ELECTRE II ad III are ot reliable ad robust eough to offer a firm aswer to a decisio problem. Usually, decisio makers udertake some kid of sesitivity aalysis to appreciate the sesitivity of the fial rakigs ad the robustess of the rakig procedures to chages i the criteria weights ad thresholds whe they use ELECTRE methods to solve decisio problems. However, the above rakig irregularities ca war decisio aalysts that they should be cautious i acceptig the rakig recommedatios of the ELECTRE methods eve after a careful sesitivity aalysis is udertake. 5. A Empirical Study This sectio describes a empirical study that focused o how ofte these rakig irregularities may happe uder the ELECTRE II ad III methods. Some computer programs were writte i MATLAB i order to geerate simulated decisio problems ad test the performace of ELECTRE II ad III uder the three test criteria described i sectio 2. I these test problems, the umber of the alteratives was equal to the followig te differet values: 3, 5, 7, 9, 11, 13, 15, 17, 19, ad 21. However, there is ot a commo rage of the criteria for all the tests. Compared with the tests of ELECTRE II, a wider rage of criteria for the tests of ELECTRE III was eeded i these experimets i order to clearly show how the rakig irregularity rates uder ELECTRE III will fluctuate with the icrease o the umber of the criteria. For the three tests of ELECTRE II, the umber of criteria was equal to 3, 5, 7, 9, 11, 13, 15, 17, 19, 21, 23, 25, 27, 29, ad 31. Thus, a total of 150 differet cases were examied with 10,000 radomly geerated decisio problems (i order to derive statistically sigificat results) per case. For the three tests of ELECTRE III, the umber of criteria was equal to the odd umbers betwee 3 ad 61. Thus, a total of 300 differet cases were examied with 10,000 radom decisio problems per case. Each radom decisio problem was aalyzed first by usig the ELECTRE II or III method ad the was aalyzed agai by usig the same method after oe of the o-optimal alteratives was replaced by a worse oe or the whole decisio problem was decomposed ito smaller problems as described i the last two test criteria. Ay occurred rakig irregularity was recorded. Figures 4 to 9 summarize these test results. I these figures, differet curves correspod to cases with differet umbers of alteratives; the X axis stads for the umber of criteria ad the Y axis is the rate of rakig irregularities that occurred i the 10,000 simulated decisio problems. Figures 4 ad 5 describe how ofte rak reversal happeed to ELECTRE II ad III methods uder test criterio #1 i this empirical test. That is, how ofte the idicatio of the best alterative is chaged whe a o-optimal alterative is replaced by aother worse alterative (give that the relative importace of each decisio criterio remais uchaged). The basis of ay ELECTRE method is to decide the pairwise outrakig relatios betwee alteratives. Give alteratives, whe a o-optimal alterative was replaced by a worse oe, the umber of pairwise outrakig relatios that might be chaged is at most (-1). This idicates that the higher the umber of the alteratives is the more possible becomes that the variatio of a sigle alterative may have a oticeable ifluece o the outrakig relatios. It is the chage of the outrakig relatios that results i the rak reversals. This is why the rak reversal rates usually icrease with the icrease o the umber of alteratives. It should be clarified here that eve if a case passed test criterio #1, this does ot mea that this case is immue to the rak reversal situatio described i test criterio #1. Whe applyig test criterio #1, oe 12

13 o-optimal alterative eeds to be pick up ad replaced by a worse oe. Which o-optimal alterative will be selected ad how worse it could be to trigger the rak reversal to happe were all radomly chose by the program. Whe replacig a o-optimal alterative by a worse oe, the program oly makes the selected o-optimal alterative to be worse tha before to a certai degree to test if it is eough to trigger the rak reversal to occur. If o rak reversal happes, the case will be released ad marked as havig passed test criterio #1. It is ot possible to test all the possibilities i terms of give sigle case. Therefore, eve if a case passed test criterio #1 i a sigle experimet, that does ot mea it is immue to the type oe rak reversal. Figures 6 ad 7 depict how the rakig irregularity rates of ELECTRE II ad III varied with the icrease o the umber of alteratives ad the umber of criteria i terms of test criterio #2. Oe ca see from these figures that the rates geerally icrease with the icrease o the umber of alteratives. This happes because the higher the umber of alteratives is, the higher is the umber of smaller problems that a decisio problem was decomposed ito, ad the the more likely it is for a cotradictio betwee the smaller problems to happe. I order to apply test criterio #3, 10,000 radom decisio problems whose rakigs follow the trasitivity property by usig the ELECTRE II or III method must be geerated ad the be examied uder test criterio #3. However, as oe ca see from Figures 6 ad 7, whe the umber of alterative is up to 7 or 9, the rakigs of the radom decisio problems almost ever follow the trasitivity property whe the umber of criteria is i some rage. It is difficult to fid 10,000 radom decisio problems per case that ca be used i the third test. Thus, oly the cases where the umber of alteratives was equal to 3, 5 or 7 were tested. Figures 8 ad 9 show how ofte the rakig irregularity will happe to these cases uder test criterio #3. The reaso why this rate icreases with the umber of alteratives is the same as that of the experimets uder test criterio #2. What is the relatioship betwee these rakig irregularity rates with the umber of the decisio criteria? From these figures oe ca see that, i geeral, the rakig irregularity rates will first icrease with the icrease o the umber of the criteria but the decrease whe the umber of criteria icreased beyod a certai value for each case. Please recall that the pairwise outrakig relatios betwee each pair of alteratives are decided by the cocordace ad discordace idices which are computed by their performace uder each criterio. For a fixed umber of alteratives, with the icrease o the umber of criteria, if the umber of criteria is beyod some value, the pairwise outrakig relatios ad the subsequet rakig of the alteratives is more likely to become more stable tha before. Just like it will be more believable if oe decides that Car A is better tha Car B i terms of 7 decisio criteria tha i terms of 3 decisio criteria. 13

14 1 # o r te i r C i e st T e r d u I R E C T E E L o f s R a te a l rs R eve R ak No. of Alts.=15 No. of Alts.=11 No. of Alts.=9 No. of Alts.=7 No. of Alts.=5 No. of Alts.=3 No. of Alts.=21 No. of Alts.=19 No. of Alts.=17 No. of Alts.= Number of Criteria Figure 4. Rak Reversal Rates of ELECTRE II uder Test Criterio #1. 1 # o r i s t C rite T e e r d u I R E C T E L E o f a t e s R a l rs R eve R ak No. of Alts.=15 No. of Alts.=13 No. of Alts.=21 No. of Alts.=19 No. of Alts.=11 No. of Alts.=17 No. of Alts.=3 No. of Alts.=7 No. of Alts.=5 No. of Alts.= Number of Criteria Figure 5. Rak Reversal Rates of ELECTRE III uder Test Criterio #1. 14

15 2 # o ri te r i s ṫ C T e e r d u I R E C T E L E o f s R ate y a rit gu l e g Ir i R ak No. of Alts.=21 No. of Alts.=19 No. of Alts.=17 No. of Alts.=15 No. of Alts.=13 No. of Alts.=11 No. of Alts.=9 No. of Alts.=7 No. of Alts.=5 No. of Alts.= Number of Criteria Figure 6. Rakig Irregularity Rates of ELECTRE II uder Test Criterio #2. 2 # o r i s t C rite T e e r d u I R E C T E L E o f s R ate y a rit gu l e g Ir i R ak No. of Alts.=3 No. of Alts.=21 No. of Alts.=19 No. of Alts.=17 No. of Alts.=15 No. of Alts.=13 No. of Alts.=11 No. of Alts.=9 No. of Alts.=7 No. of Alts.= Number of Criteria Figure 7. Rakig Irregularity Rates of ELECTRE III uder Test Criterio #2. 15

16 3 # o r i s t C rite T e e r d u I R E C T E L E o f s R ate y a rit gu l e g Ir i R ak No.of Alts.=3 No.of Alts.=5 No.of Alts.= Number of Criteria Figure 8. Rakig Irregularity Rates of ELECTRE II uder Test Criterio #3. 3 # o r te i r C i e st T e r d u I R E C T E L E o f s R ate y a rit gu l g Ire i R ak No. of Alts.=7 No. of Alts.=5 No. of Alts.= Number of Criteria Figure 9. Rakig Irregularity Rate of ELECTRE III uder Test Criterio #3. 16

17 A differet type of experimets was ru as well. The goal ow was to examie if there are ay explicit coectios betwee the results uder the three test criteria, especially betwee test criterio #1 ad test criterio #2. The experimetal tests were executed as follows. First, a large umber (i.e., 10,000) of radomly geerated decisio problems were examied by usig the ELECTRE II or III method i terms of test criterio #1 ad the radom test problems were divided ito two groups. Oe group had the problems that passed test criterio #1 ad the other group had those that did ot pass test criterio #1. Next, the problems withi each of these two groups were examied i terms of the test criterio #2 ad the rates of how ofte they passed or failed to pass this test criterio were recorded ad plotted for each oe of the two groups. Next, a test process similar to the above oe was performed. Agai, a large umber of radomly geerated decisio problems were examied by usig the ELECTRE II or III method but ow the process started by first testig for behavior uder the test criterio #2. The problems were divided ito two groups idicatig passig or ot passig this test criterio. The the problems withi each oe of these two groups were examied i terms of the test criterio #1 ad the rates of how ofte they will pass or fail to pass this test criterio are recorded ad plotted as before for each oe of these two groups. Similar tests as the above oes were also performed betwee test criterio #1 ad test criterio #3. From the above experimetal test results, o clear tedecy was foud to idicate that failure i oe test criterio would have a tedecy to lead to failure i terms of aother test criterio. That is, ot ay explicit coectio betwee the results uder the three differet test criteria was foud from these types of experimets. 6. Case Studies The previous computatioal results revealed that the rakig irregularities studied i this paper may occur frequetly i simulated decisio problems. This raised the questio whether the same could be true with real-life decisio problems. I order to ehace the uderstadig of this situatio, te real-life cases were studied. These cases were selected radomly from the published literature. That is, o special screeig was performed. The oly requiremet was to be able to extract the umerical data eeded to form a decisio matrix ad the weights of the criteria. It is better if threshold values could be give i the published case to avoid the icoveiece with the usig of the ewly defied thresholds. I these experimets, the required thresholds for case 1 to case 8 have bee specified i the refereced publicatios. For the last two cases, the thresholds were specified appropriately accordig to the score rage of each criterio. After gettig the data, every case was tested by usig the ELECTRE II or III method as i the refereced publicatio. The the three types of rakig irregularities were recorded wheever they occurred. Please refer to Table I for the summary of the experimetal results. Actually, the two examples preseted i sectio 3 are amog these 10 tested cases. Uder test criterio #1, that is, whe replacig oe of the o-optimal alteratives by a worse oe, there are maily two types of rak reversal situatios: 1. The optimal alteratives of the chaged decisio problem are partially differet from that of the origial problem. The umber of the optimal alteratives of the chaged problem is more or less tha that of the origial problem. For example, i terms of case 7, origially the optimal alterative is A 8. Next the optimal alteratives may become A 7 ad A 8 uder test criterio #1; for case 2, origially the optimal alteratives are A 6 ad A 3, ad the it may be just A 6 after oe of the o-optimal alteratives was replaced by a worse oe. 2. The optimal alterative of the ew problem is totally differet from that of the origial problem. For example, for case 8, origially the optimal alterative is A 9, ad the it becomes A 7 whe oe of the o-optimal alteratives was replaced by a worse oe; for case 6, origially the optimal alterative is, it may become 0 ad 8 uder the first test criterio #1. I terms of the same case, the above two situatios might both happe or just oe of them happeed i the tests. The emphasis is that the idicatio of the best alterative had bee chaged for those cases if ay of the two situatios occurred to them. The oe ca coclude that rak reversals occurred to those cases ad they 17

18 failed to pass test criterio #1. From Table I, it ca be see that 6 out 10 cases failed to pass test criterio #1. Also, 9 out 10 cases failed to pass test criterio #2. For the oly case which could be tested uder test criterio #3, it failed to pass it too. 7. Coclusios Although MCDM plays a critical role i may real-life problems, it is hard to accept a MCDM method as beig accurate all the time. The preset research results complemet previous oes ad reveal that eve more MCDM methods suffer of rakig irregularities. The ELECTRE methods are widely used today i practice. However, the rakig irregularities should fuctio as a warig i acceptig ELECTRE s recommedatios without questioig their validity. Previous ad curret research idicates that such rakig irregularities ted to occur whe the alteratives appear to be very close to each other. If, o the other had, the alteratives are very distictive from each other, the it is less likely that these rakig irregularities will take place. However, oe eeds more powerful MCDM method whe alteratives are closely related with each other. I previous studies [Triataphyllou, 2000 ad 2001], it was foud that the Multiplicative AHP does ot suffer of the previous three types of rakig irregularities. The way i which alteratives are raked by the Multiplicative AHP utilizes less iformatio tha the ELECTRE methods. I Table II, we compared the alteratives rakigs of some of the previous te real-life cases by usig the Multiplicative AHP ad the ELECTRE II or III method, respectively. From Table II, oe ca see that the majority of the situatios will result i the same optimal alterative but the rakigs of the o-optimal alteratives uder both methods have a sigificat differece. Sice the rakig irregularities uder the three test criteria will ot happe uder the Multiplicative AHP, it is temptig to try to see if a ew method ca be desiged, which combies qualities from both the Multiplicative AHP ad the ELECTRE methods or some other MCDM methods, ad still ot to suffer of these rakig irregularities. Aother directio for future research is to defie more test criteria agaist which existig ad future MCDM methods ca be evaluated. Some iterestig work i this area has bee coducted. For example, i [Kujawski, 2005], the author proposed three properties for a desirable MCDM approach. They are about idepedece of domiated alteratives, o imposed rak reversal ad egative side effects associated with iferior substitutios. Clearly, this is a fasciatig area of research ad it is of paramout sigificace to both researchers ad practitioers i the multi-criteria decisio-makig field. 18

19 Table I. Summary of Case Studies. Case Referece Domai of applicatio ad method used Size of decisio problem Did it fail Did it fail Did it fail umber No. of alteratives No. of criteria T. C. #1? T. C. #2? T. C. #3? 1 Hokkae, J., ad P. Salmie, Choosig a solid waste maagemet 22 8 Yes Yes [1997a] system (ELECTRE III) 2 Belto, V., ad T.J. Stewart, Busiess locatio problem 7 6 No Yes [2001] (ELECTRE III) 3 Rogers, M., ad M. Brue, Evirometal appraisal 9 9 No No Yes [1996] (ELECTRE II) 4 Rogers, M.G., M. Brue, ad Site selectio for a wastewater treatmet 5 7 Yes Yes L.-Y. Maystre, [1999] plat (ELECTRE II) 5 Raj, P.A., [1995] Water resources plaig (ELECTRE II) 27 6 Yes Yes 6 Buchaa, J., P. Sheppard, ad Project rakig (ELECTRE III) 5 5 No Yes D.V. Lamsade, [1999] 7 Hokkae, J., ad P. Salmie, Choosig a solid waste maagemet 11 8 Yes Yes [1997b] system (ELECTRE III) 8 Rogers, M.G., M. Brue, ad Choosig a waste icieratio strategy Yes Yes L.-Y. Maystre, [1999] (ELECTRE III) 9 Poh, K.L., ad B.W. Ag, Choosig a alterative fuel system for 4 6 No Yes [1999] lad trasportatio (ELECTRE II) 10 Leyva-López, J.C., ad E. Ferádez-Gozález, [2003] Selectio of a alterative electricity power plat (ELECTRE III) 6 6 Yes Yes * T. C. stads for Test Criterio. * If a case failed uder test criterio #2, it will ot be able to get a overall rakig of the alteratives from the smaller problems. That meas it will ot be able to apply test criterio #3 to that case. The symbol was used i the correspodig cell to stad for this situatio. 19

20 Table II. Rakig Compariso Rakig from the Multiplicative AHP Rakig from the ELECTRE method method No. Of Cases Case 1 A 9 >2 >A 6 >1 >5 >8 >2 >1 >A 8 > >4 >7 >0 >A 3 >A 7 >0 > >3 > 6 > >9 > Case 2 A 7 > > A 3 > A 6 > > > A 6 =A 9 =2 =5 =1 > =A 8 =1 =4 =8 =0 > = A 7 =0 =3 =6 =7 =9 >2 >A 3 > = (ELECTRE III) A 3 A 7 Case 3 > > A 6 > A 3 > > > A 7 > A 8 > A 9 A 6 (ELECTRE III), A 6 A 3, A 7, A 9 A 8 (ELECTRE II) Case 4 > A 3 > > > > > A 3 > > (ELECTRE II) Case 5 > > >8 >0 >A 7 >A 3 >1 >6 > 7 >5 >2 =2 > >4 >6 >7 >A 8 =3 >A 9 =0 >A 6 >1 >9 >5 >4 Becaue of the space costrait, the rakig of case 5 is show as a footote to this table. (ELECTRE II) Case 7 A 8 > > >A 6 >1 > >A 7 >A 3 > >0 >A 9 A 8 1 A 3 A 7 A 9 0, A 6 (ELECTRE III) Case 9 `>A 3 > > `> >A 3 = (ELECTRE II) Case 10 >A 3 > >A 6 > > A 3 A 6 (ELECTRE III) Note: Because some of the performace values i the decisio matrices of case 6 ad case 8 are zeros ad the Multiplicative AHP method forbids the use of zero performace values durig its rakig process, we did ot apply the Multiplicative AHP methods to these two cases. *The rakig derived for case 5 by usig the ELECTRE III method is as follows: 0 4 1, 7 A 3 2, 2, 3 0, A , A A 7 20