1 Analytic Hierarchy Process, a Psychometric Approach Christine Choirat and Raffaello Seri Dept. of Economics, Università dell Insubria Varese, Italy Abstract. The Analytic Hierarchy Process, or AHP for short, is a decision method that is often used in Management and Economics. In this paper, we study real data that we obtained from a simple experiment. Then, from the analysis of these data, we point out that some pathological behaviours of the AHP correspond to phenomena already encountered in Psychometric Choice Theory. Keywords: Analytic Hierarchy Process, Psychometrics 1 Introduction and Presentation of the Experiment The Analytic Hierarchy Process (AHP) is a method often used in Management and Economics for the ranking of a set of alternatives and the choice of a preferred one. The AHP has been applied to measure performances of productive systems (Rangone 1996), to the ABC classification of warehouse items (Partovi et al. 1993), to strategic planning (Armel and Orgler 1990), to evaluate priorities in customers needs (Armacost et al. 1994) and to investment analysis in innovative technologies [(Stout et al. 1991), (Weber 1993), (Wicks and Boucher 1993)]; other applications are in (Saaty 1994). Moreover, software products implementing this technique are available on the market, such as Expert Choice, HIPRE3+, Criterium, AUTOMAN [see (Stout et al. 1991), (Weber 1993)] and NCIC (Wicks and Boucher 1993). The AHP is connected with Psychometric Choice Theory, which is inspired from results of psychophysical experiments. In these experiments, a subject has to compare the intensities of physical stimuli of different magnitude, in order to rank them. The Fechnerian interpretation consists in associating sensations to stimuli: sensations caused by external stimuli allow the individual to discriminate between stimuli. In particular the Weber-Fechner law states that sensations are not linearly related to stimuli, but the relation is increasing and convex (like the logarithm). 1 The Thurstonian theory 2 in Psychometrics (Thurstone 1927) lends itself to this idea, under the additional hypothesis that sensation is the realization of a random variable: this implies that the individual is not able to perfectly discriminate between stimuli. Hence, the comparison of two stimuli amounts to the comparison of the realizations of two random variables representing the sensations induced by the two stimuli. These random variables are called discriminal processes and this principle is called law of comparative judgement. As (Yellot 1977) states: The law of comparative judgement postulates an imaginary psychological process, underlying observable behaviour, wherein choice objects are represented by random variables ( discriminal processes ) which the subject compares in order to arrive at a decision. The application of the AHP involves 3 steps [for a more comprehensive introduction see (Saaty 1986)]: 1. The problem is reduced to a hierarchy of levels (see Figure 1): the highest level corresponds to the overall objective (say, the convenience of a workplace). The lowest level is formed by a set of alternatives (say, a set of companies among which to choose). The intermediary levels are composed by attributes (called criteria) of the upper levels (as an example, for a workplace, the stability, the economic treatment and so on). 2. The elements of any level are subject to a series of paired comparisons (on the Saaty s scale, ranging from 1/9 to 9) and a pairwise comparison matrix is built. If ψ i is the relevance of criterion i, a ij is intended to be an estimate of ψ i /ψ j : the matrix A has therefore the characteristic that a ij = 1/a ji. The eigenvector associated with the greatest eigenvalue of A is considered an estimator of ψ = [ψ i ]. The extent with which the matrix A reflects these measures is called implicit judgements consistency index and is defined as: α = (λ max (A) n) / (n 1) where λ max (A) is the greatest eigenvalue of A and n is the number of criteria in the level. 1 The Fechner approach is widely present also in the first works on utility theory. 2 A good introduction is in (Nunnally 1978), pp
2 3. The estimated weights are then aggregated in an overall score that makes it possible to order the alternatives. From a descriptive point of view, the AHP yields a framework in which the usual decisional process of the individual can be interpreted through a decomposition in elementary units. The bases on which the AHP is grounded are essentially linked to the theory of preference, that is to Economics, and to the theory of judgement measurement, that is to Psychometrics. Objective 1 Criterion 1 Criterion 2 Criterion 3 Criterion 4 Criterion 3.1 Criterion 3.2 Alternative 1 Alternative 2 Figure 1. Structure of the AHP. From a normative point of view, the AHP tries to replicate, clarify and correct the mental process that carries the decision maker to prefer an alternative to the others: it gives, without any doubt, a useful instrument of decision support. It is not clear, on the other hand, whether the order of the alternatives that emerges from the AHP is the correct one and if the chosen alternative is really better than the others. To illustrate this, we have tried to analyze whether the AHP method based on pairwise comparisons of alternatives on Saaty scale is able to recover the importances of different alternatives. We do not tackle the related (but much more complex) problem of aggregating preferences in tree value diagrams. We have asked six people to locate on Saaty s scale (1 9) the ratio between the distance from Milan to some city A and the distance from Milan to some city B. The cities of the sample are Turin, Venice, Rome, Naples and Bari; the distances to Milan are respectively 140, 270, 575, 785 and 880 kilometres, and are sufficiently regular to avoid the presence of outliers. Moreover, their positions are well known to the interviewed people. In what follows, people answering the questions will be called Decision Makers (DM). The paper will be organized as follows. In Section 2.1 we show that the pairwise comparisons of an individual can be grossly biased: in particular, the relation between the real ratios and the elicited ones seems to be concave and increasing; this means that the elements of the matrix A chosen by the decision maker can be badly biased estimates of the real ones. In Section 2.2, we consider the performances of the eigenvector method when the elements of the matrix A are given by the real ratios distorted by deterministic functions: this shows the relevance of the scale chosen and the distorsions caused by the inability of the decision maker to locate ratios on the scale. Section 2.3 displays the weights (in our case, the distances) implicitly chosen by the decision makers, in order to show that some distorsions can indeed arise in practice. 2 Remarks In this Section, we analyse the data obtained from our experiment. In the same spirit as Neural Networks or Genetic Algorithms, the AHP is biomimicking: this method indeed breaks up the decision process into elementary units, just as the decision maker is supposed to be doing mentally. But, the most critical aspect of that method, which we are about to detail, appears in the ability of the subjects to estimate pairwise preferences expressed on a ratio scale. The analysis of our empirical results shows that the very limits of the AHP can be interpreted through a psychometric viewpoint. 2.1 Linearity of the Individual Judgements The first problem concerns the capacity of the individual to evaluate stimuli of varying intensities: in particular, it is to be hoped that the individual doubles its response when confronted to a stimulus doubling. The responses of Decision Maker 1 are reported in Table 1: if we consider only the weights greater than 1 (since the others are characterized by the reciprocity property), we get a relation of logarithmic type, as shown in Figure 2.
3 Table 1. Matrix A for Decision Maker 1. Turin Venice Rome Naples Bari Turin Venice Rome Naples Bari The logarithmic regression of the stimuli on the responses, given by: ( ) di a ij = log has an excellent fit and a determination coefficient (the so-called R 2 ) equal to 94.22%. A linear regression on the same data gives a determination coefficient equal to 83.19%, which is clearly worse. Nonlinear and nonnegative relations showing a saturation effect are rather frequent in psychology. When the stimuli increase in strength, it often becomes hard for the individual to evaluate properly the differences among the stimuli: therefore, and above all in presence of alternatives characterized by very different levels of magnitude, the individual may not be able to evaluate correctly the relative levels. A relation of that type appeared for the first time in the solution given by Daniel Bernoulli to the socalled Paradox of Saint-Petersburg: he supposed that the moral fortune (ancestor of the modern utility) was a logarithmic function of the physical fortune (the gain obtained from the game in his case); for discussion on the Paradox, see (Boyer 1968). Moreover, for human sensations the so-called Weber-Fechner law holds: on the basis of empirical observation, it states that there is a logarithm-like relationship between the stimulus on the sense organs and the sensation. d j Real ratios Figure 2. Relation between the real ratios (d i /d j on the X axis) and the estimted ones (a ij on the Y axis).
4 2.2 Eigenvector under Alternative Distortion Scales The problem pointed out in the former paragraph is a sort of bias or systematic error that the Decision Maker makes when he evaluates the ratio d i /d j through his personal estimate a ij : the underlying idea is that the individual confronted to stimuli responds according to a scale that can be a linear or nonlinear function of the real stimuli. We wonder whether this phenomenon leads to important distortions in weight estimation. To evaluate the robustness of that model with respect to scale transformations, we adapt the former example to see whether a scale change (i.e. using a logarithmic or linear scale, 1 5, 1 9, etc.) yields notably different results. To separate the analysis of systematic bias from the study of judgement subjectivity, we use real data d i /d j, proceeding as follows: 1. we consider the values of the ratio d i /d j greater than 1 and we apply the following increasing transformations: Series 1 a ij = exp (d i /d j 1) Series 2 a ij = log (d i /d j ) + 1 Series 3 a ij = (d i /d j ) 2 Series 4 a ij = d i /d j Moreover we apply some linear transformations (identified with Series 5 9) so as to rescale the values on a scale going from 1 to M where M = 1.9, 5, 9, 15, 35; 2. we calculate the matrix A through the reciprocity property; 3. for any matrix we obtain the eigenvector associated with the maximal eigenvalue. The results are shown in Figure 3: from the graph it is evident that the choice of a scale has a strong impact on the calculation of the final weights ψ i s. As an example, the exponential scale (Series 1) and the logarithmic one (Series 2) have very different behaviours, since the former gives less importance to small observations, while the latter gives more importance to small observations. On the other hand, linear scales behave better, but in the present case the best one appears to be the 1 5 scale, since it reflects the range of the data (that is from 1 to ) S1 S2 S3 S4 S5 S6 S7 S8 S Real distances Figure 3. Relation between the real distances (d i on the X axis) and the estimated ones (ψ i on the Y axis).
5 Therefore, the choice of the upper limit of the scale can have a nonnegligible impact on the weights obtained from the judgements of the decision maker: (Weber 1993) remarks that two alternatives (say 1, 2) one of which (1) is weakly preferred to the other (2), would yield on Saaty s scale a ratio and final weights equal to (ψ 1, ψ 2 ) = (0.75, 0.25): this does not coincide with the usual notion of weak preference. The choice of the upper limit of the scale is then crucial since it provides the decision maker with a benchmark with respect to which the alternatives are evaluated. The weights obtained from the eigenvector method, the correlation with the real values and the consistency index are displayed in Table 2. Table 2. Eigenvectors for the 9 series, correlation with the true values, greatest eigenvalue and Saaty s α. Data Series 1 Series 2 Series 3 Series 4 Eigenvalue Correlation Eigenvalue Alpha Series 5 Series 6 Series 7 Series 8 Series 9 Eigenvalue Correlation Eigenvalue Alpha From this analysis, it is clear that, if the decision maker is able to express linearly the relative relevances of the alternatives, the scale has only a limited impact that sometimes can be relevant. On the other hand, if the perceptions of the decision maker are nonlinearly related to the stimuli, then the calculated weights can yield a very misleading picture of the real ones. In particular exponential or logarithmic scales lead to serious distortions. 2.3 Impact of Individual Variability In this paragraph, we present the answers of the 6 Decision Makers, to show the impact of individual subjectivity on weight calculation and to argue that the systematic distortion of the weights is a typical feature. In order to investigate the sensitivity of the AHP as concerns the choice of the decision maker, we reproduce in Table 3 the weights calculated for the Decision Makers, together with the values of the consistency index and the true values. It is not difficult to remark that the weights calculated in this way are quite far from the expected linear relation: the weight of the most important criterion (or alternative) would be overestimated and the contrary would hold for the weight of the less important. Figures 4 and 5 show that the behavior of elicited weights is almost exponential. Acknowledgments We thank A. Rangone for providing some references and F. Piccioni for many interesting and stimulating ideas. We are indebted to the six guinea-pigs that kindly participated in the experiment.
6 Table 3. Relevance of the Decision Maker. Turin Venice Rome Naples Bari Alpha Corr. Data D.M D.M D.M D.M D.M D.M DM1 DM2 DM3 DM4 DM5 DM Real distances Figure 4. Estimated distances (ψ i, Y axis) and real ones (d i, X axis) for the different decision makers. References Armacost R.L., Componation P.J., Mullens M.A., Swart W.W., 1994, An AHP framework for prioritizing customer requirements in QFD: an industrialized housing application, IIE Transactions, Vol. 26, No. 4, July, pp Armel A., Orgler Y.E., 1990, An application of the AHP to bank strategic planning: the mergers and acquisitions process, European Journal of Operational Research, Vol. 48, pp Boyer C.B., 1968, A History of Mathematics, John Wiley & Sons, New York. Nunnally J.C., 1978, Psychometric Theory, McGraw-Hill, New York. Partovi F.Y., Burton J., 1993, Using the Analytic Hierarchy Process for ABC analysis, IJOPM, Vol. 13, No. 9. Rangone A., 1996, An Analytical Hierarchy Process framework for comparing the overall performance of
7 DM1 DM2 DM3 DM4 DM5 DM Real distances Figure 5. Logarithm of the estimated distances (log (ψ i ), Y axis) and real ones (d i, X axis) for the different decision makers. manufacturing departments, IJOPM, Vol. 16, No. 8. Saaty T.L., 1986, Axiomatic foundation of the Analytic Hierarchy Process, Management Science, Vol. 32, No. 7, pp Saaty T.L., 1994, Highlights and critical points in the theory and application of the Analytic Hierarchy Process, European Journal of Operational Research, Vol. 74, pp Stout D.E., Liberatore M.J., Monahan T.F., 1991, Decision support software for capital budgeting, Management Accounting, July, pp Thurstone L.L., 1927, A law of comparative judgement, Psychological Review, Vol. 34, pp Weber S.F., 1993, A modified Analytic Hierarchy Process for automated manufacturing decisions, Interfaces, Vol. 23, No. 4, pp Wicks E.M., Boucher T.O., 1993, NCIC: a software tool for capital investment analysis in manufacturing, Computer Industrial Engineering, Vol. 24, No. 2, pp Yellott, J.I., 1977, The relationship between Luce s Choice Axiom, Thurstone s theory of comparative judgement, and the double exponential distribution, Journal of Mathematical Psychology, Vol. 5, pp