A MCDM Tool to Evaluate Government Websites in a Fuzzy Environment

A MCDM Tool to Evaluate Government Websites in a Fuzzy Environment Gülçin Büyüközkan Abstract This paper presents a multi criteria decision making (MCDM) framework for evaluating the performance of government websites. In line with the multidimensional characteristics of website quality, MCDM provides an effective framework for an inter-website comparison involving the evaluation of multiple attributes. It thus ranks different websites compared in terms of their overall performance. In addition, the subjectivity and vagueness in the assessment process is dealt with fuzzy logic. The proposed framework is effectively illustrated to evaluate Turkish government websites. Keywords Multi criteria decision making Evaluation of website performance VIKOR method Fuzzy logic Government websites 1 Introduction During the last decade a revolution in information and communication technologies is being witnessed. This revolution is not only changing the daily life of people, but also changing characteristics of the interaction between the governments and their citizens. These changes, in turn, are rapidly being transformed into new forms of government, namely, electronic (e-)government. E-government is concerned with providing or attainment of information, services or products through electronic means, by and from governmental agencies, at the desired time and place, and thus offering an extra value for all the participant parties (citizens, business partners, employees, other agencies and government entities) (Zweers and Planque 2001). The potential of e-government as a new public medium depends on the quality of web pages and the offered e-services. For this reason, the importance of measuring G. Büyüközkan Department of Industrial Engineering, Galatasaray University, Çıra gan Caddesi No: 36 Ortaköy, İstanbul, Turkey, e-mail: gbuyukozkan@gsu.edu.tr M.Ehrgottetal.,Multiple Criteria Decision Making for Sustainable Energy and Transportation Systems, Lecture Notes in Economics and Mathematical Systems 634, DOI 10.1007/978-3-642-04045-0 17, c Springer Physica-Verlag Berlin Heidelberg 2010 201

202 G. Büyüközkan the performance of e-government initiatives cannot be overemphasized (Criado and Ramilo 2003) and assessing factors associated with website success are needed. Although many studies have already proposed evaluation criteria for information sources and web-sites, there are limited studies that have provided guidelines to measure the web-site quality, especially with analytic approaches. Similarly, while a significant body of academic literature exists on e-government services, little is known about government website quality (Eschenfelder et al. 1997; Kaylor et al. 2001; Smith 2001; Barnes and Vidgen 2003). In addition, the focus of the academic literature on government website up to date has rather focused on the conceptual framework of government websites success factors. As a result of the abovementioned motivations, a multi criteria decision making (MCDM) evaluation model that practitioners and researchers can use for assessing the quality of government websites is developed and validated in this paper. MCDM is a powerful tool widely used for evaluating and ranking problems containing multiple, usually conflicting criteria (Pomerol and Barba Romero 2000). In line with the multi-dimensional characteristics of website quality, MCDM provides an effective framework for an inter-websites comparison involving the evaluation of multiple attributes. Recently, a compromise ranking method, namely the VIKOR method, has been introduced as an applicable technique to implement within MCDM (Opricovic and Tzeng 2004; Tzeng et al. 2005a). It introduces a multi-criteria ranking index based on the particular measure of closeness to the ideal solution (Opricovic and Tzeng 2004). In addition, the subjectivity and vagueness in the assessment process is dealt with fuzzy logic (Zadeh 1975). In this study, the VIKOR method is then extended in fuzzy environment to take account the ambiguities encountered in the government website performance assessment. The paper is organized as follows. Section 2 describes the details of the proposed evaluation framework and the methods used. To validate our model and to examine its effectiveness, we evaluate the quality of Turkish government websites in Sect. 3. Finally, some concluding remarks are given in the last section. 2 An Evaluation Framework for Government Websites Websites provide the key interface for users of the Internet. In this context, the quality of a government website has to be analyzed in a detailed manner. Several papers have been published on website quality (such as Aladwani and Palvia 2002; Cox and Dale 2002; Li et al. 2002; Tan et al. 2003). Many of them offer frameworks containing groups of quality dimensions that are similar to the SERVQUAL (Service Quality) model proposed by Parasuraman et al. (1988). In addition, some publications are concerned with methodologies for specific website types such as e-government (Eschenfelder et al. 1997; Kaylor et al. 2001) or healthcare (Bedell et al. 2004). In a number of publications, quantitative methods are used for the website quality evaluation. Statistical methods are the most widely used assessment tools

A MCDM Tool to Evaluate Government Websites 203 (Cox and Dale 2002; Toms and Taves 2004). Additionally, other methods such as multidimensional scaling and correspondence analysis (Van der Merwe and Bekker 2003) or multi criteria analysis (Grigoroudis et al. 2008) are also used in assessing and improving the website quality. In this study, fuzzy extension of VIKOR method within a group decision-making framework is suggested to evaluate government websites. With its ability to provide compromise solution(s), VIKOR is selected in this work as a suitable method for evaluating website performances. However, VIKOR is based on exact and numerical assessments. Owing to the unavailability and/or uncertainty of information, it can be very difficult to obtain exact assessment data for evaluation criteria. Moreover, evaluators tend to give assessments based on their knowledge, past experience and subjective judgments. Fuzzy set theory (Zadeh 1975) plays a significant role to deal with the vagueness of human thought. Furthermore, such concepts have been recently applied to website evaluation problems (Tzeng et al. 2005b; Bilsel et al. 2006). Group decision-making is another important concern in this study. Multiple evaluators are often preferred rather than a single evaluator to avoid bias and to minimize the partiality in the decision process. The fuzzy Delphi method (Kaufmann and Gupta 1988) that is a systematic procedure for evoking expert group opinion is used in this study. Finally, the evaluation procedure consists of three main steps as follows: Step 1. Identify the government websites evaluation criteria that are considered the most important. Step 2. Evaluate the identified government websites through fuzzy Delphi method. Step 3. Determine the compromise solutions set by applying the fuzzy VIKOR methodology. The content of each step are detailed in each of the following subsections. 2.1 Website Evaluation Criteria In order to identify the important criteria influencing the user assessment on government websites, we conducted an in-depth literature survey (such as Eschenfelder et al. 1997; Parasuraman et al. 1988; Kaylor et al. 2001; Smith 2001; Cox and Dale 2002; Li et al. 2002; Barnes and Vidgen 2003; Tan et al. 2003). Taking the structure proposed by Li et al. (2002) as a base and according to the critics and suggestions of information technology and e-government project experts, we established evaluation criteria including six core dimensions. The dimension named tangibles (C 1 ) comprises criteria describing visual and physical aspects of a website. Criteria in the dimension of reliability (C 2 ) are introduced to measure both the trustworthiness of a website and to evaluate the security of a website. Criteria describing the ability of web-based service systems to perform the online service consistently and accurately are stated in the dimension of responsiveness (C 3 ). Level of individualized

204 G. Büyüközkan attention that the firm provides to its customers and the availability of links to other web pages in a government website are stated in the dimension of empathy (C 4 ). The dimension of quality of information (C 5 ) defines the attributes of information contained in the website and describes the content of information quality dimension. The dimension of integration of communication (C 6 ) measures the availability of complementary functions of the traditional communication media to digital media. 2.2 Fuzzy Delphi Method The Delphi technique is a study method of generating ideas and facilitating consensus among individuals who have special knowledge to share. The Delphi method uses a panel of carefully selected experts who answer a series of questionnaires. Although the traditional Delphi methods have provided much merit, as with many other survey techniques, the problems of ambiguity and uncertainty still exist. Therefore, the fuzzy Delphi method was originally introduced by Kaufmann and Gupta (1988). In this study, we use the weighted fuzzy Delphi method as detailed in (Bojadziev and Bojadziev 1997). As in business, management and science, the knowledge, experience and expertise of some experts are often preferred among others in a group of experts. This can be expressed by assigning unequal weights to the experts. That leads to the weighted fuzzy Delphi method, which is summarized as follows (Bojadziev and Bojadziev 1997): Delphi-Step 1: To determine the aggregated fuzzy weight Qw i of criterion c i i D 1;:::;n, and the aggregated fuzzy rating f Q ij of alternative a j j D 1;:::;munder criterion c i, K experts are asked to provide the evaluation data in the form of linguistic variables, which in turn corresponds to the fuzzy triangular numbers fq ij k and Qw k i. In this method, each expert has a weight k according to his/her degree of experience. Delphi-Step 2: First, the weighted average computed as follows: Q f ij of all fq ij k s and Qw i of all Qw k i s are fq ij D. 1 f Q ij 1/ :::. k f Q ij k/ ; Qw i D. 1 Qw 1 i / :::. k Qw k i / 1 C :::C k 1 C :::C k Then for each expert, the deviation between the weighted average f Q ij and f Q ij k is computed. The same procedure is also applied for the deviation between the weighted average Qw i and Qw k i. At this point, the distance of fuzzy numbers is calculated using our proposed method in order to evaluate the deviations between the weighted averages and experts evaluation data. Delphi-Step 3: A threshold value is defined so that the deviation is sent back to the expert if the distance between the weighted average and expert s evaluation data is

A MCDM Tool to Evaluate Government Websites 205 greater than this value. If there is any distance value being greater than the threshold value then the relevant expert is notified and the process starting with the step 2 is repeated until there is no distance value exceeding the threshold value. This process is repeated until two successive averages are reasonably close to each other. It is assumed that the distance being less than or equal to 0.2 corresponds to two reasonably close fuzzy estimates (Cheng and Lin 2002). We suggest this threshold. 2.3 Fuzzy VIKOR Method As a method belonging to the compromise programming category, the VIKOR method was introduced as an applicable technique to implement within MCDM (Opricovic and Tzeng 2004; Tzeng et al. 2005a). It focuses on ranking and selecting from a set of alternatives in the presence of conflicting criteria. The VIKOR method determines the compromise ranking-list and the compromise solution by introducing the multi criteria ranking index based on the particular measure of closeness to the ideal solution (Opricovic and Tzeng 2004). The compromise solution is a feasible solution, which is the closest to the ideal, and here compromise means an agreement established by mutual concessions. In this study, a fuzzy extension of VIKOR method within a group decision-making framework is suggested to determine the compromise solution. Denote m alternatives under consideration as a 1 ;a 2 ;:::;a m, n evaluation criteria as c 1 ;c 2 ;:::;c n. Then, the steps of the suggested approach are as follows. Step 1. Construct a committee of experts with K members and determine the alternatives and evaluation criteria. Step 2. Identify the evaluation base, in other words the linguistic variables for weighting criteria and the linguistic ratings for the alternatives. Step 3. Apply fuzzy Delphi method summarized in Sect. 2.2. Step 4. If the supports of triangular fuzzy numbers expressing linguistic variables in Step 2 do not belong to the interval [0,1], then a scaling is needed to transform them back in this interval. Here, we use a linear scale transformation to have a comparable number. As an example, if we transform the rating of alternatives, we have Qr ij D fij 1 max =fi ;fij 2 max =fi ;fij 3 max =fi where f Q ij D fij 1;f2 ij ;f3 ij, fi max D max j fij 3 i D 1;2;:::;n. Step 5. Compute the values of QS j and QR j j D 1;2;:::;m by the relations QS j D nid1 Qw i d Q1; Qr ij and QR j D max id1 Qw i d Q1; Qr ij,where QS j and QR j are used for formulating the ranking measure of group utility and the individual regret respectively. Here, d Q1; Qr ij represents the distance of an alternative rating to the positive ideal solution Q1 D.1;1;1/calculated by area compensationmethod. This method is due to Fortemps and Roubens (1996) and has reasonable ordering

206 G. Büyüközkan properties and computational easiness. Note that the maximum among Qw i d Q1; Qr ij values is the one that is the most distant from Q1. Step 6. Compute the values QQ j j D 1;2;:::;m with the equation QQ j D QS 0 j.1 / QR 0 j,where QS 0 j and QR 0 j are normalized QS j and QR j values using the linear scale transformation. Here, is introduced as a weight of the strategy of the majority of criteria as proposed in the original VIKOR method. The compromise can be selected with voting by majority ( >0:5), with consensus ( D 0:5), or with veto ( <0:5). Step 7. The ranking order of alternatives is determined with the help of the area compensation method. First QS j 0, QR j 0,and QQ j values are defuzzified into crisp Sj 0, Rj 0 and Q j values. Then, alternatives are ranked by sorting each Sj 0, R0 j and Q j values in increasing order as in the original VIKOR method. The result is a set of three ranking lists denoted as SŒ 0, R0 Œ and Q Œ. The alternative j 1 corresponding to Q Œ1 (the smallest among Q j values) is proposed as a compromise solution if CS1. The alternative j 1 has an acceptable advantage,inotherwordsq Œ2 Q Œ1 DQ where DQ D 1=.m 1/ and m is the number of the alternatives. CS2. The alternative j 1 is stable within the decision making process,inotherwords it is also the best ranked in SŒ 0 or R0 Œ. If one of the above conditions is not satisfied, then a set of compromise solutions is proposed, which consists of: The alternatives j 1 and j 2 where Q j2 D Q Œ2 if only the condition CS2 is not satisfied, or The alternatives j 1 ;j 2 ;:::;j k if the condition CS1 is not satisfied; and j k is determined by the relation Q Œk Q Œ1 <DQfor maximum k where Q jk D Q Œk (the positions of these alternatives are in closeness). 3 Case Study: Implementation of the Proposed Framework Turkish transition to e-government is an essential obligation for a country at the gate of the European Union. Turkey engaged in e-transition Turkey Project in 2004, within the Information Society Department (ISD) of State Planning Organization (Akman et al. 2005). ISD reports on the projects and applications in e-government and they observe all government websites included in the e-transition Turkey Project. According to an independent yearly report evaluating public websites in 198 countries worldwide, Turkey s public websites rank 9th in 2007, sharing the top ten with Asian and North American leaders as well as the United Kingdom (5th) and Portugal (7th). In comparison to last year s results, Turkey achieved a dramatic leap by gaining 18 ranks, thus leaving EU countries behind (West 2007). Therefore, as e-government is such an important issue, Turkish government websites must be evaluated and judged in order to improve the quality. 13 ministry websites given in Table 1 and included in the ISD s report are analyzed as a case study.

A MCDM Tool to Evaluate Government Websites 207 Step 1. A group of three domain experts, E 1,E 2 and E 3, has conducted the evaluation process. These experts are professionals who have been studying on various projects about developing e-government systems for a certain time. Yet, the first expert E 1 is more experienced with 40% importance, while the two others have equal 30% importance. Step 2. As mentioned previously, the experts expressed their preference for criteria weights and alternatives linguistically (see Tables 2 and 3). 13 determined websites given in Table 1 are to be evaluated according to six evaluation criteria given in Sect. 2.1. Such an example, the evaluation results of E 1 are given in Table 4. Table 1 Analyzed Turkish government websites Label Government agencies Web addresses W 1 Police Headquarters www.egm.gov.tr W 2 Retirement Credit Institution www.emekli.gov.tr W 3 Ministry of Justice www.adalet.gov.tr W 4 Ministry of Customs www.gumruk.gov.tr W 5 Ministry of National Education www.meb.gov.tr W 6 Ministry of Health and Social Services www.saglik.gov.tr W 7 Ministry of Culture www.kultur.gov.tr W 8 Central Bank www.tcmb.gov.tr W 9 Income General Director www.gelirler.gov.tr W 10 Ministry of Foreign Affairs www.mfa.gov.tr W 11 Ministry of Labor and Social Security www.calisma.gov.tr W 12 Grand National Assembly www.tbmm.gov.tr W 13 State Planning Organization www.dpt.gov.tr Table 2 Linguistic variables for evaluating criteria importance Variable Symbol Fuzzy Scale Very High VH.0:7; 1:0; 1:0/ High H.0:5; 0:7; 1:0/ Medium M.0:2; 0:5; 0:8/ Low L.0:0; 0:3; 0:5/ Very Low VL.0:0; 0:0; 0:3/ Table 3 Linguistic variables for evaluating government websites Variable Symbol Fuzzy scale Outstanding O.0:875; 1:000; 1:000/ Very Good VG.0:750; 0:875; 1:000/ Fairly Good FG.0:625; 0:750; 0:875/ Good G.0:500; 0:625; 0:750/ Moderate M.0:375; 0:500; 0:625/ Poor P.0:250; 0:375; 0:500/ Fairly Poor FP.0:125; 0:250; 0:375/ Very Poor VP.0:000; 0:125; 0:250/ Negligeable N.0:000; 0:000; 0:125/

208 G. Büyüközkan Table 4 The linguistic values provided by E 1 W 1 W 2 W 3 W 4 W 5 W 6 W 7 W 8 W 9 W 10 W 11 W 12 W 13 Weight c 1 VG FG G FG VG VG G G G G G FG FG M c 2 FG G FG G G G G FG FG G G FG G H c 3 VG FG VG G VG VG FG FG VG FG VG VG VG L c 4 FP FP VP FP P G P VP P VP P FP N M c 5 VG FG M VG O VG O FG VG M FG M G VH c 6 FP FP VP FP P FG FP VP FP FP M FP FP M Table 5 S j, R j and Q j values for D 0:7 S 0 Rank R 0 Rank Q Rank W 1 0.45 11 0.45 3 0.45 4 W 2 0.05 7 0.53 8 0.49 8 W 3 0.48 12 0.60 10 0.56 11 W 4 0.40 7 0.53 7 0.49 7 W 5 0.35 3 0.39 2 0.38 2 W 6 0.32 2 0.28 1 0.29 1 W 7 0.40 7 0.48 4 0.46 5 W 8 0.51 13 0.58 9 0.56 10 W 9 0.45 11 0.49 5 0.48 6 W 10 0.39 4 0.71 13 0.62 13 W 11 0.27 1 0.49 6 0.43 3 W 12 0.43 9 0.61 11 0.55 9 W 13 0.41 8 0.64 12 0.57 12 Step 3. The fuzzy decision matrix is constructed with the linguistic evaluations of experts. Then, the aggregated fuzzy weights of criteria and the aggregated fuzzy ratings of alternatives are calculated through the weighted fuzzy Delphi methodology. Step 4. The fuzzy decision matrix was normalized in this step. The normalized matrix is the same as one given in Step 3 because of the scale selected for linguistic variables. Step 5 and 6. S, R and Q values were computed by selecting D 0:7 and are shown in Table 5. Step 7. The evaluation results point out that the website of the Ministry of Health and Social Services has the best performance overall, followed by the website of the Ministry of National Education. Providing a set of compromise solutions as here is an important property of the suggested method since industrial service providers can adapt the best practices of different websites for an ideal system design. Note that the value of the weight v has a central role in the ranking of alternatives. A sensitivity analysis can be undertaken by setting systematically v to some values between 0 and 1 and by tracking the changes in the ranking. The results of such an analysis are presented in Table 6.

A MCDM Tool to Evaluate Government Websites 209 Table 6 Compromise solutions obtained by varying 0:1 0:3 0:5 0:7 0:9 Solutions W 6,W 5,W 11 W 6,W 5,W 11 W 6,W 5,W 11 W 6,W 5 W 6 4 Concluding Remarks This study proposed a fuzzy MCDM framework to effectively evaluate government website quality under a fuzzy environment. The approach helps to achieve an acceptable compromise of the maximum group utility of the majority and the minimum of the individual regret of the opponent. The method is able to treat linguistic assessments for the importance weights of criteria and the ratings of alternatives by using fuzzy logic. By this way, the ambiguities involved in the assessment data could be effectively represented and processed to assure a more convincing and effective evaluation process. An empirical case study of 13 Turkish government websites is used to illustrate the approach. The proposed method might be assisting the government agencies in making critical decisions during the improvement and/or development of government websites. Although the extended method presented in this study is applied to government websites evaluation problem, it can also be used to identify acceptable compromises in many website and/or e-service evaluation problems. In the future, we aim to consider evaluation criteria dependency which is generally less involved issue in MCDM methods. More precisely, we want to apply a decision framework based on the Choquet integral aggregation (Grabisch 1996), which takes into account interaction among evaluation criteria. We believe that it will be good practice to exploit this method for the government website evaluation problem and to compare the results. Acknowledgements This work has been financially supported by Galatasaray University Research Fund. The author would like to express deep gratitude to the industrial experts for their unlimited support in evaluation of the framework. References Akman, I., Yazicib, A., Mishra, A., & Arifoglu, A. (2005). E-government: a global view and an empirical evaluation of some attributes of citizens. Government Information Quarterly, 22, 239 257. Aladwani, A., & Palvia, P. (2002). Developing and validating an instrument for measuring userperceived web quality. Information & Management, 39, 467 476. Barnes, S., & Vidgen, R. (2003). Measuring web site quality improvements: a case study of the forum on strategic management knowledge exchange. Industrial Management & Data Systems, 103, 297 309. Bedell, S., Agrawal, A., & Petersen, L. (2004). A systematic critique of diabetes on the world wide web for patients and their physicians. International Journal of Medical Informatics, 73, 687 694.

210 G. Büyüközkan Bilsel, R., Buyukozkan, G., & Ruan, D. (2006). A fuzzy preference-ranking model for a quality evaluation of hospital web sites. International Journal of Intelligent Systems, 21, 1181 1197. Bojadziev, G., & Bojadziev, M. (1997). Fuzzy Logic for Business, Finance, and Management (Vol. 12), of Advances in Fuzzy Systems. Singapore: World Scientific. Cheng, C., & Lin, Y. (2002). Evaluating the best main battle tank using fuzzy decision theory with linguistic criteria evaluation. European Journal of Operational Research, 142, 174 186. Cox, J., & Dale, B. (2002). Key quality factors in web site design and use: an examination. International Journal of Quality & Reliability Management, 19, 862 888. Criado, J., & Ramilo, M. (2003). E-government in practice. The International Journal of Public Sector Management, 16, 191 218. Eschenfelder, K., Beachboard, J., McClure, C., & Wyman, S. (1997). Assessing U.S. federal government websites. Government Information Quarterly, 14, 173 189. Fortemps, P., & Roubens, M. (1996). Ranking and defuzzification methods based on area compensation. Fuzzy Sets and Systems, 82, 319 330. Grabisch, M. (1996). The application of fuzzy integrals in multi-criteria decision-making. European Journal of Operational Research, 89, 445 456. Grigoroudis, E., Litos, C., Moustakis, V., Politis, Y., & Tsironis, L. (2008). The assessment of userperceived web quality: application of a satisfaction benchmarking approach. European Journal of Operational Research, 187, 1346 1357. Kaufmann, A., & Gupta, M. (1988). Fuzzy mathematical models in engineering and management science. Amsterdam: North-Holland. Kaylor, C., Deshazo, R., & Eck, D. (2001). Gauging e-government: A report on implementing services among american cities. Government Information Quarterly, 18, 293 307. Li, Y., Tan, K., & Xie, M. (2002). Measuring web-based service quality. Total Quality Management, 13, 685 700. Opricovic, S., & Tzeng, G. (2004). Compromise solution by mcdm methods: a comparative analysis of vikor and topsis. European Journal of Operational Research, 156, 445 455. Parasuraman, A., Zeithaml, V., & Berry, L. (1988). Servqual: a multi-item scale for measuring consumer perceptions of service quality. Journal of Retailing, 64, 2 40. Pomerol, J.-C., & Barba Romero, S. (2000). Multicriterion decision in management: principles and practice. Norwell: Kluwer Academic Publishers. Smith, A. (2001). Applying evaluation criteria to new zealand government websites. International Journal of Information Management, 21, 137 149. Tan, K., M., X., & Li, Y. (2003). A service quality framework for web based information systems. The TQM Magazine, 15, 164 172. Toms, E., & Taves, A. (2004). Measuring user perceptions of web site reputation. Information Processing and Management, 40, 291 317. Tzeng, G., Lin, C., & Opricovic, S. (2005a). Multi-criteria analysis of alternative-fuel buses for public transportation. Energy Policy, 33, 1373 1383. Tzeng, G.-H., Yang, Y.-P., Lin, C.-T., & Chen, C.-B. (2005b). Hierarchical madm with fuzzy integral for evaluating enterprise intranet web sites. Information Sciences, 169, 409 426. Van der Merwe, R., & Bekker, J. (2003). A framework and methodology for evaluating e-commerce web sites. Internet Research: Electronic Networking Applications and Policy, 13, 330 341. West, D. (2007). Global e-government. Technical report, Center for Public Policy, Brown University. Zadeh, L. (1975). The concept of a linguistic variable and its applications to approximate reasoning. Information Sciences, 8, 199 249 (I), 301 357 (II). Zweers, K., & Planque, K. (2001). Electronic government: From an organizational based perspective towards a client oriented approach. In J. Prins (Ed.), Designing E-government. Boston: Kluwer Law International.