Using VIKOR Methodology for Ranking Universities by Academic Performance

Transcription

1 Using VIKOR Methodology for Ranking Universities by Academic Performance Seyhan Nisel Quantitative Methods Department School of Business, Istanbul University Istanbul, Turkey Rauf Nisel Department of Quantitative Methods Faculty of Business Administration, Marmara University Istanbul, Turkey Abstract Ranking universities by academic performance can be considered a multicriteria complex problem. Several ranking systems in the academic world have been implemented by the use of many criteria. In an effort to address the need of alternative academic performance evaluation systems, the purpose of this study is to provide a multi-criteria model for ranking Turkish universities by academic performance. The model uses the VIKOR (Vise Kriterijumska Optimizacija I Kompromisno Resenje) tool that is a novel methodology developed for solving multi-criteria problems with conflicting and complex criteria. In the study, universities were ranked in accordance with the evaluation criteria such as number of articles, number of citations, number of PhD students, faculty student ratio and so on. The findings showed that the alternative ranking approach can be applied to obtain the overall academic performance of universities. Keywords- University, Academic Ranking, VIKOR Method, Performance Evaluation, MCDM I. INTRODUCTION Recently universities have tried to obtain innovative vision by performing contemporary methods under the conditions of globalized world for providing better education and research services. Acquisition of a prestigious position among the best performing universities, one should measure effectively the existing service quality under the heading of progress in performance. Determination of weaknesses and strengths and making effective strategic planning and its effective application must also be considered parts of this progress. That is why universities periodically need to know their positions with respect to other universities at national and international level in order to have better assessments of their progresses. Various rankings consider measures of research excellence, different aspects of education, student choices, demographics, and some other indicators. Some rankings evaluate universities within a single country, while others assess universities worldwide. Rankings are conducted by magazines, newspapers, governments and academics. The subject has produced much debate about rankings' usefulness and accuracy. The expanding diversity in rating methodologies and accompanying criticisms of each indicate the lack of consensus in the field. We believe that diversity in rating methodologies and accompanying criticisms will help to improve existing methodologies and open accesses to new methodologies. This will create a positive impact on the quality of measurement. The purpose of our study is to introduce an alternative way for evaluating the performances of universities by using multi criteria ranking model. In our academic performance ranking model, 42 public universities in Turkey were taken into consideration as a sample data and VIKOR (Vise Kriterijumska Optimizacija I Kompromisno Resenje) method was used for ranking firstly with equal weights of criteria. In the following stage, varying weights were used for the ranking criteria. Varying weights were computed with respect to the sizes of variations of the normalized variables which were used as criteria of ranking. In the final stage results with equal weights and varying weights were compared and findings of the analysis showed that VIKOR method could be used as a consistent and an effective model for ranking of academic performances of universities. II. WORLD-WIDE UNIVERSITY RANKING SYSTEMS Worldwide university ranking systems are based on complicated process using several conflicting indicators for the evaluation and rankings of university at international level. Nowadays world-wide rankings have been conducted by several organizations and institutions and the rankings also have been conducted at national level by most countries. Multiindicator ranking of global universities has attracted a great deal of attention from universities, governments, students, business organizations and media. Some of the rankings of world universities are regarded to be most influential and widely observed. For instance, The Academic Ranking of World Universities (ARWU ranking)[2,3], by the Institute of Higher Education of Shanghai Jiao Tong University, Times Higher Education World University Rankings (THE ranking) by Thomson Reuters DOI: / _ GSTF

2 partnership)[45], World University Rankings published by U.S. News & World Report as the World s Best Universities (QS ranking) [46], Leiden Ranking [25,27] by QS Centre for Science and Technology Studies (CWTS) of Leiden University, and The Performance Ranking of Scientific Papers for World Universities (HEEACT ranking) [17] by Higher Education Evaluation and Accreditation Council of Taiwan (HEEACT) are well-known university ranking systems in the world. Other than the mentioned ranking systems, there are other ranking systems exist at regional, national and international level. All the ranking systems have received variety of criticism. There has been a considerable amount of studies in academic literature where structures, concepts, methodologies, criteria, weights and indicators compared and criticised. And in some of the studies, the ranking systems have been compared from general perspective [8,19,31,42]. The similarities between the ranking systems have been discussed in some other studies. In their study, Casal et al. compare ARWU, THE, CEST and Asia Week. Their results show that although some indicators differ considerably across selected rankings and even many indicators are unique, indicators referred to research and scientific productivity from university academic staff have a prominent role across all approaches[9]. In another study, Aguillo et al. compare ARWU, THE (QS), WR, (Web Ranking of World Universities by the Cybermetrics Lab), HEEACT and Leiden ranking systems. The findings show that there are reasonable similarities between the rankings, even though each applies a different methodology. The highest similarities were observed between the Taiwanese and the Leiden rankings from European universities[18]. Chen et al., in their study, consider these ranking activities an important phenomena and aim to investigate correlation of different ranking systems taking bibliometric approach[24]. In Waltman s study, Leiden ranking system has been discussed in detail[25]. Huang in his study, discusses several debates on QS rankings. The study provides a comprehensive discussion of the indicators and weightings adopted in the QS survey[30]. In some studies, the ranking systems have been compared in terms of weaknesses and strengths of systems. In their study, Jovanovic et al. stress potential weaknesses in the ARWU rankings in regard to the fact that the normalization of data significantly alters the rankings[32].in another study, Dehon et al. use robust principal component analysis to uncover the underlying factors measured by ARWU ranking. They found out that the ARWU rankings reflect not one but two different and uncorrelated aspects of academic research: overall research output and top-notch researchers, and the relative weight placed upon these two factors determines to a large extent the final ranking[4]. For ARWU ranking, Jeremic et al. propose the statistical I-distance method as they address that ARWU rankings are sensitive to the relative weight they attribute to each variable[48]. Kivinen and Hedman, in their study, they test the reliability of the ARWU ranking with regard to the academic performance of 25 Scandinavian universities. By resampling and reanalyzing the ARWU data, they proposes an input output analysis for measuring universities scientific productivity with special emphasis on those universities which meet the productivity threshold. They conclude that consequently the ranking based on scientific productivity deviates significantly from the ARWU[36]. However, in another study, Taylor and Braddock provides critical examination of the THE and ARWU rankings and argue that ARWU ranking, although far from perfect, is a better indicator of university excellence [38]. Some studies in the literature consider the data or chosen indicators in the ranking systems. According to Benito and Romero, university rankings usually provide a single score that situate institutions in the worldwide context or in their national context, this simplicity can be highly misleading in that most rankings are based on a simple formula that aggregate subjectively, most of the times, chosen indicators [28]. And Ioannidis says that despite the failure of current international ranking systems, reliable information on specific performance indices may be useful, if properly analyzed and interpreted [21]. If the studies can be examined extensively in which the ranking systems and bibliometric techniques are criticised, we believe that usage of quantitative decision models in support of the analysis of mentioned systems and techniques will provide more reliable results. For instance, Billaut proposes a critical analysis of ARWU ranking in terms of criteria used, methodology and structuring. The author conclude that ARWU do not qualify as a useful and pertinent tool to discuss the quality of academic institutions but suggest that the combination of OR with sophisticated bibliometric techniques will offer more reliable rankings. [20]. In literature, the studies in which particularly MCDM techniques are recommended as alternative ranking methods, support this approach. For instance, Giannoulis and Ishizaka, by the use of ELECTRE III methodology, present an alternatif ranking system, reflecting personal preferences, where information is uncertain and vague. The study describes a three-tier Web-system, which produces a customized ranking of British Universities [5]. In another study, Lukman introduces a model, which is enabling a comparison between universities regarding research, educational and environmental performances, by using the analytic hierarchy process (AHP) methodology for assigning criteria weights [39]. Finally Wu et al., similar to our study, apply a hybrid multiple-criteria decision making (MCDM) model to rank 12 private universities in Taiwan by using VIKOR method and by utilizing the performance evaluation indices weighted by the AHP methodology. They suggest that universities are able to self-evaluate by using the varied, weighted evaluation indices complied by the model [15]. III. METHOD The VIKOR method, which was first introduced by Opricovic and Tzeng, is a novel multi-criteria methodology that can be defined as a multi-criteria optimization of complex systems[10,43,44]. The method focuses on ranking and selecting from a set of alternatives in the presence of conflicting criteria[44]. Assuming that each alternative is evaluated according to each criterion function, the compromise ranking could be performed by comparing the measure of closeness to the ideal alternative. The compromise solutions could be the basis for negotiations, involving the preference of GSTF

3 decision makers by criteria weights[43,44]. VIKOR is a helpful tool in multi-criteria decision making, particularly in a situation where the decision maker is not able, or does not know to express his/her preference at the beginning of system design [44]. Since its introduction, The VIKOR methodology has been used in many areas such as airline service quality evaluation [23,34], banking performance[14], assessing the risk of general anesthesia process[12], safety evaluation of the bidding products[41], material selection[1,49], vendor selection[6], construction project selection[40], performance evaluation of aircraft maintenance staff[13], contractor selection[22], improving tourism policy implementation[7], railway route planning[33], insurance company selection[11], financial performance evaluation[35], evaluating airlines websites effectiveness[50], assessment of university innovation capital indicators[16], optimization of multi-response processes[26], and finally university performance ranking [15]. IV. AN EMPIRICAL STUDY In this study multi criteria ranking model is introduced as an alternative way for evaluating the academic performances of universities. Academic performance measures of 42 public universities in Turkey were taken into consideration as a sample data and following conditions were considered in order to increase the homogeneity and reliability of the sample data. 1. To be a public university, 2. To be established before the year 2000, 3. Having a school of medicine, 4. Having more than one field of study. First, VIKOR (Vise Kriterijumska Optimizacija I Kompromisno Resenje) method was used with equal weights for ranking universities. In the following stage, varying weights were used for the ranking. Varying weights were computed with respect to the sizes of variations of the normalized variables, which was different way than Wu et al. s study that uses AHP methodology for weighting criteria.[15]. In the final stage, results with equal weights and varying weights were compared and findings of the analysis showed that VIKOR method could be used as a consistent and an effective model for ranking of academic performances of universities. Universities were ranked only on the basis of their academic performance and this factor was also considered in selection of the criteria. URAP (University Ranking by Academic Performance Research Laboratory at Informatics Institute of Middle East Technical University in Turkey) [47] was mostly the base for selection appropriate criteria. Moreover, some modifications have been made on the ranking criteria based on the literature review. Definitions of total of 10 criteria which were used in the model are as follows: 1. Total number of articles: Total number of articles published in 2011 and indexed by Web of Science. 2. Number of articles per faculty: Total number of articles published in 2011 and indexed by Web of Science divided by the number of faculty in Rate of articles written in English: Total number of articles written in English published in 2011 divided by the total number of articles published in 2011 (indexed by Web of Science). 4. Total number of citations: Total number of citations received between years for the articles published between years of (indexed by Web of Science). 5. Number of citations per faculty: Total number of citations received between years for the articles published between years of (indexed by Web of Science) divided by the number of faculty in Total number of publications: Total number of publications published between years and indexed by Web of Science. 7. Number of publications per faculty: Total number of publications published between years (indexed by Web of Science) divided by the number of faculty in Total number of PhD students: Total number of PhD students by academic year. 9. Faculty student ratio: Total number of students divided by the total number of faculty. 10. Rate of PhD students: Total number of PhD students divided by the total number of students ( academic year). Increase in the number of PhD students and respectively increase in the faculty staff will have an impact on total number of articles published. That is why total numbers in criteria were not be sole factors in terms of sufficiency of criteria. That is why number of publications per faculty and number of articles per faculty were considered. Also rate of articles written in English also was added to the model due to articles written in English would have a larger target population. The data were obtained from websites of ISI Web of Science Database and Evaluation, Selection and Placement Center in Turkey (OSYM)[37]. At the first stage equal weights were considered and S,R and Q values were computed for each alternative. Later, varying weights of criteria were computed on the basis of a range of variations of normalized values for each criterion. Then S, R and Q values were computed from varying weighted criteria for second ranking. In both rankings, v value was considered 0.5 (the maximum group utility). All results were summarized in Table 1 in Appendix. In the Table 1, U23 was the best ranked university, when Q values were obtained from criteria with equal weights. U23 and U40 satisfied the condition of C2, having acceptable stability due to similarities of S, R and Q ranking values. And U4, U5, U16, U17, U19, U20, U21, U23, U25, U40 had acceptable advantage as they satisfied condition C1. U23 was again best ranked university in Turkey by URAP ranking in GSTF

4 In the first ranking, indicators were weighted equally in order determine general position of 42 universities with respect to each other. U23, U5,U26, U6 and U16 were 5 best ranked respectively. Although indicators used in our ranking were extremely similar to the indicators which were used by URAP, their classifications of indicators by ownership and field of study were as not homogeneous as to make a comparison with our ranking results. Our ranking results were consistent with well-known best performing, prestigious universities. First 5 universities in our ranking have considerable reputation as best performing universities in Turkey. And also C1 and C2 conditions in VIKOR algorithms show to what extent ranked universities can keep their position stable in the case if they were ranked again under different condition by using two different type of information as acceptable advantage and acceptable stability. This situation reflected one of the strengths of VIKOR method. One of the most discussed subject in ranking systems finding a concrete and objective base for weighting the criteria. In ranking problems weights of criteria can be determined by AHP, ANP (Analytic Network Process) like MCDM methods. VIKOR method can be used with these mentioned methods. Although subjective judgments can be transformed into objective values by experts opinions used in such methods like AHP and ANP, there can be still a degree of subjectivity to a certain extent in experts opinions. That is why instead of using subjective assessment for weighting criteria in our ranking model we use simple but more meaningful method that has inherited within the characteristic for fundamental principle in ranking which is variation. It is obvious that if the data of a criterion has no variation at all, in other words if all values are same for that particular criterion there will be no ranking power. We assumed that the criterion with a higher variation would have a higher precision in other words higher importance in determination of ranks of universities. We eliminated effect of different measurement units and magnitude of values on the data of each criterion by normalizing data and obtained different range of normalized values for each data set. These ranges were used in computation of weights of criteria. These weights are shown in the Table 5 where faculty student ratio has the highest weight of 12.6% and total number of articles has the lowest of 8.2%. The second ranking of universities where varying weights were used was shown in the Table 1. Results showed that two rankings systems with equal and varying weights had minor difference in terms of order of the places of universities in ranking. But we recommended that weighting of criteria in VIKOR method minimized outlier effect on results in case some criterion had an outlier problem even the data was normalized. But as we have mentioned before two ranking systems showed insignificant differences when Q ranking values were compared. In the Table 1, Kendall s coefficient of concordance [29] showed almost a complete agreement in between the results of two ranking systems. V. CONCLUSION As we have mentioned before released results of all reputable worldwide academic performance ranking systems were subject to continuous criticisms. Sources of criticisms mainly concentrated on differences in structures, methodologies, criteria or indicators. The main debate was about determination of accuracy and reliability of the different results which were obtained from different ranking systems. Ranking of universities in different fields based on various performance measures is difficult and complex decision problem. There is a need for extensive analysis of the differences in results of ranking systems of academic performances in terms of structures and methodologies, indicators used, criteria selected. It is obvious that more sophisticated algorithms are needed together with bibliometric techniques. One of the reasons of using VIKOR method in our analysis is to be as responsive as possible for needs of this field. The other reasons of using VIKOR as the main tool for the analysis were, having novel and effective characteristics, easy to understand, having a practical algorithm, working integrated with different quantitative models, successful aggregation of data in different measurement units. In our analysis VIKOR method proved itself to be a strong alternative method with respect to other existing ranking methods in terms of the mentioned characteristics and consistency of the results obtained under different conditions. REFERENCES [1] A. C., R. Vaish, A comparative study on material selection for micro-lectromechanical systems, Materials and Design, vol. 41, pp , [2] ARWU, Shanghai Jiao Tong University, Academic Ranking of World Universities, available online 14 Dec 2012 from [3] ARWU, Academic Ranking of World Universities, available online 14 Dec 2012 from ARWUMethodology2010.jsp [4] C. Dehon, A. McCathie, V. Verardi, Uncovering excellence in academic rankings: a closer look at the Shanghai ranking, Scientometrics, vol. 83, pp , [5] C. Giannoulis, A. Ishizaka, AWeb-based decision support system with ELECTRE III for a personalised ranking of British universities, Decision Support Systems, vol. 48, pp , [6] C. H. Hsu, F. K. Wang, G. H. Tzeng, The best vendor selection for conducting the recycled material based on a hybrid MCDM model combining DANP with VIKOR, Resources, Conservation and Recycling, vol. 66, pp , [7] C. H. Liu, G. H. Tzeng, M. H. Lee, Improving tourism policy implementation - The use of hybrid MCDM models, Tourism Management, vol. 33, pp , [8] D. D. Dill, M. Soo, Academic Quality, League Tables, and Public Policy: A Cross-National Analysis of University Ranking Systems, Higher Education, Vol. 49(4), pp , June2005. [9] G. Buela-Casal, Olga G. Martinez, M. P. Bermudez-Sanchez, O. Vadillo-Munoz, Comparative study of international academic rankings of universities, Scientometrics, vol. 71(3), pp , June GSTF

5 [10] G. H. Tzeng, C. W. Lin, S. Opricovic, Multi-criteria analysis of alternative-fuel buses for public transportation, Energy Policy, vol. 33, pp , [11] G. N. Yucenur, N. C. Demirel, Group decision making process for insurance company selection problem with extended VIKOR method under fuzzy environment, Expert Systems with Applications, vol. 39, pp , [12] H. C. Liu, L. Liu, N. Liu, L. X. Mao, Risk evaluation in failure mode and effects analysis with extended VIKOR method under fuzzy environment, Expert Systems with Applications, vol. 39,pp , [13] H. Y. Wu, J. K. Chen, I.S. Chen, Performance evaluation of aircraft maintenance staff using a fuzzy MCDM approach, International Journal Of Innovative Computing Information and Control, Vol. 8(6), pp , June [14] H. Y. Wu, G. H. Tzeng, Y. H. Chen, A fuzzy MCDM approach for evaluating banking performance based on Balanced Scorecard, Expert Systems with Applications, vol. 36,pp , [15] H. Y. Wu, J. K. Chen, I. S. Chen, H. H. Zhuo, Ranking universities based on performance evaluation by a hybrid MCDM model, Measurement, vol. 45, pp , [16] H. Y. Wu, J. K. Chen, I S. Chen, Innovation capital indicator assessment of Taiwanese Universities: A hybrid fuzzy model application, Expert Systems with Applications, vol. 37, pp , [17] Higher Education Evaluation and Accreditation Council of Taiwan, Performance Ranking of Scientific Papers for World Universities, available online 14 Dec 2012 from [18] I. F. Aguillo, J. Bar-Ilan, M. Levene, J. L. Ortega, Comparing university rankings, Scientometrics, vol. 85, pp , [19] I. Stolz, D. D. Hendel, A. S. Horn, Ranking of rankings: benchmarking twenty-five higher education ranking systems in Europe, Higher Education, vol. 60, pp , [20] J. C. Billaut, D. Bouyssou, P. Vincke, Should you believe in the Shanghai ranking? An MCDM view, Scientometrics, vol. 84, pp , [21] J. P. A. Ioannidis et al., International ranking systems for universities and institutions: a critical appraisal, BMC Medicine, vol. 5(30), October [22] J. R. San Cristobal, Contractor selection using multicriteria decision-making methods, Journal Of Construction Engineering and Management, Vol. 138(6) pp , June [23] J.H. Liou, C.Y. Tsai, R.H. Lin, G. H. Tzeng, A modified VIKOR multiple-criteria decision method for improving domestic airlines service quality, Journal of Air Transport Management, vol. 17, pp , [24] K. H. Chen, P. Y. Liao, A comparative study on world university rankings: a bibliometric survey, Scientometrics, vol. 92, pp , [25] L. Waltman, The Leiden ranking 2011/2012: Data collection, indicators, and interpretation, Journal of the American Society for Information Science and Technology, vol. 63(12), pp , [26] L.I. Tong, C. C. Chen, C. H. Wang, Optimization of multiresponse processes using the VIKOR method,international Journal of Advanced Manufacturing Technology, vol. 31, pp , [27] Leiden Ranking, available online 14 Dec 2012 from [28] M. Benito, R. Romera, Improving quality assessment of composite indicators in university rankings: a case study of French and German universities of excellence, Scientometrics, vol. 89, pp , [29] M. G. Kendall, Trust the treatment of ties in ranking problems, Biometrika, Vol. 33(3), Nov. 1945, pp [30] M. H. Huang, Opening the black box of QS World University Rankings, Research Evaluation, vol. 21, pp , [31] M. H. Huang, A comparison of three major academic rankings for world universities: From a research evaluation perspective, Journal of Library and Information Studies, vol. 9(1), pp.1-25, June [32] M. Jovanovic, V. Jeremic, G. Savic, M. Bulajic, M. Martic, How does the normalization of data affect the ARWU ranking?, Scientometrics, vol. 93, pp , [33] M. Kosijer, M. Ivic, M. Markovic, M. Belosevic, I, Multicriteria decision-making in railway route planning and design, Gradevinar, Vol. 64(3), pp , March [34] M. S. Kuo, G. S. Liang, Combining VIKOR with GRA techniques to evaluate service quality of airports under fuzzy environment, Expert Systems with Applications, vol. 38, pp , [35] N. Yalcin, A. Bayrakdaroglu, C. Kahraman, Application of fuzzy multi-criteria decision making methods for financial performance evaluation of Turkish manufacturing industries, Expert Systems with Applications, vol. 39, pp , [36] O. Kivinen, J. Hedman, World-wide university rankings: A Scandinavian approach, Scientometrics, Vol. 74(3), pp , [37] OSYM, available online [38] P. Taylor, R. Braddock, International university ranking systems and the idea of university excellence, Journal of Higher Education Policy and Management, vol. 29(3), pp , [39] R. Lukman, D. Krajnc, P. Glavic, University ranking using research, educational and environmental indicators, Journal of Cleaner Production, vol. 18, pp , [40] S. Ebrahimnejad, S.M. Mousavi, R. Tavakkoli-Moghaddam, H. Hashemi, B. Vahdani, A novel two-phase group decision making approach for construction project selection in a fuzzy environment, Applied Mathematical Modelling, vol. 36, pp , [41] S. H. Bin, L. Hong, L. Lu, W. Li, A synthetic evaluation model for consumer products quality safety based on the VIKOR method, Information - An International Interdisciplinary Journal, Vol.15(10) pp , October [42] S. Marginson, Global University Rankings: Implications in general and for Australia, Journal of Higher Education Policy and Management, vol. 29(2), pp , [43] S. Opricovic, G. H. Tzeng, Extended VIKOR method in comparison with outranking methods, European Journal of Operational Research, vol. 178, pp , [44] S. Opricovic, G. H. Tzeng, Compromise Solution by MCDM Methods: A comparative analysis of VIKOR and TOPSIS, European Journal of Operational Research, vol. 156, pp , [45] THE Times Higher Education World University Rankings, available online 14 Dec 2012 from [46] Understanding QS world university rankings methodology, Available online 22 Dec 2012 from [47] URAP, University Ranking by Academic Performance, available online 12 Dec 2012 from [48] V. Jeremic, M. Bulajic, M. Martic, Z. Radojicic, A fresh approach to evaluating the academic ranking of world universities, Scientometrics, vol. 87, pp , [49] V. M. Athawale, S. Chakraborty, Material selection using multicriteria decision-making methods: a comparative study, Proceedings of the Institution of Mechanical Engineers, Part L: Journal of Materials Design and Applications, pp , [50] W. H. Tsai, W. C. Chou, J. D. Leu, An effectiveness evaluation model for the web-based marketing of the airline industry, Expert Systems with Applications, vol. 38, pp , GSTF

6 TABLE 1: THE VIKOR RANKING RESULTS OF UNIVERSITIES First Ranking Second Ranking Univ. # Univ. Name Q J Rank (Q J) U1 Abant Izzet Baysal Univ U2 Adnan Menderes Univ U3 Afyon Kocatepe Univ U4 Akdeniz Univ U5 Ankara Univ U6 Ataturk Univ U7 Balikesir Univ U8 Bulent Ecevit Univ U9 Celal Bayar Univ U10 Cumhuriyet Univ U11 Canakkale Onsekiz Mart Univ U12 Cukurova Univ U13 Dicle Univ U14 Dokuz Eylul Univ U15 Dumlupinar Univ U16 Ege Univ U17 Erciyes Univ U18 Eskisehir Osmangazi Univ U19 Firat Univ U20 Gazi Univ U21 Gaziantep Univ U22 Gaziosman Pasa Univ U23 Hacettepe Univ U24 Harran Univ U25 Inonu Univ U26 Istanbul Univ U27 Kafkas Univ U28 Kahramanmaras Sutcu Imam Univ U29 Karadeniz Teknik Univ U30 Kirikkale Univ U31 Kocaeli Univ U32 Marmara Univ U33 Mersin Univ U34 Mustafa Kemal Univ U35 Ondokuz Mayis Univ U36 Pamukkale Univ U37 Sakarya Univ U38 Selcuk Univ U39 Suleyman Demirel Univ U40 Trakya Univ U41 Uludag Univ U42 Yuzuncu Yil Univ Criteria weights in the second ranking C1 = C2 = Kendall's W a C 1 C 2 Univ. # Univ. Name Q J Rank (Q J) C 1 C 2 35 NO NO U1 Abant Izzet Baysal Univ NO NO 36 NO NO U2 Adnan Menderes Univ NO NO 33 NO NO U3 Afyon Kocatepe Univ NO NO 16 YES NO U4 Akdeniz Univ NO NO 2 YES NO U5 Ankara Univ YES NO 4 NO NO U6 Ataturk Univ NO NO 42 NO NO U7 Balikesir Univ YES NO 34 NO NO U8 Bulent Ecevit Univ NO NO 32 NO NO U9 Celal Bayar Univ NO NO 38 NO NO U10 Cumhuriyet Univ NO NO 24 NO NO U11 Canakkale Onsekiz Mart Univ NO NO 8 NO NO U12 Cukurova Univ NO NO 22 NO NO U13 Dicle Univ NO NO 17 NO NO U14 Dokuz Eylul Univ YES NO 37 NO NO U15 Dumlupinar Univ NO NO 5 YES NO U16 Ege Univ YES NO 6 YES NO U17 Erciyes Univ YES NO 20 NO NO U18 Eskisehir Osmangazi Univ NO NO 10 YES NO U19 Firat Univ YES NO 12 YES NO U20 Gazi Univ NO NO 9 YES NO U21 Gaziantep Univ YES NO 21 NO NO U22 Gaziosman Pasa Univ NO NO 1 YES YES U23 Hacettepe Univ YES YES 26 NO NO U24 Harran Univ NO NO 18 YES NO U25 Inonu Univ NO NO 3 NO NO U26 Istanbul Univ YES NO 31 NO NO U27 Kafkas Univ NO NO 39 NO NO U28 Kahramanmaras Sutcu Imam Univ NO NO 13 NO NO U29 Karadeniz Teknik Univ NO NO 29 NO NO U30 Kirikkale Univ NO NO 28 NO NO U31 Kocaeli Univ NO NO 7 NO NO U32 Marmara Univ YES NO 30 NO NO U33 Mersin Univ NO NO 27 NO NO U34 Mustafa Kemal Univ NO NO 11 NO NO U35 Ondokuz Mayis Univ YES NO 25 NO NO U36 Pamukkale Univ NO NO 40 NO NO U37 Sakarya Univ NO NO 14 NO NO U38 Selcuk Univ YES NO 15 NO NO U39 Suleyman Demirel Univ NO NO 41 YES YES U40 Trakya Univ YES NO 19 NO NO U41 Uludag Univ NO NO 23 NO NO U42 Yuzuncu Yil Univ YES NO C3 = C4 = C5 = C6 = C7 = C8 = C9 = C10 = GSTF

7 Seyhan Nisel is an associate professor of Quantitative Methods at School of Business, Istanbul University. She earned her BS in Business Administration from the Istanbul University and her MA and PhD in Quantitative Methods from the Social Science Institute of the Istanbul University. Her PhD thesis was on Ranking Cities of Turkey by Livability Using Analytic Hierarchy Process. In 2007, she was visiting research scholar at Operations and Decision Technologies Department at Kelley School of Business, Indiana University. Her primary fields of research are spreadsheet modeling, performance measurement and ranking, multicriteria decision making in business, simulation optimization, and management science applications in sport. She has several national and international articles published in reputed academic journals. She is an associate editor of the Journal of the School of Business Administration, Istanbul University. She works for Quantitative Methods Department, School of Business since Professor Rauf Nisel holds the Chair of Quantitative Methods at Faculty of Business Administration, Marmara University since He received his BA in Business Administration from Bosphorus University and his MA in Business Administration and his PhD in Statistics from Istanbul University. He was a visiting professor at Glasgow University and Strathclyde University. His primary areas of academic research are related to multivariate statistical modeling, decision science and business surveys in various fields GSTF