Challenges in Ranking of Universities

Transcription

1 Challenges in Ranking of Universities First International Conference on World Class Universities, Jaio Tong University, Shanghai, June 16-18, 2005 Anthony F.J. van Raan Center for Science and technology Studies(CWTS) Leiden University

2

3

4 One Basic Question: How can we identify the best universities in the world?

5 6 Research Questions: 1. Research or Teaching? 2. How to Measure Performance? 3. For all universities in the world? 4. One numerical value? 5. Significance of positions? 6. How many?

6 Two most influential international rankings: Shanghai Jiao Tong University (60% bibliom.) Times Higher Education Supplement (20% bibliom.)

7 Important national rankings: Germany: CHE DFG

8 We can ask experts for their judgment

9 VOLUME 88, Number 13 PHYSICAL REVIEW LETTERS 1 April 2002 VOLUME 88, Number 13 PHYSICAL REVIEW LETTERS 1 April 2002 VOLUME 88, Number 13 PHYSICAL REVIEW LETTERS 1 April 2002 or let scientific output Truncation and of Power Law Behavior in Scale-Free Network Models its impact speak: Truncation of Power Law Behavior in Scale-Free Network Models due to Information Filtering due to Information Filtering Stefano Mossa,1,2 Marc Barthélémy,3 H. Eugene Stanley,1 Luís A. Nunes Amaral1 Stefano Mossa,1,2 Marc Barthélémy,3 H. Eugene Stanley,1 and Luís A. Nunes Amaral1 1 Center for Polymer Studies Department of Physics, Boston University, Boston, Massachusetts Center for Polymer Studies and Department of Physics, Boston University, Boston, Massachusetts Dipartimento di Fisica, INFM UdR, INFM Center for Statistical Mechanics Complexity, Universitàdi Roma La Sapienza, Piazzale Aldo Moro 2, I-00185, Roma, Italy 2 Dipartimento di Fisica, INFM UdR, and INFM Center for Statistical Mechanics and Complexity, 3 CEA-Service de Physique de la Matière Condensée, BP 12, Bruyères-le-Châtel, France Universitàdi Roma La Sapienza, Piazzale Aldo Moro 2, I-00185, Roma, Italy (Received 18 October 2001; published 14 March 2002) 3 CEA-Service de Physique de la Matière Condensée, BP 12, Bruyères-le-Châtel, France (Received 18 October 2001; published 14 March 2002) We formulate a general model for the growth of scale-free networks under filtering information conditions that is, when the nodes can process information about only a subset of the existing nodes in the We formulate a general model for the growth of scale-free networks under filtering information network. We find that the distribution of the number of incoming links to a node follows a universal scaling conditions that is, when the nodes can process information about only a subset of the existing nodes in the form, i.e., that it decays as a power law with an exponential truncation controlled not only by the system size network. We find that the distribution of the number of incoming links to a node follows a universal scaling bibliometric but also by a feature not previously analysis considered, the subset of the network accessible to the node. We test our form, i.e., that it decays as a power law with an exponential truncation controlled not only by the system size model with empirical data for the World Wide Web and find agreement. but also by a feature not previously considered, the subset of the network accessible to the node. We test our model with empirical data for the World Wide Web and find agreement. DOI: /PhysRevLett PACS numbers: Hh, i, Da, Hc DOI: /PhysRevLett PACS numbers: Hh, i, Da, Hc There is a great deal of current interest in understanding the structure and growth mechanisms of global networks [1 3], such as the World Wide Web (WWW) [4,5] and the Internet [6]. Network structure critical in many contexts such as Internet attacks [2], spread of virus [7], or dynamics of human epidemics [8]. In all these problems, the nodes with the largest number of links play important role on the dynamics of the There is a great deal of current interest in understanding the structure and growth mechanisms of global networks [1 3], such as the World Wide system. It is therefore important to know the global structure of the network as well as its precise distribution of the number of links. Web (WWW) [4,5] and the Internet [6]. Network structure is critical in many contexts such as Internet attacks [2], spread of an virus [7], or Recent empirical studies report that both the Internet and the WWW have scale-free properties; that is, the number of incoming links and the dynamics of human epidemics [8]. In all these problems, the nodes with the largest number of links play an important role on the dynamics of the number of outgoing links at a given node have distributions that decay with power law tails [4 6]. It has been proposed [9] that the scale-free system. It is therefore important to know the global structure of the network as well as its precise distribution of the number of links. structure of the Internet and the WWW may be explained by a mechanism referred to as preferential attachment [10] in which new nodes link Recent empirical studies report that both the Internet and the WWW have scale-free properties; that is, the number of incoming links and the to existing nodes with a probability proportional to the number of existing links to these nodes. Here we focus on the stochastic character of the number of outgoing links at a given node have distributions that decay with power law tails [4 6]. It has been proposed [9] that the scale-free preferential attachment mechanism, which we understand in the following way: New nodes want to connect to the existing nodes with the largest structure of the Internet and the WWW may be explained by a mechanism referred to as preferential attachment [10] in which new nodes link number of links i.e., with the largest degree because of the advantages offered by being linked to a well-connected node. For a large network it to existing nodes with a probability proportional to the number of existing links to these nodes. Here we focus on the stochastic character of the is not plausible that a new node will know the degrees of all existing nodes, so a new node must make a decision on which node to connect with preferential attachment mechanism, which we understand in the following way: New nodes want to connect to the existing nodes with the largest based on what information it has about the state of the network. The preferential attachment mechanism then comes into play as nodes with a number of links i.e., with the largest degree because of the advantages offered by being linked to a well-connected node. For a large network it larger degree are more likely to become known. is not plausible that a new node will know the degrees of all existing nodes, so a new node must make a decision on which node to connect with VOLUME 88, Number 13 PHYSICAL REVIEW LETTERS 1 April 2002 based on what information it has about the state of the network. The preferential attachment mechanism then comes into play as nodes with a larger degree are more likely to become known. VOLUME 88, Number 13 PHYSICAL REVIEW LETTERS 1 April 2002 Truncation of Power Law Behavior in Scale-Free Network Models due to Information Filtering Truncation of Power Law Behavior in Scale-Free Network Models due to Information Filtering Stefano Mossa,1,2 Marc Barthélémy,3 H. Eugene Stanley,1 and Luís A. Nunes Amaral1 1 Center for Polymer Studies and Department of Physics, Boston University, Boston, Massachusetts Stefano Mossa,1,2 Marc Barthélémy,3 H. Eugene Stanley,1 and Luís A. Nunes Amaral1 2 Dipartimento di Fisica, INFM UdR, and INFM Center for Statistical Mechanics and Complexity, 1 Center for Polymer Studies and Department of Physics, Boston University, Boston, Massachusetts Universitàdi Roma La Sapienza, Piazzale Aldo Moro 2, I-00185, Roma, Italy 3 CEA-Service de Physique de la Matière Condensée, BP 12, Bruyères-le-Châtel, France 2 Dipartimento di Fisica, INFM UdR, and INFM Center for Statistical Mechanics and Complexity, (Received 18 October 2001; published 14 March 2002) Universitàdi Roma La Sapienza, Piazzale Aldo Moro 2, I-00185, Roma, Italy 3 CEA-Service de Physique de la Matière Condensée, BP 12, Bruyères-le-Châtel, France We formulate a general model for the growth of scale-free networks under filtering information (Received 18 October 2001; published 14 March 2002) conditions that is, when the nodes can process information about only a subset of the existing nodes in the network. We find that the distribution of the number of incoming links to a node follows a universal scaling We formulate a general model for the growth of scale-free networks under filtering information form, i.e., that it decays as a power law with an exponential truncation controlled not only by the system size conditions that is, when the nodes can process information about only a subset of the existing nodes in the but also by a feature not previously considered, the subset of the network accessible to the node. We test our network. We find that the distribution of the number of incoming links to a node follows a universal scaling model with empirical data for the World Wide Web and find agreement. form, i.e., that it decays as a power law with an exponential truncation controlled not only by the system size but also by a feature not previously considered, the subset of the network accessible to the node. We test our DOI: /PhysRevLett PACS numbers: Hh, i, Da, Hc model with empirical data for the World Wide Web and find agreement. DOI: /PhysRevLett PACS numbers: Hh, i, Da, Hc There is a great deal of current interest in understanding the structure and growth mechanisms of global networks [1 3], such as the World Wide Web (WWW) [4,5] and the Internet [6]. Network structure is critical in many contexts such as Internet attacks [2], spread of an virus [7], or dynamics of human epidemics [8]. In all these problems, the nodes with the largest number of links play an important role on the dynamics of the system. It is therefore important to know the global structure of the network as well as its precise distribution of the number of links. Recent empirical studies report that both the Internet and the WWW have scale-free properties; that is, the number of incoming links and the number of outgoing links at a given node have distributions that decay with power law tails [4 6]. It has been proposed [9] that the scale-free structure of the Internet and the WWW may be explained by a mechanism referred to as preferential attachment [10] in which new nodes link to existing nodes with a probability proportional to the number of existing links to these nodes. Here we focus on the stochastic character of the preferential attachment mechanism, which we understand in the following way: New nodes want to connect to the existing nodes with the largest number of links i.e., with the largest degree because of the advantages offered by being linked to a well-connected node. For a large network it is not plausible that a new node will know the degrees of all existing nodes, so a new node must make a decision on which node to connect with based on what information it has about the state of the network. The preferential attachment mechanism then comes into play as nodes with a larger degree are more likely to become known. There is a great deal of current interest in understanding the structure and growth mechanisms of global networks [1 3], such as the World Wide Web (WWW) [4,5] and the Internet [6]. Network structure is critical in many contexts such as Internet attacks [2], spread of an virus [7], or dynamics of human epidemics [8]. In all these problems, the nodes with the largest number of links play an important role on the dynamics of the system. It is therefore important to know the global structure of the network as well as its precise distribution of the number of links. Recent empirical studies report that both the Internet and the WWW have scale-free properties; that is, the number of incoming links and the number of outgoing links at a given node have distributions that decay with power law tails [4 6]. It has been proposed [9] that the scale-free structure of the Internet and the WWW may be explained by a mechanism referred to as preferential attachment [10] in which new nodes link to existing nodes with a probability proportional to the number of existing links to these nodes. Here we focus on the stochastic character of the preferential attachment mechanism, which we understand in the following way: New nodes want to connect to the existing nodes with the largest number of links i.e., with the largest degree because of the advantages offered by being linked to a well-connected node. For a large network it is not plausible that a new node will know the degrees of all existing nodes, so a new node must make a decision on which node to connect with based on what information it has about the state of the network. The preferential attachment mechanism then comes into play as nodes with a larger degree are more likely to become known. Truncation of Power Law Behavior in Scale-Free Network Models due to Information Filtering Stefano Mossa,1,2 Marc Barthélémy,3 H. Eugene Stanley,1 and Luís A. Nunes Amaral1 1 Center for Polymer Studies and Department of Physics, Boston University, Boston, Massachusetts Dipartimento di Fisica, INFM UdR, and INFM Center for Statistical Mechanics and Complexity, Universitàdi Roma La Sapienza, Piazzale Aldo Moro 2, I-00185, Roma, Italy 3 CEA-Service de Physique de la Matière Condensée, BP 12, Bruyères-le-Châtel, France (Received 18 October 2001; published 14 March 2002) We formulate a general model for the growth of scale-free networks under filtering information conditions that is, when the nodes can process information about only a subset of the existing nodes in the network. We find that the distribution of the number of incoming links to a node follows a universal scaling form, i.e., that it decays as a power law with an exponential truncation controlled not only by the system size but also by a feature not previously considered, the subset of the network accessible to the node. We test our model with empirical data for the World Wide Web and find agreement. DOI: /PhysRevLett PACS numbers: Hh, i, Da, Hc There is a great deal of current interest in understanding the structure and growth mechanisms of global networks [1 3], such as the World Wide Web (WWW) [4,5] and the Internet [6]. Network structure is critical in many contexts such as Internet attacks [2], spread of an virus [7], or dynamics of human epidemics [8]. In all these problems, the nodes with the largest number of links play an important role on the dynamics of the system. It is therefore important to know the global structure of the network as well as its precise distribution of the number of links. Recent empirical studies report that both the Internet and the WWW have scale-free properties; that is, the number of incoming links and the number of outgoing links at a given node have distributions that decay with power law tails [4 6]. It has been proposed [9] that the scale-free structure of the Internet and the WWW may be explained by a mechanism referred to as preferential attachment [10] in which new nodes link to existing nodes with a probability proportional to the number of existing links to these nodes. Here we focus on the stochastic character of the preferential attachment mechanism, which we understand in the following way: New nodes want to connect to the existing nodes with the largest number of links i.e., with the largest degree because of the advantages offered by being linked to a well-connected node. For a large network it is not plausible that a new node will know the degrees of all existing nodes, so a new node must make a decision on which node to connect with based on what information it has about the state of the network. The preferential attachment mechanism then comes into play as nodes with a larger degree are more likely to become known. VOLUME 88, Number 13 PHYSICAL REVIEW LETTERS 1 April 2002 Truncation of Power Law Behavior in Scale-Free Network Models due to Information Filtering Stefano Mossa,1,2 Marc Barthélémy,3 H. Eugene Stanley,1 and Luís A. Nunes Amaral1 1 Center for Polymer Studies and Department of Physics, Boston University, Boston, Massachusetts Dipartimento di Fisica, INFM UdR, and INFM Center for Statistical Mechanics and Complexity, Universitàdi Roma La Sapienza, Piazzale Aldo Moro 2, I-00185, Roma, Italy 3 CEA-Service de Physique de la Matière Condensée, BP 12, Bruyères-le-Châtel, France (Received 18 October 2001; published 14 March 2002) We formulate a general model for the growth of scale-free networks under filtering information conditions that is, when the nodes can process information about only a subset of the existing nodes in the network. We find that the distribution of the number of incoming links to a node follows a universal scaling form, i.e., that it decays as a power law with an exponential truncation controlled not only by the system size but also by a feature not previously considered, the subset of the network accessible to the node. We test our model with empirical data for the World Wide Web and find agreement. DOI: /PhysRevLett PACS numbers: Hh, i, Da, Hc There is a great deal of current interest in understanding the structure and growth mechanisms of global networks [1 3], such as the World Wide Web (WWW) [4,5] and the Internet [6]. Network structure is critical in many contexts such as Internet attacks [2], spread of an virus [7], or dynamics of human epidemics [8]. In all these problems, the nodes with the largest number of links play an important role on the dynamics of the system. It is therefore important to know the global structure of the network as well as its precise distribution of the number of links. Recent empirical studies report that both the Internet and the WWW have scale-free properties; that is, the number of incoming links and the number of outgoing links at a given node have distributions that decay with power law tails [4 6]. It has been proposed [9] that the scale-free structure of the Internet and the WWW may be explained by a mechanism referred to as preferential attachment [10] in which new nodes link to existing nodes with a probability proportional to the number of existing links to these nodes. Here we focus on the stochastic character of the preferential attachment mechanism, which we understand in the following way: New nodes want to connect to the existing nodes with the largest number of links i.e., with the largest degree because of the advantages offered by being linked to a well-connected node. For a large network it is not plausible that a new node will know the degrees of all existing nodes, so a new node must make a decision on which node to connect with based on what information it has about the state of the network. The preferential attachment mechanism then comes into play as nodes with a larger degree are more likely to become known.

10 C(w) C P P(w)

11 C(w) P Research group CPP P(w,f) FCS Whole world, relevant field(s)

12 Book chapters Conf. Proceed. Reports Journal articles Books within CI. and also field-specific!

13 Correlation between impact and ranking Worldwide top-universities in life/biomedical sciences CPP r

14 P C CPP/FCSm USA 994, ,747, JAPAN 278, , GREAT BRITAIN 270, , GERMANY 251, , FRANCE 180, , PEOPLES R CHINA 162, , ITALY 132, , CANADA 131, , SPAIN 94, , RUSSIA 91, , AUSTRALIA 83, , NETHERLANDS 75, , SOUTH KOREA 70, , INDIA 68, , SWEDEN 59, , SWITZERLAND 54, , BRAZIL 46, , TAIWAN 45, , POLAND 42, , BELGIUM 40, , ,681,790 9,850,029

15 Expert Survey Problems: (methodological) 1. Biases: geographical, field-specific 2. Responding > Non-responding characteristics 3. Sample size > reliability of measurement 4. Nomination procedure 5. Scaling procedure 6. Controlling variables 7. Standard deviation scores 8. Statistical significance

16 Correlation between Expert Scores with Citation-Analysis Based Scores THE Ranking E 10 y = x R 2 = C

17 so we have some problems here..

18 Bibliometric Analysis Problems: 1. Technical 2. Methodological

19 1. Technical problems: - citing-cited mismatches - definition & unification of institutions (specific responsibility)

20 2. Methodological Problems: - Field definition - Field-normalization of citation counts - Black box indicators - Highly cited scientists > highly cited article - Article-type normalization of citation counts - US bias - Language bias (Germany: 25%!) - Engineering, Social Sciences, Humanities - Same data, same methodology, different rankings

21 New Approaches: - Iteration of Expert Survey focused on top - Output-specific analysis engineering fields, social science and humanities - Top-10% bibliometric analysis

22 Top-10% Approach: 1. Identify universities with P > 200/y (N ~ 250) 2. Collect all publication so these universities 3. Ranking by: - entire oeuvre - top-10% of the oeuvre in both cases: CPP and CPP/FCSm - CPP/FCSm(top) x P(top)

23 Outlook: - Improved ranking procedures will further de-equalize universities and reinforce a scientific elite league - Excessive evaluation hypes will les to science destruction - Balance has to be found by data-system improvement and automation of advanced bibliometric assessment procedures

24 Characteristics of a successful university in a bibliometric approach

25 FIELD (CPP/FCSm) Leiden University ASTRON & ASTROPH (1.38) BIOCH & MOL BIOL (0.96) ONCOLOGY (1.05) IMMUNOLOGY (1.22) HEMATOLOGY (1.27) GENETICS & HERED (1.48) PHARMACOL & PHAR (1.11) PHYSICS,MULTIDIS (1.84) PHYSICS, COND MA (1.21) ENDOCRIN & METAB (0.99) MEDICINE,GENERAL (3.35) RAD,NUCL MED IM (1.04) CHEM, PHYSICAL (1.00) CARD & CARD SYST (0.95) RHEUMATOLOGY (1.75) CLIN NEUROLOGY (1.72) NEUROSCIENCES (0.86) CHEM, INORG&NUC (1.82) PHYSICS, AT,M,C (0.87) PERIPHL VASC DIS (1.10) CELL BIOLOGY (1.05) MULTIDISCIPL SC (1.31) CHEM, ORGANIC (1.02) PLANT SCIENCES (1.04) PATHOLOGY (1.56) SURGERY (1.34) CHEMISTRY (1.60) COMPU SCI,THEORY (1.05) PEDIATRICS (1.56) >50% above, and no field below internat. average Share of the output (%) IMPACT: LOW AVERAGE HIGH

26 Research output from Shanghai (PRC) 50,000 45,000 40,000 35,000 30,000 25,000 P C+sc 20,000 15,000 10,000 5,

27 Normalized impact scores CPP/JCSm CPP/FCSm JCSm/FCSm

28 Percentages 'Publications not cited' and 'self-citations' 90% 80% 70% 60% 50% 40% 30% % P not cited % Self-citations 20% 10% 0%

29 Field normalized impact scores for scientific cooperation types Single address National International

30 Growing (inter)national scientific cooperation 0% 20% 40% 60% 80% 100% Single address National International

31 Thank you for you attention and thanks to the Institute of Higher Education, Shanghai Jiao Tong University for organizing a first conference on this very hot topic of ranking