Challenges in Ranking of Universities First International Conference on World Class Universities, Jaio Tong University, Shanghai, June 16-18, 2005 Anthony F.J. van Raan Center for Science and technology Studies(CWTS) Leiden University
One Basic Question: How can we identify the best universities in the world?
6 Research Questions: 1. Research or Teaching? 2. How to Measure Performance? 3. For all universities in the world? 4. One numerical value? 5. Significance of positions? 6. How many?
Two most influential international rankings: Shanghai Jiao Tong University (60% bibliom.) Times Higher Education Supplement (20% bibliom.)
Important national rankings: Germany: CHE DFG
We can ask experts for their judgment
VOLUME 88, Number 13 PHYSICAL REVIEW LETTERS 1 April 2002 VOLUME 88, Number 13 PHYSICAL REVIEW LETTERS 1 April 2002 VOLUME 88, Number 13 PHYSICAL REVIEW LETTERS 1 April 2002 or let scientific output Truncation and of Power Law Behavior in Scale-Free Network Models its impact speak: Truncation of Power Law Behavior in Scale-Free Network Models due to Information Filtering due to Information Filtering Stefano Mossa,1,2 Marc Barthélémy,3 H. Eugene Stanley,1 Luís A. Nunes Amaral1 Stefano Mossa,1,2 Marc Barthélémy,3 H. Eugene Stanley,1 and Luís A. Nunes Amaral1 1 Center for Polymer Studies Department of Physics, Boston University, Boston, Massachusetts 02215 1 Center for Polymer Studies and Department of Physics, Boston University, Boston, Massachusetts 02215 2 Dipartimento di Fisica, INFM UdR, INFM Center for Statistical Mechanics Complexity, Universitàdi Roma La Sapienza, Piazzale Aldo Moro 2, I-00185, Roma, Italy 2 Dipartimento di Fisica, INFM UdR, and INFM Center for Statistical Mechanics and Complexity, 3 CEA-Service de Physique de la Matière Condensée, BP 12, 91680 Bruyères-le-Châtel, France Universitàdi Roma La Sapienza, Piazzale Aldo Moro 2, I-00185, Roma, Italy (Received 18 October 2001; published 14 March 2002) 3 CEA-Service de Physique de la Matière Condensée, BP 12, 91680 Bruyères-le-Châtel, France (Received 18 October 2001; published 14 March 2002) We formulate a general model for the growth of scale-free networks under filtering information conditions that is, when the nodes can process information about only a subset of the existing nodes in the We formulate a general model for the growth of scale-free networks under filtering information network. We find that the distribution of the number of incoming links to a node follows a universal scaling conditions that is, when the nodes can process information about only a subset of the existing nodes in the form, i.e., that it decays as a power law with an exponential truncation controlled not only by the system size network. We find that the distribution of the number of incoming links to a node follows a universal scaling bibliometric but also by a feature not previously analysis considered, the subset of the network accessible to the node. We test our form, i.e., that it decays as a power law with an exponential truncation controlled not only by the system size model with empirical data for the World Wide Web and find agreement. but also by a feature not previously considered, the subset of the network accessible to the node. We test our model with empirical data for the World Wide Web and find agreement. DOI: 10.1103/PhysRevLett.88.138701 PACS numbers: 89.20.Hh, 84.35.+i, 89.75.Da, 89.75.Hc DOI: 10.1103/PhysRevLett.88.138701 PACS numbers: 89.20.Hh, 84.35.+i, 89.75.Da, 89.75.Hc There is a great deal of current interest in understanding the structure and growth mechanisms of global networks [1 3], such as the World Wide Web (WWW) [4,5] and the Internet [6]. Network structure critical in many contexts such as Internet attacks [2], spread of Email virus [7], or dynamics of human epidemics [8]. In all these problems, the nodes with the largest number of links play important role on the dynamics of the There is a great deal of current interest in understanding the structure and growth mechanisms of global networks [1 3], such as the World Wide system. It is therefore important to know the global structure of the network as well as its precise distribution of the number of links. Web (WWW) [4,5] and the Internet [6]. Network structure is critical in many contexts such as Internet attacks [2], spread of an Email virus [7], or Recent empirical studies report that both the Internet and the WWW have scale-free properties; that is, the number of incoming links and the dynamics of human epidemics [8]. In all these problems, the nodes with the largest number of links play an important role on the dynamics of the number of outgoing links at a given node have distributions that decay with power law tails [4 6]. It has been proposed [9] that the scale-free system. It is therefore important to know the global structure of the network as well as its precise distribution of the number of links. structure of the Internet and the WWW may be explained by a mechanism referred to as preferential attachment [10] in which new nodes link Recent empirical studies report that both the Internet and the WWW have scale-free properties; that is, the number of incoming links and the to existing nodes with a probability proportional to the number of existing links to these nodes. Here we focus on the stochastic character of the number of outgoing links at a given node have distributions that decay with power law tails [4 6]. It has been proposed [9] that the scale-free preferential attachment mechanism, which we understand in the following way: New nodes want to connect to the existing nodes with the largest structure of the Internet and the WWW may be explained by a mechanism referred to as preferential attachment [10] in which new nodes link number of links i.e., with the largest degree because of the advantages offered by being linked to a well-connected node. For a large network it to existing nodes with a probability proportional to the number of existing links to these nodes. Here we focus on the stochastic character of the is not plausible that a new node will know the degrees of all existing nodes, so a new node must make a decision on which node to connect with preferential attachment mechanism, which we understand in the following way: New nodes want to connect to the existing nodes with the largest based on what information it has about the state of the network. The preferential attachment mechanism then comes into play as nodes with a number of links i.e., with the largest degree because of the advantages offered by being linked to a well-connected node. For a large network it larger degree are more likely to become known. is not plausible that a new node will know the degrees of all existing nodes, so a new node must make a decision on which node to connect with VOLUME 88, Number 13 PHYSICAL REVIEW LETTERS 1 April 2002 based on what information it has about the state of the network. The preferential attachment mechanism then comes into play as nodes with a larger degree are more likely to become known. VOLUME 88, Number 13 PHYSICAL REVIEW LETTERS 1 April 2002 Truncation of Power Law Behavior in Scale-Free Network Models due to Information Filtering Truncation of Power Law Behavior in Scale-Free Network Models due to Information Filtering Stefano Mossa,1,2 Marc Barthélémy,3 H. Eugene Stanley,1 and Luís A. Nunes Amaral1 1 Center for Polymer Studies and Department of Physics, Boston University, Boston, Massachusetts 02215 Stefano Mossa,1,2 Marc Barthélémy,3 H. Eugene Stanley,1 and Luís A. Nunes Amaral1 2 Dipartimento di Fisica, INFM UdR, and INFM Center for Statistical Mechanics and Complexity, 1 Center for Polymer Studies and Department of Physics, Boston University, Boston, Massachusetts 02215 Universitàdi Roma La Sapienza, Piazzale Aldo Moro 2, I-00185, Roma, Italy 3 CEA-Service de Physique de la Matière Condensée, BP 12, 91680 Bruyères-le-Châtel, France 2 Dipartimento di Fisica, INFM UdR, and INFM Center for Statistical Mechanics and Complexity, (Received 18 October 2001; published 14 March 2002) Universitàdi Roma La Sapienza, Piazzale Aldo Moro 2, I-00185, Roma, Italy 3 CEA-Service de Physique de la Matière Condensée, BP 12, 91680 Bruyères-le-Châtel, France We formulate a general model for the growth of scale-free networks under filtering information (Received 18 October 2001; published 14 March 2002) conditions that is, when the nodes can process information about only a subset of the existing nodes in the network. We find that the distribution of the number of incoming links to a node follows a universal scaling We formulate a general model for the growth of scale-free networks under filtering information form, i.e., that it decays as a power law with an exponential truncation controlled not only by the system size conditions that is, when the nodes can process information about only a subset of the existing nodes in the but also by a feature not previously considered, the subset of the network accessible to the node. We test our network. We find that the distribution of the number of incoming links to a node follows a universal scaling model with empirical data for the World Wide Web and find agreement. form, i.e., that it decays as a power law with an exponential truncation controlled not only by the system size but also by a feature not previously considered, the subset of the network accessible to the node. We test our DOI: 10.1103/PhysRevLett.88.138701 PACS numbers: 89.20.Hh, 84.35.+i, 89.75.Da, 89.75.Hc model with empirical data for the World Wide Web and find agreement. DOI: 10.1103/PhysRevLett.88.138701 PACS numbers: 89.20.Hh, 84.35.+i, 89.75.Da, 89.75.Hc There is a great deal of current interest in understanding the structure and growth mechanisms of global networks [1 3], such as the World Wide Web (WWW) [4,5] and the Internet [6]. Network structure is critical in many contexts such as Internet attacks [2], spread of an Email virus [7], or dynamics of human epidemics [8]. In all these problems, the nodes with the largest number of links play an important role on the dynamics of the system. It is therefore important to know the global structure of the network as well as its precise distribution of the number of links. Recent empirical studies report that both the Internet and the WWW have scale-free properties; that is, the number of incoming links and the number of outgoing links at a given node have distributions that decay with power law tails [4 6]. It has been proposed [9] that the scale-free structure of the Internet and the WWW may be explained by a mechanism referred to as preferential attachment [10] in which new nodes link to existing nodes with a probability proportional to the number of existing links to these nodes. Here we focus on the stochastic character of the preferential attachment mechanism, which we understand in the following way: New nodes want to connect to the existing nodes with the largest number of links i.e., with the largest degree because of the advantages offered by being linked to a well-connected node. For a large network it is not plausible that a new node will know the degrees of all existing nodes, so a new node must make a decision on which node to connect with based on what information it has about the state of the network. The preferential attachment mechanism then comes into play as nodes with a larger degree are more likely to become known. There is a great deal of current interest in understanding the structure and growth mechanisms of global networks [1 3], such as the World Wide Web (WWW) [4,5] and the Internet [6]. Network structure is critical in many contexts such as Internet attacks [2], spread of an Email virus [7], or dynamics of human epidemics [8]. In all these problems, the nodes with the largest number of links play an important role on the dynamics of the system. It is therefore important to know the global structure of the network as well as its precise distribution of the number of links. Recent empirical studies report that both the Internet and the WWW have scale-free properties; that is, the number of incoming links and the number of outgoing links at a given node have distributions that decay with power law tails [4 6]. It has been proposed [9] that the scale-free structure of the Internet and the WWW may be explained by a mechanism referred to as preferential attachment [10] in which new nodes link to existing nodes with a probability proportional to the number of existing links to these nodes. Here we focus on the stochastic character of the preferential attachment mechanism, which we understand in the following way: New nodes want to connect to the existing nodes with the largest number of links i.e., with the largest degree because of the advantages offered by being linked to a well-connected node. For a large network it is not plausible that a new node will know the degrees of all existing nodes, so a new node must make a decision on which node to connect with based on what information it has about the state of the network. The preferential attachment mechanism then comes into play as nodes with a larger degree are more likely to become known. Truncation of Power Law Behavior in Scale-Free Network Models due to Information Filtering Stefano Mossa,1,2 Marc Barthélémy,3 H. Eugene Stanley,1 and Luís A. Nunes Amaral1 1 Center for Polymer Studies and Department of Physics, Boston University, Boston, Massachusetts 02215 2 Dipartimento di Fisica, INFM UdR, and INFM Center for Statistical Mechanics and Complexity, Universitàdi Roma La Sapienza, Piazzale Aldo Moro 2, I-00185, Roma, Italy 3 CEA-Service de Physique de la Matière Condensée, BP 12, 91680 Bruyères-le-Châtel, France (Received 18 October 2001; published 14 March 2002) We formulate a general model for the growth of scale-free networks under filtering information conditions that is, when the nodes can process information about only a subset of the existing nodes in the network. We find that the distribution of the number of incoming links to a node follows a universal scaling form, i.e., that it decays as a power law with an exponential truncation controlled not only by the system size but also by a feature not previously considered, the subset of the network accessible to the node. We test our model with empirical data for the World Wide Web and find agreement. DOI: 10.1103/PhysRevLett.88.138701 PACS numbers: 89.20.Hh, 84.35.+i, 89.75.Da, 89.75.Hc There is a great deal of current interest in understanding the structure and growth mechanisms of global networks [1 3], such as the World Wide Web (WWW) [4,5] and the Internet [6]. Network structure is critical in many contexts such as Internet attacks [2], spread of an Email virus [7], or dynamics of human epidemics [8]. In all these problems, the nodes with the largest number of links play an important role on the dynamics of the system. It is therefore important to know the global structure of the network as well as its precise distribution of the number of links. Recent empirical studies report that both the Internet and the WWW have scale-free properties; that is, the number of incoming links and the number of outgoing links at a given node have distributions that decay with power law tails [4 6]. It has been proposed [9] that the scale-free structure of the Internet and the WWW may be explained by a mechanism referred to as preferential attachment [10] in which new nodes link to existing nodes with a probability proportional to the number of existing links to these nodes. Here we focus on the stochastic character of the preferential attachment mechanism, which we understand in the following way: New nodes want to connect to the existing nodes with the largest number of links i.e., with the largest degree because of the advantages offered by being linked to a well-connected node. For a large network it is not plausible that a new node will know the degrees of all existing nodes, so a new node must make a decision on which node to connect with based on what information it has about the state of the network. The preferential attachment mechanism then comes into play as nodes with a larger degree are more likely to become known. VOLUME 88, Number 13 PHYSICAL REVIEW LETTERS 1 April 2002 Truncation of Power Law Behavior in Scale-Free Network Models due to Information Filtering Stefano Mossa,1,2 Marc Barthélémy,3 H. Eugene Stanley,1 and Luís A. Nunes Amaral1 1 Center for Polymer Studies and Department of Physics, Boston University, Boston, Massachusetts 02215 2 Dipartimento di Fisica, INFM UdR, and INFM Center for Statistical Mechanics and Complexity, Universitàdi Roma La Sapienza, Piazzale Aldo Moro 2, I-00185, Roma, Italy 3 CEA-Service de Physique de la Matière Condensée, BP 12, 91680 Bruyères-le-Châtel, France (Received 18 October 2001; published 14 March 2002) We formulate a general model for the growth of scale-free networks under filtering information conditions that is, when the nodes can process information about only a subset of the existing nodes in the network. We find that the distribution of the number of incoming links to a node follows a universal scaling form, i.e., that it decays as a power law with an exponential truncation controlled not only by the system size but also by a feature not previously considered, the subset of the network accessible to the node. We test our model with empirical data for the World Wide Web and find agreement. DOI: 10.1103/PhysRevLett.88.138701 PACS numbers: 89.20.Hh, 84.35.+i, 89.75.Da, 89.75.Hc There is a great deal of current interest in understanding the structure and growth mechanisms of global networks [1 3], such as the World Wide Web (WWW) [4,5] and the Internet [6]. Network structure is critical in many contexts such as Internet attacks [2], spread of an Email virus [7], or dynamics of human epidemics [8]. In all these problems, the nodes with the largest number of links play an important role on the dynamics of the system. It is therefore important to know the global structure of the network as well as its precise distribution of the number of links. Recent empirical studies report that both the Internet and the WWW have scale-free properties; that is, the number of incoming links and the number of outgoing links at a given node have distributions that decay with power law tails [4 6]. It has been proposed [9] that the scale-free structure of the Internet and the WWW may be explained by a mechanism referred to as preferential attachment [10] in which new nodes link to existing nodes with a probability proportional to the number of existing links to these nodes. Here we focus on the stochastic character of the preferential attachment mechanism, which we understand in the following way: New nodes want to connect to the existing nodes with the largest number of links i.e., with the largest degree because of the advantages offered by being linked to a well-connected node. For a large network it is not plausible that a new node will know the degrees of all existing nodes, so a new node must make a decision on which node to connect with based on what information it has about the state of the network. The preferential attachment mechanism then comes into play as nodes with a larger degree are more likely to become known.
C(w) C P P(w)
C(w) P Research group CPP P(w,f) FCS Whole world, relevant field(s)
Book chapters Conf. Proceed. Reports Journal articles Books within CI. and also field-specific!
100.00 Correlation between impact and ranking Worldwide top-universities in life/biomedical sciences CPP 10.00 1.00 1 10 100 1000 r
P 2001-2004 C 2001-2004 CPP/FCSm USA 994,650 0.270 3,747,932 0.380 1.38 JAPAN 278,420 0.076 605,876 0.062 0.86 GREAT BRITAIN 270,517 0.073 851,704 0.086 1.22 GERMANY 251,365 0.068 743,582 0.075 1.12 FRANCE 180,145 0.049 490,137 0.050 1.05 PEOPLES R CHINA 162,771 0.044 174,316 0.018 0.57 ITALY 132,091 0.036 320,773 0.033 0.95 CANADA 131,469 0.036 382,198 0.039 1.18 SPAIN 94,005 0.026 197,001 0.020 0.91 RUSSIA 91,749 0.025 83,335 0.008 0.43 AUSTRALIA 83,675 0.023 213,928 0.022 1.07 NETHERLANDS 75,903 0.021 251,170 0.025 1.27 SOUTH KOREA 70,878 0.019 101,548 0.010 0.78 INDIA 68,685 0.019 70,965 0.007 0.46 SWEDEN 59,144 0.016 187,035 0.019 1.15 SWITZERLAND 54,484 0.015 215,893 0.022 1.39 BRAZIL 46,005 0.012 59,758 0.006 0.6 TAIWAN 45,948 0.012 60,273 0.006 0.72 POLAND 42,490 0.012 54,078 0.005 0.6 BELGIUM 40,916 0.011 116,947 0.012 1.15 3,681,790 9,850,029
Expert Survey Problems: (methodological) 1. Biases: geographical, field-specific 2. Responding > Non-responding characteristics 3. Sample size > reliability of measurement 4. Nomination procedure 5. Scaling procedure 6. Controlling variables 7. Standard deviation scores 8. Statistical significance
Correlation between Expert Scores with Citation-Analysis Based Scores THE Ranking 2004 1000 100 E 10 y = 53.985x 0.0397 R 2 = 0.005 1 0.1 1 10 100 1000 C
so we have some problems here..
Bibliometric Analysis Problems: 1. Technical 2. Methodological
1. Technical problems: - citing-cited mismatches - definition & unification of institutions (specific responsibility)
2. Methodological Problems: - Field definition - Field-normalization of citation counts - Black box indicators - Highly cited scientists > highly cited article - Article-type normalization of citation counts - US bias - Language bias (Germany: 25%!) - Engineering, Social Sciences, Humanities - Same data, same methodology, different rankings
New Approaches: - Iteration of Expert Survey focused on top - Output-specific analysis engineering fields, social science and humanities - Top-10% bibliometric analysis
Top-10% Approach: 1. Identify universities with P > 200/y (N ~ 250) 2. Collect all publication so these universities 3. Ranking by: - entire oeuvre - top-10% of the oeuvre in both cases: CPP and CPP/FCSm - CPP/FCSm(top) x P(top)
Outlook: - Improved ranking procedures will further de-equalize universities and reinforce a scientific elite league - Excessive evaluation hypes will les to science destruction - Balance has to be found by data-system improvement and automation of advanced bibliometric assessment procedures
Characteristics of a successful university in a bibliometric approach
FIELD (CPP/FCSm) Leiden University ASTRON & ASTROPH (1.38) BIOCH & MOL BIOL (0.96) ONCOLOGY (1.05) IMMUNOLOGY (1.22) HEMATOLOGY (1.27) GENETICS & HERED (1.48) PHARMACOL & PHAR (1.11) PHYSICS,MULTIDIS (1.84) PHYSICS, COND MA (1.21) ENDOCRIN & METAB (0.99) MEDICINE,GENERAL (3.35) RAD,NUCL MED IM (1.04) CHEM, PHYSICAL (1.00) CARD & CARD SYST (0.95) RHEUMATOLOGY (1.75) CLIN NEUROLOGY (1.72) NEUROSCIENCES (0.86) CHEM, INORG&NUC (1.82) PHYSICS, AT,M,C (0.87) PERIPHL VASC DIS (1.10) CELL BIOLOGY (1.05) MULTIDISCIPL SC (1.31) CHEM, ORGANIC (1.02) PLANT SCIENCES (1.04) PATHOLOGY (1.56) SURGERY (1.34) CHEMISTRY (1.60) COMPU SCI,THEORY (1.05) PEDIATRICS (1.56) >50% above, and no field below internat. average 0 1 2 3 4 5 6 7 Share of the output (%) IMPACT: LOW AVERAGE HIGH
Research output from Shanghai (PRC) 50,000 45,000 40,000 35,000 30,000 25,000 P C+sc 20,000 15,000 10,000 5,000 0 1980-1983 1981-1984 1982-1985 1983-1986 1984-1987 1985-1988 1986-1989 1987-1990 1988-1991 1989-1992 1990-1993 1991-1994 1992-1995 1993-1996 1994-1997 1995-1998 1996-1999 1997-2000 1998-2001 1999-2002 2000-2003 2001-2004
Normalized impact scores 1.20 1.00 0.80 0.60 0.40 0.20 CPP/JCSm CPP/FCSm JCSm/FCSm 0.00 1980-1983 1981-1984 1982-1985 1983-1986 1984-1987 1985-1988 1986-1989 1987-1990 1988-1991 1989-1992 1990-1993 1991-1994 1992-1995 1993-1996 1994-1997 1995-1998 1996-1999 1997-2000 1998-2001 1999-2002 2000-2003 2001-2004
Percentages 'Publications not cited' and 'self-citations' 90% 80% 70% 60% 50% 40% 30% % P not cited % Self-citations 20% 10% 0% 1980-1983 1981-1984 1982-1985 1983-1986 1984-1987 1985-1988 1986-1989 1987-1990 1988-1991 1989-1992 1990-1993 1991-1994 1992-1995 1993-1996 1994-1997 1995-1998 1996-1999 1997-2000 1998-2001 1999-2002 2000-2003 2001-2004
Field normalized impact scores for scientific cooperation types 1.20 1.00 0.80 0.60 0.40 0.20 Single address National International 0.00 1980-1983 1981-1984 1982-1985 1983-1986 1984-1987 1985-1988 1986-1989 1987-1990 1988-1991 1989-1992 1990-1993 1991-1994 1992-1995 1993-1996 1994-1997 1995-1998 1996-1999 1997-2000 1998-2001 1999-2002 2000-2003 2001-2004
Growing (inter)national scientific cooperation 0% 20% 40% 60% 80% 100% 1980-1983 1981-1984 1982-1985 1983-1986 1984-1987 1985-1988 1986-1989 1987-1990 1988-1991 1989-1992 1990-1993 1991-1994 1992-1995 1993-1996 1994-1997 1995-1998 1996-1999 1997-2000 1998-2001 1999-2002 2000-2003 2001-2004 Single address National International
Thank you for you attention and thanks to the Institute of Higher Education, Shanghai Jiao Tong University for organizing a first conference on this very hot topic of ranking