Rankings and competitive metrics from a university perspective May 06, 2014 Björn Forslöw, Director s Office Details are hidden on a few slides, two slides with bibliometric maps are excluded, due to licence agreements. Certain data included herein are derived from the Web of Science prepared by THOMSON REUTERS, Inc. (Thomson ), Philadelphia, Pennsylvania, USA: Copyright THOMSON REUTERS 2014 All rights reserved. The network was visualized with Gephi, an open-source graph engine
Today s presentation - content Background Do rankings tell us something (significant)? Examples of tailor made tools for monitoring Collaboration and publication mapping Bibliometric performance comparisons 2 13 maj 2014 2
Organisation of intelligence monitoring and analysis at KI (synoptic) Management Team Board of PhD Edu. Board of Research Board of Education KI/SLL Clinical development Projects University Library Databases Bibliometrics Director s Office Rankings Benchmarking Method. support Risk Managem. Communications Office Media monitoring Media analyses Faculty Office and international Relations International Board Support Research Support office Grants Funding Innovations 13 maj 2014 3
Are metrics and science indicators indispensable? What are the drivers? Globalization and increased competition Transparency on quality and output requested by stakeholders Planning and organizational development Incentives and resource allocation Communication and marketing Increased data availability more applications requested just because it s possible 13 maj 2014 4
Rankings often play a paradoxical role how are they percepted in different target groups? Target groups Students the most frequent readers Deliveries In general - poor education indicators Scientists don t use rankings Research indicators dominate rankings Opinion makers use simple numbers Multi-indicators, field tables Publicerad : 2 oktober 2013 Allra högst rankat av de skandinaviska universiteten är Karolinska Institutet, som dessutom klättrar från plats 42 till 36 i årets rankning.. Men de allra viktigaste jämförelserna för oss är de som analyserar medicinska universitet, universitet med samma uppdrag som vi har säger Anders Hamsten. 13 maj 2014 5
KI is frequently evaluated, a selected monitoring is required International Domestic High priority 5 Medium priority ~16 High priority 8 Medium priority ~15 In-house High priority 11 Medium priority ~7 Total: >60 different evaluation frames 13 maj 2014 6
Karolinska Institutet international ranking positions 2013/2014 Academic Ranking of World Universities, ARWU 2013 Total rank world Total rank Europe Clinical Medicine & Pharmacy world rank Clinical Medicine & Pharmacy Europe rank Agriculture and Life Sciences world rank Agriculture and Life Sciences Europe rank KI position 44 10 11 2 19 3 Times Higher Clinical, Clinical, Education World Preclinical & Preclinical & University Rankings 2013* Total rank world Total rank Europe Health world rank Health Europe rank KI position 36 7 14 6 Total rank world (only Total rank Europe (only Pharmacy & Pharmacy & QS World University multi-faculty multi-faculty Medicine Medicine Pharmacology Pharmacology Rankings 2014 universities) universities) world rank Europe rank world rank Europe rank KI position N.A. N.A. 16 5 7 6 * = Officially named 2013/2014 Whole Universities Subjects or Fields 13 maj 2014 7
Different methodologies different results!? Indicators and Weights for ARWU Criteria Indicator Code Weight Quality of Education Alumni of an institution winning Nobel Prizes and Fields Medals Alumni 10% Quality of Faculty Staff of an institution winning Nobel Prizes and Fields Medals Highly cited researchers in 21 broad subject categories Award 20% HiCi 20% Papers published in Nature and Science* N&S 20% Research Output Papers indexed in Science Citation Index-expanded and Social Science Citation Index PUB 20% QS subject tables - weights Per Capita Performance Per capita academic performance of an institution PCP 10% Total 100% * For institutions specialized in humanities and social sciences such as London School of Economics, N&S is not considered, and the weight of N&S is relocated to other indicators. Bibliometrics Reputation surveys Efficiency Ratios Awards 13 maj 2014 8
Small numbers big changes! Ranking positions are vulnerable Total index score of European top15 Universities ARWU 80 73,6 Total Index Score 60 40 59,7 43,7 42,6 41,7? 32,9 32,1 32 31,8 31,4 31,3 31,2 30 30 29,9 20 0 Univ Cambridge Univ Oxford Imperial Coll London Univ Coll London Swiss Fed Inst Tech - Zurich Univ Utrecht Karolinska Inst Stockholm Univ Paris 06 Univ Edinburgh Univ Munich Tech Univ Munich Univ Manchester Source: Institute of Higher Education, Shanghai Jiao Tong University, Academic Ranking of World Universities (ARWU), 2005 Univ Copenhagen Univ Zurich Uppsala Univ 13 maj 2014 9
Linear ranking order or die tossing within tiers of equals? THE total ranking all categories 13 maj 2014 10
Rankings based entirely on reputation are of little value Regarding the impact of reputation-based rankings, as the reputation-based score dwindles rapidly from the first top-ranked university down to the 50th, the reputation indicators have a very limited impact on the THE World University Ranking. This implies that rankings based entirely on reputation are of little value. The very steep reputation-based curve means that the influence of reputation is substantial for the first few most highly ranked institutions but quickly decreases in significance thereafter. Source: Andrejs Rauhvargers: Global university rankings and their impact, EUA 2013 13 maj 2014 11
The THE Reputation Ranking index distribution is bisectional 13 maj 2014 12
Can we use ranking tables as proof of long term progress? 13 maj 2014 13
Can uncompounded indicators tell us more? 13 maj 2014 14
Each institution needs to find its own set of tools and indicators to monitor and analyse Databases International Research and Education agreements International research collaboration mapping Research publication output benchmarking 13 maj 2014 15
13 maj 2014 16
The Karolinska bibliometric infrastructure Web of Science Verification Karolinska Institutet + SLL University Hospitals SQL database Statistics, Mappings, Evaluations Medline MeSH-terms 13 maj 2014 17
Search KI articles Overview and Refine Link to Web of Science Verifying KI authors in green http://kipublications.ki.se/ 13 maj 2014 18
Karolinska Collaboration Map https://bibliometrics.ki.se/users/login 13 maj 2014 19
Karolinska Collaboration Map xxxxx xxxxx 13 maj 2014 20
Agreement list Singapore 13 maj 2014 21
Copublications KI-A*STAR 2008-2012 Limit: Authors with more than 3 copublications, MeSH headings stronger than 4 13 maj 2014
Copublications in Medicine and Life Science Nordic universities Publication years 2000-2011 Low filter: 50 copublications 13 maj 2014
The broad field of investigation can be defined by MedLine and accessed by Web of Science WoS/Medline overlap Providing data to this study covers 4800 journals and magazines Web of Science 1995 and onwards Provider of basic data to KI publication database covers 12300 journals and Magazines (2012) Medline 1995 and onwards covers 5200 journals and Magazines (2012) November 2013 24
Important definitions and delimitations Investigated institutions Universities are defined as medical centers including associated hospitals Document types Only articles and reviews are considered (peer-reviewed) Counting methods Full count (not fractionalized by the number of co-authors) 13 maj 2014 25
A selection of bibliometric indicators P, No of Publications Cf, field normalized citation score ( average citation impact ) Sum of Cf, Cf of each paper aggregated ( total, size and quality dependent impact ) The normalization of the citation indicator is used to compensate for differences in citation pattern between research areas, document types and publication years Molecular biology, Surgery Original Article, review, letter, note 1995, 2011, 13 maj 2014 26
A bibliometric trend for selected International and Nordic universities 2002-2011 Selected International and Nordic institutions Medicine/LifeScience (MedLine delimited) Y-axis: Average Cf X-axis: Publications Size: SumCf Colour: Share Top5% http://www2.uk-ambetet.se/docs/hogskoleutforskaren/#story=0 13 maj 2014 27
Quality adjusted publication output ( field normalized total citation impact ) 2002-2011 Selected International and Nordic institutions Medicine/LifeScience (MedLine delimited) Univ A Univ B Univ C Univ D Univ E 13 maj 2014 28
Thank You for your Attention!