Sustainability in University Rankings 21 October 2013, Venice (Italy) Rankings and Sustainability Stefano PALEARI University of Bergamo, Rector and CRUI, President Results are from the research project: Le università nel nuovo secolo: nuovi modelli per una società in trasformazione Team Higher Education UnBG-CRUI
Our approach Sustainability issues are to be contextualized Why do we use measures (rankings) in Higher Education? Sustainability in Higher Education Measures (Rankings) and Sustainability 2
Why do we need measures for HE? Efficient allocation of limited resources Accountability towards funders and society Universities are (also) financed by the state Transparency of results Main World University Rankings: Academic Ranking of World Universities (ARWU) - 2003 Quacquarelli Symonds (QS) - 2004 Times Higher Education Ranking (THE) 2010 3
ARWU Criteria Indicator Weight Quality of Education Quality of Faculty Alumni of an institution winning Nobel Prizes and Fields Medals Staff of an institution winning Nobel Prizes and Fields Medals Highly cited researchers in 21 broad subject categories 10% 20% 20% Research Output Papers published in Nature and Science* 20% Papers indexed in Science Citation Index-expanded and Social Science Citation Index 20% Per Capita Performance Per capita academic performance of an institution 10% Total 100% 4
QS Criteria Indicator Weight Teaching Faculty Student Ratio 20% Reputation Academic Peer Review from global survey 40% Recruiter Review from global survey 10% Research Citations per Faculty from Sciverse Scopus 20% International Orientation Percentage of intanational students 5% Percentage of intanational staff 5% Total 100% 5
THE Criteria Indicator Weight 2011 Weight 2012 Reputational survey - teaching 15% 15% = PhD awards per academic 6% 6% = Teaching - the learning environment Undergraduates admitted per academic 4,5% 4,5% = Income per academic 2,25% 2,25% = PhD awards/ bachelor's awards 2,25% 2,25% = Citations Research Citations impact (normalised avarage citations per paper) from Scopus 32,5% 30% Research - volume, income and reputation Reputational survey - research 19,5% 18% Research income (scaled) 5,25% 6% Papers per academic and research staff 4,5% 6% International Outlook - staff, students and research Industry Income - innovation Ratio of international to domestic staff 0,75% 2,5% Ratio of international to domestic students 3% 2,5% Proportion of internationally co-authored research papers Research income from industry (per academic staff) 2% 2,5% 2,5% 2,5% = Total 100% 100% 6
Worldwide University Ranking Methodological issues Subjective measures i.e.: Subjectivity of weights and indicators Biased measures Rankings favour hard science and publications in English language Reputation vs. Contemporaneous performance measures Advantage for big universities Advantage for historical universities (Nobel prize winners) Measures affected by exogenous factors Total income of institutions 7
World University Ranking Correlation Spearman coefficient: d i : difference among the position of the i university n: number of universities listed High level of correlation among top-100 ranked institutions (Sample: 137 universities listed in all the three rankings) ARWU 2011 THE 2011 QS 2011 ARWU 2011 1,000 THE 2011 0,684 1,000 QS 2011 0,421 0,614 1,000 2011-2012 Correlation ARWU 0,994 THE 0,957 QS 0,989 Spearman correlation Matrix in year 2011 and among different rankings and among the same ranking in 2011 and 2012 8
Best University according to THE 1/2 Top 10 universities in Times Higher Education Ranking 2012 Rank Università Students Faculty Student / Faculty 1 California Institute of Technology 2.175 425 5 2 University of Oxford 20.466 4.962 4 2 Stanford University 15.666 1.910 8 4 Harvard University 21.229 2.242 9 5 Massachusetts Institute of Technology (MIT) 10.384 1.009 10 6 Princeton University 7.567 1.172 6 7 University of Cambridge 18.396 2.957 6 8 Imperial College London 15.215 3.594 4 9 University of California, Berkeley 36.142 2.393 15 10 University of Chicago 15.438 2.168 7 Top 10 universities in Times Higher Education Ranking 2012 Average students number "top 10" 16.268 Average students per faculty "top 10" 7 Avarage students per faculty in Italy 30 9
Best University according to THE 2/2 Top university by country in THE 2012 Istituzione Studenti Docenti Student to faculty ratio % studenti internazionali % graduate students California Institute of Technology 2.130 425 5,0 25,0% 55,4% University of Oxford 18.703 4.962 3,8 29,2% 35,8% ETH Zürich 14.116 1.150 12,3 33,1% 50,2% University of Toronto 71.100 10.650 6,7 10,3% 18,8% University of Tokyo 27.821 4.719 5,9 7,6% 49,5% University of Melbourne 34.046 3.595 9,5 28,5% 27,3% National University of Singapore 30.244 5.252 5,8 36,3% 21,0% University of Hong Kong 15.863 3.046 5,2 30,0% 41,7% Karolinska Institute 7.051 1.371 5,1 7,1% 51,6% Peking University 38.228 2.900 13,2 5,6% 51,0% Ludwig-Maximilians-Universität München 49.180 3.576 13,8 14,1% 14,0% Pohang University of Science and Technology 3.217 429 7,5 1,6% 48,0% Katholieke Universiteit Leuven 32.756 6.781 4,8 15,5% 47,2% École Normale Supérieure 2.700 800 3,4 11,1% 77,8% Utrecht University 30.344 2.904 10,4 6,3% 12,9% 25.167 3.504 7,5 17,4% 40,1% 10
Europe vs. US Leading position of US universities: 17 out of 20, and 57 of the top 100 in the ARWU 2011/2012 Considering the top 500 universities, Europe with 202 overtakes America with 182 top-ranked institutions In Europe, with a predominant focus on broad-based and regionally distributed support for a fairly large number of universities, resources were more evenly distributed Key issue: What is Excellence? E.g. the University of Göttingen does rank very highly in rankings, but it is not successful in the German Initiative on Excellence 11
Why do current rankings fail in measuring sustainability? Use of a sole indicator to evaluate multi-output organisation They do not consider emerging aspects as sustainability and diversity of institutions They measure only measurable facts Measure only what is measurable, not all what matters Describe the whole only through a part In sports F1: who determines the result? the driver, the car, the team Does one component really represent the whole? 12
Do rankings improve (a fair) university competition? If universities are to challenge, rankings should refer to universities in the same league, By contrast they do not discriminate: Different budget Different recruitment procedures Human capital is the most important productive factor into a university! Different tuition fees Once again in sports NBA: the same salary cap for all the teams (with fines for those not complying) 13
Funding of European HE Many countries reduced public funding to their tertiary education systems from the beginning of the crisis Reduction of funding mainly in the countries in the south and east of Europe Trends in Public Funding to Higher Education Europe over the period 2008-2012 Source EUA s Public Funding Observatory (June 2012) 14
Funding per citizen (2008-2012) Country * Including EU Funds Funding 2012 - (mln ) Population 2011 ('000) Funding per citizen Change 2008-2012 Fonte Elaborazione propria dati EUA s Public Funding Observatory e Word Bank Statistics Change 2008-2012 Inflation-adjusted Norway 3.621 4.953 731 22,0% 21,0% Sweden 6.235 9.449 660 22,0% 21,0% Germany 24.900 81.798 304 23,0% 20,0% France 19.800 65.434 303 8,8% 6,4% Iceland 87 319 273 13,0% 7,2% Ireland 1.236 4.576 270-20,0% -21,0% Austria 2.169 8.424 257 15,0% 13,0% Netherland 3.232 16.693 194 10,0% 7,5% Spain* 7.258 46.175 157-9,5% -11,0% UK 9.815 62.744 156-10,0% -13,0% Italy 6.633 60.724 109-12,0% -14,0% Croatia 369 4.403 84 5,3% 1,8% Slovakia 447 5.398 83 2,1% -1,5% Poland* 3.015 38.534 78 12,0% 8,6% Czech Reupublic 802 10.496 76-14,0% -17,0% Lithuania 189 3.030 62-19,0% -22,0% Portugal 602 10.557 57-1,5% -4,1% Hungury 542 9.972 54-20,0% -24,0% Greece 200 11.300 18-25,0% -25,0% Belgium - French Community 585 n.a. 0 19,0% 16,0% 15
Funding development: Italy Fonte EUA s Public Funding Observatory 16
Funding development: other countries Fonte EUA s Public Funding Observatory 17
Do rankings help universities to work more efficiently? If rankings have to increase efficiency: They need to be an instrument, they should not be the objective! Universities should be empowered the instruments to correct their performances i.e.: more autonomy in recruitment 18
Italy: Consequence of funding drop To do more with less? Sharp declining of academic staff Permanent academic staff went back to the level of 2001 Dramatic declining of non-academic staff Increase in Student to Faculty Ratio State universities 2012 2011 2010 2009 2008 Δ Δ % Bachelor & Master students n.a. 1.625.787 1.665.060 1.684.726 1.699.038-73.251-4,3% Tenure Academic staff 52.456 53.901 55.199 58.307 60.254-7.798-12,9% Full Professors 13.841 14.532 15.169 17.174 18.218-4.377-24,0% Associate Professors 15.435 15.884 16.229 16.858 17.547-2.112-12,0% Tenure Researchers 23.180 23.485 23.801 24.275 24.489-1.309-5,3% Non-tenured researchers 1.770 1.049 732 408 304 1.466 482,2% Non-academic staff 55.810 57.459 58.966 61.873 69.916-14.106-20,2% Studenti to Faculty Ratio n.a 30 30 29 28 Source: Reprocessing data from MIUR e CINECA database 19
Something new: the U-Multiranking Advantages Better transparency of performance evaluation Measure excellence in different missions and levels User-oriented Evaluation not only at institutional level, but also at cource level Drawbacks: Still developing (first release in February 2014) Costly for institution to collect and provide information and data 20
How can we improve rankings? Efficiency oriented Including at least one measure which considers both inputs (staff, funding) and outputs Importance of network What does really matter? To be the first, or to add value to the system? Importance of diversity Promote best practices, rather then the use of synthetic (and biased) rankings 21
How can we improve the HE system as a whole? Set the «ultimate goal» for the HE system, and set rankings accordingly Is the HE system designed to train the top students, or to improve the quality of the system as a whole? Is the HE system designed to mitigate the tendency to inequality in the social system? 22
Conclusions Sustainability is Efficiency The role of the university in the society Competing in the same field (classifying universities) Competing with the same rules Setting long-term goals 23