Looking at apples and oranges. Learning from university rankings and other composite indicators

Transcription

1 JRC UNIVERSITIES: HOW TO INCREASE THE CONTRIBUTION OF UNIVERSITIES TO SCIENCE AND INNOVATION 11 APRIL 2013, Berlaymont Building, Rue de la Loi 200, Schuman Room Looking at apples and oranges. Learning from university rankings and other composite indicators Andrea Saltelli Unit Econometrics and Applied Statistics European Commission, Joint Research Centre 12 April

2 University ranking and composite indicators Saisana, M., d Hombres, B., Saltelli, A., Rickety numbers: Volatility of university rankings and policy implications, 2011, Research Policy, Paruolo, P., Saisana, A., Saltelli, A., 2013, Ratings and rankings: Voodoo or Science? Journal Royal Statistical Society A, 176 (2), April

3 Volatility of university ranking, example ARWU (Shanghai) 2008 data Simulated rank range - SJTU 2008 Legend: Frequency lower 15% Frequency between 15 and 30% Frequency between 30 and 50% Frequency greater than 50% Note: Frequencies lower than 4% are not shown Original rank Harvard Univ USA Stanford Univ USA Univ California - Berkeley 97 3 USA Univ Cambridge UK Massachusetts Inst Tech (MIT) USA California Inst Tech USA Columbia Univ USA Princeton Univ USA Univ Chicago USA Univ Oxford UK Yale Univ USA Cornell Univ USA Univ California - Los Angeles USA Univ California - San Diego USA Univ Pennsylvania USA Univ Washington - Seattle USA Univ Wisconsin - Madison USA Univ California - San Francisco USA Tokyo Univ Japan Johns Hopkins Univ USA Invasive sensitivity analysis: shaking the assumptions shows that rankings are volatile

4 Internal coherence of ranking systems Non-invasive sensitivity analysis. Comparing the internal coherence of ARWU versus THES by testing the weights declared by developers with effective importance measures suggests that ARWU is more coherent than THES. 12 April

5 Research Excellence Indicators A cooperation between JRC and RTD 12 April

6 Research Excellence indicators (JRC -RTD) Code and Name Definition HICIT Field-normalized count of the 10% (Highly cited most highly cited publications / GDP publications) PCTPAT (Patent Cooperation Treaty applications) TOPINST (World class universities) ERC (ERC grants received) Patent applications filed under PCT by inventors country of residence (fractional counting) in all IPC classes / GDP Country scores on global top 250 universities (based on the Leiden Ranking) / GDP Value of ERC grants received by country of host organization, equally spread over project duration / GDP Sources Science Metrix (Scopus) OECD CWTS (Web of Science) ERC, DG-RTD CORDIS 6

7 Research Excellence indicator 89.8 CH SE DK NL IL FI UK BE AT DE NO US IS EU27 IE FR KR JPSI EE IT ES PT GR CY HU CZ CN LU MT HR LT SK TR PL LV BG RO BR IN RU Research Excellence 41 country global comparison Composite based on 3 indicators: Highly cited Publications Top universities PCT Patent applications

8 Research Excellence indicator 92.4 CH SE IL FI DK NL UK IS BE AT DE NO EU27 FR CY IE IT ES SI HU EE GR PT CZ BG LU MT PL HR LT TR SK LV RO Research Excellence - ERA countries Composite based on 4 indicators: Highly cited Publications Top universities PCT Patent applications ERC grants received

9 Research Excellence indicator TOPINSTpGDP Top Universities indicator 0 Research Excellence Index versus one of its variables: Top Universities CH NL DK SE UK Countries with no universities in global top 250 US BE NOAT FI EU27 IE FR DE IL BG CZ CY GR ES IT RU BR IN RO LVMT PL TR SKLTLU HR CN KR HUPT EESI JP IS Index_WLD_08 Research Excellence Composite

10 Using ARWU (Shanghai) ranking, EU versus US, Top one hundred in ranking, per capita, US= April

11 Using ARWU (Shanghai) ranking, EU versus US, Top fifty in ranking, per capita, US= April

12 Using ARWU (Shanghai) ranking, EU versus US, Top ten in ranking, per capita, US= April

13 Top 250 universities/gdp and PCT patents/pct Looking at invividual US states and Chinese region a-la-aghion. (Leiden ranking) 12 April

14 University rankings at the regional scale (JRC with DG REGIO) Using of the EUMIDA dataset 25 EU member states Census of all HEIs from 25 EU member states (2,400 HEIs) 2008 Vertesy, D., Annoni, P., Nardo, M., University systems: beyond league tables. Engines of growth growth or ivory towers?, Proceedings of the IREG 2012 conference, Taiwan.

15 University regional scale CODE HEI density SI5 intensity IS5 mean VARIABLE NAME Higher education density (Nr. of HEIs / pop. aged 18-30) ISCED5 student intensity (Nr. of ISCED5 Students / pop. aged 18-26) Regional average of international students share (ISCED5) per HEI SI6 intensity IS6 mean RAC DDA intensity SSR mean Doctoral student (ISCED6) intensity (Nr. of ISCED6 Students / pop. aged 22-30) Regional average of international doctoral student share (ISCED6) per HEI Ratio of HEIs defined as research active Intensity of Doctoral Degrees Awarded (DDA per region)/(pop. age 22-30) * 1000 Regional average of student to staff ratio per HEI

16 University 222 regional scores 25 countries

17 University E Scotland (96) NE Scotland (92) Berkshire, Buckinghamshire & Oxfordshire (100) Capital & City Regions in Top20: Bremen (88) Vienna (85) Gr. London (81) Gr. Manchester (78) Prague (78) Gr. Brussels (75) Stockholm (75)

18 LU UK AT SE FI DE BE NL SK CZ IE ES GR CY HU RO IT BG SI PT EE LT LV MT PL RSU Index University regional scale Country average Country_Average Regional scores Regional_Score

19 University regional scale (*) This is one pillar out of 11 pillars of RCI, which includes variables on employment, short- and long-term unemployment, employment gender gap. RCI-Labour market performance index (*) University System Research Index

20 RCI Labor Market Efficiency Index University regional scale University System Research Performance index Not efficient labour market - stronger research performance Spanish/Greek regions (ES41, ES61; GR11, GR13, GR23) Pearson corr.: 0.559

21 Innovation measures Summary innovation index 2010

22 Intergenerational Mobility in Accessing Tertiary Education; JRC + World Bank Upward mobility over the period countries Micro data for more than 1 million adults, including information on their parents educational background d'hombres, B., and Nguyen-Hoang (2011) International and Time Comparisons of Inequality in Tertiary Education. Background report to the World Bank's global study on Equity of access and success in tertiary education. Wide disparities across Europe

23 How about getting the environment right? 12 April

24 Education GDP GEE GGE/GDP GEE/GGE Year on year change of General Government expenditure on education and GDP at constant prices, of GG expenditure/gdp and of GG expenditure on education / Total GG expenditure (EU27) EU Governments rather than investing in education, have constantly reduced education in the total budget during the crisis (Data from Eurostat). 12 April

25 Unweighted Average Unweighted Average Education Reading domain PISA 2006 PISA 2009 Cycle EU countries OECD countries PISA data: EU versus the rest of OECD; boys and girls inn their 15 s Mathematics domain PISA 2006 PISA 2009 Cycle EU countries OECD countries 12 April

26 Unweighted Average Education Sciences domain PISA 2006 PISA 2009 Cycle EU countries OECD countries 12 April

27 6th dimension of the Rule of Law Index (World Justice Project) 83 survey questions 97 countries (20 EU)

28 7 th dimension of the Rule of Law Index (World Justice Project) 56 survey questions 97 countries (20 EU)

29 Corruption Perceptions Index (Transparency International) 13 sources 176 countries (27 EU) 12 April

30 Singapore United States Denmark Norway United Kingdom Australia Finland Malaysia Sweden Iceland Ireland Canada Germany Estonia Saudi Arabia Japan Latvia Lithuania Switzerland Austria Portugal Netherlands Belgium France Slovenia Cyprus Chile Peru Spain Colombia Slovak Republic Mexico St. Lucia Hungary Poland Luxembourg Samoa Ghana Czech Republic Bulgaria Azerbaijan Turkey Romania Italy Seychelles Bahamas, The Greece Croatia Jamaica China Malta Paraguay Central African Republic World Bank Index Ease of Doing Business + one of its sub-indices: Paying taxes ; blue bars = EU countries Ease of Doing Business Rank Starting a Business Dealing with Construction Permits Getting Electricity Registering Property Getting Credit Protecting Inv estors Paying Taxes Trading Across Borders Enf orcing Contracts Resolv ing Insolv ency Paying taxes Ease of Doing Business Rank Paying Taxes 12 April

31 The case of Sweden 12 April

32 Acknowledgements and thanks Beatrice d Hombres, Patricia Dinis Mota Da Costa, Luca Pappalardo, Fiammetta Rossetti, Michaela Saisana, Daniel Vertesy Unit Econometrics and Applied Statistics, European Commission Joint Research Centre 12 April