Rankings and the Humanities Making Visible the Invisible? Gero Federkeil, CHE Centre for Higher Education www.che.de Evaluation of Research in Social Sciences and Humanities, Venice, 16/17 March 2012
Introduction Rankings have become popular: National rankings in more than 40 countries Emergence of more than 10 global rankings within one decade The duality of rankings The growing importance of rankings is a consequence of growing competition in higher education; at the same time rankings reinforce this competition by their own results Rankings have an impact on individual decision making (students, academic staff?) on institutional strategies (becoming a WCU) on funding on policy making (excellence programmes)
Rankings and Humanities/Social Sciences First international rankings were institutional rankings: comparison of whole universities across fields Results reflect unweighted averages of fields: Big fields contribute more to result No consideration of level effects Recently introduction of field-based rankings (e.g. QS, ARWU) or broad groups of fields (ARWU, THE) Still: Institutional rankings are far more prominent
Presentation THESIS I The contribution of humanities & social sciences to the research performance of their universities is heavily underrepresented in institutional rankings. THESIS II The existing international field based rankings (ARWU, QS) in the humanities and the social sciences are biased in terms of language and region. Discussion Options for Rankings in the Humanities and Social Sciences- The U-Multirank approach to multidimensional ranking
THESIS I: First evidence The European Top Universities in humanities are not ranked among the top overall universities. Positions in QS World Rankings History Pos. Institutional Pos. FU Berlin 30 ETH Zürich 18 U Leiden 34 Copenhagen 52 Paris I 39 Heidelberg 53 Trinity College Dublin 39 TU München 54 KU Leuven 50 Glasgow 59
Indicators in Global Rankings Shanghai Jiaotong Ranking QS World Rankings Indicator Weight Indicator Weight SCI publications 20 % Reputation among scholars 40 % Publications Science & Nature 20 % Reputation among employers 10 % Highly cited authors 20 % Citations 20 % Nobel Prizes & Field Medals 20 % Student-staff-ratio 20 % Alumni with NobelPrizes 10 % International students 10 % Size 10 % International staff 10 %
Indicators I: Bibliometric Indicators Bibliometric indicators actually do not measure research in humanities /social sciences adequately Data bases cover journal articles mainly Different citation cultures in different fields Language bias in bibliometric data bases Field-normalized citation rates are an important advance to control for different citation cultures but cannot deal with the problems of non-/under-coverage of books/conference proceedings Positive developments: Book Citation Index
Indicators II: Nobel Prizes Not available at all in humanities Social sciences: only economics Question: Are there (equivalently) relevant prizes in the humanities / social sciences? Is there an agreement about their relevance? IREG initiatives to develop a broad list of high reputation awards
THESIS II The existing international field based rankings (ARWU, QS) in the humanities and the social sciences are biased in terms of language and region.
QS World Ranking Humanities by Regions Modern Languages, 1-20 Europe; 1 Asia; 1 Philosophy, 1-20 Europe; 0 Asia; 0 History, 1-20 Europe; 0 Asia; 1 English; 5 UK; 2 US; 11 English; 6 UK; 3 US; 11 English; 5 UK; 3 US; 11 Modern Languages, 1-50 Europe; 6 Asia; 6 US; 20 English; 10 Philosophy, 1-50 Asia; 4 Europe; 5 US; 20 English; 11 History, 1-50 Europe; 6 Asia; 1 English; 9 US; 25 UK; 8 UK; 10 UK; 9 Top 50 positions: US 42,3 % UK 18.0 % English Speaking 20.0 % Europe 11,3 % Asia 7,3 % 81.3% 18.7 %
QS World Ranking Social Sciences by Regions Sociology, 1-20 Pol. Science, 1-20 Economics, 1-20 Europe; 0 Asia; 1 Europe; 0 Asia; 1 Europe; 0 Asia; 1 English; 5 English; 4 English; 3 UK; 3 US; 11 UK; 3 US; 12 UK; 4 US; 12 Sociology, 1-50 Asia; 4 other; 1 Europe; 7 US; 22 English; 9 UK; 7 Pol. Science, 1-50 Asia; 4 other; 0 Europe; 4 US; 21 English; 9 UK; 10 Economics, 1-50 other; 0 Europe; 6 Asia; 6 US; 21 English; 10 UK; 7 Top 50 positions: US 42.7 % UK 16.0 % English Speaking 18,7 % Europe 11,3 % Asia 9,3 % Other 0,7 % 78.0 % 22.0 %
This bias is more pronounced in the humanities /social sciences than in natural sciences! History, 1-50 Europe; 6 Asia; 1 English; 9 UK; 9 US; 25 Physics, 1-50 Asia; 6 US; 18 Europe; 9 English; 6 UK; 11
Deficits of existing rankings: Institutional rankings are biased aganist humanities Other methods of aggregating field-results into institutional results? Up to now international rankings rankings cannot adequately measure the reserach performance of the humanities / social sciences adequately International field-based rankings are biased in terms of language and culture National rankings may use national field-specific bibliometrical data bases
The project Commissioned by the European Commission 2-year project, 2009 June 2011 To develop the concept and test the feasibility of a multidimensional global university ranking Report now available: http://ec.europa.eu/education/higher-education/doc/multirank_en.pdf Ján Figel, the former European Commissioner for Education, Training, Culture and Youth: - to allow stakeholders to make informed choices; - to help institutions to position themselves and improve their performance Follow-up project expected (2012-2013) o Call for Tender probably next week o Implementation of a number of pilot rankings o Business Model for implementation of system
Specification of U-Multirank Five dimensions: Teaching & learning Research Knowledge transfer International orientation Regional engagement Multi-dimensional ranking: No composite overall score! There is no one objective rankings; different users have different preferences and priorities with regard to relevance of indicators Decision about relevance of indicators is left to user: personalised ranking Groups instead of league table
Testing U-Multirank Two levels: Institution (FIR) Fields (FBR) Global sample of higher education and research institutions: 159 (target: 150), 2/3 Europe, 109 completed institutional questionnaires Two fields: Business studies Engineering (electrical and mechanical)
Basic Logic: Mapping Diversity Diversity of higher education institutions in Europe & the world Identifying comparable institutions that can be compared in one ranking Description of horizontal diversity Types/profiles + Assessment of vertical diversity Performance Complementary instruments of transparency
Mapping and Ranking Mapping: Selection of a comparable set of universities based on institutional profiles Teaching and learning Research involvement Knowledge exchange International orientation Regional engagement Student profile Example: Comprehensive, teaching oriented institution Mainly undergraduate education Low research orientation Low international orientation Regionally embedded (e.g. recruiting) Subset of comparable institutions to be compared in a ranking
Mapping and Ranking Ranking: Multi-dimensional ranking for subset of institutions No No composite indicator! No No number number 1!!
Indicators on Research Publication output Field-normalised citation rate Highly cited publications International co-publications External research grants International research grants Doctorate productivity Research orientation of degree programmes /students assessment)
Conclusions U-Multirank offers a field-based multi-dimensional approach to ranking Mapping of diversity of higher education institutions Research is one dimension amkong five Bibliometric inidcatoers remain prominent We have to face the same problems than other (fieldbased) rankings Humanities have not yet been included in the feasibility study An inclusion of humanities /social sciences will involve stkalholder consultation about indicators
Conclusions Up to now the SSH have not been adequately included in international rankings There are some promising developments to enhance the data situation The humanties have to play an active role in the development of an adequate bibliometric data base If they want to be visible U-Multirank welcomes any effort / input
But even if we solve those problems, there might be limits to rankings You re kidding! You count publications?
Rankings and the Humanities Making Visible the Invisible? Gero Federkeil, CHE Centre for Higher Education www.che.de Evaluation of Research in Social Sciences and Humanities, Venice, 16/17 March 2012