The performance of universities: an introduction to global rankings

The performance of universities: an introduction to global rankings

Introduction The June 2014 edition of The Economist newspaper with a special focus on higher education (1) claims the sector is experiencing a revolution that is fuelled by: 1. the rising costs of attending university 2. changing labour market demand due to increased automation 3. disruptive technology such as the MOOC (Massive Open Online Course) Further, if the market for higher education develops in a similar way to the one for printed newspapers in the last couple of decades, universities revenues will be more than halved, employment in the sector will fall by about 30%, more than 700 universities will close and the others will see very significant change. Against a backdrop of potentially far reaching and rapid change, this report s objective is to provide insight into the performance of universities. It will do this by setting out how the performance of universities is assessed through an introduction to some of the leading global university rankings. Rankings explored University rankings went global as recently as 2003 with the launch of the Shanghai Jiao Tong University ranking (the Shanghai Ranking ) which was originally designed to reveal what Chinese universities had to do to close the gap with world-class US research-based institutions. Several other rankings have followed since and they have become of interest and concern to university funders, administrators, academics, students, parents and host communities alike. According to The Sodexo University Lifestyle Survey 2014 (2) of students in the UK, independent online guides including league tables come second only to official university websites as the most important sources used to research universities. Indeed, on the surface, there is something immediately attractive about university league tables: they appear to arrange vast reams of information into a seemingly simple format that is accessible and easy to draw conclusions from; but not all is as it seems. In a 2011 report (3) commissioned by the European University Association in response to the growth in international and national rankings, it was the Association s view that at present it would be difficult to argue that the benefits offered by the information they provide, given the lack of transparency that we have observed, are greater than the unwanted consequences of rankings. In the same year, the academic literature carried similarly unfavourable comment by an academic formerly of the National Institute of Education at Nanyang Technological 1 - The Economist, June 28 - July 4, 2014 2 - Published by Sodexo in association with Times Higher Education 3 - Global University Rankings and their Impact, Andrejs Rauhvargers, European University Association, 2011

University in Singapore, an institution that fell 100 places when Times Higher Education switched from Quacquarelli Symonds (4) ( QS ) to Thomson Reuters as its data collection partner in 2010. In Soh s view: with the spurious precision inherent in rankings as they are currently constructed, there is as much misinformation as there is information if not more (5) and it is well advised not to read academic league tables as if there [they] were football league tables, lest the seeming simplicity and deceiving similarity misguides judgments leading to misplaced importance resulting in misinformed decisions and actions (6). Recognition of the importance and influence of university rankings dates much earlier than 2011. As early as 2006, a Berlin meeting of the International Ranking Expert Group (7) (the IREG ) founded by the UNESCO European Centre for Higher Education in Bucharest and the Institute for Higher Education Policy in Washington DC, established guidelines for university rankings known as the Berlin Principles. Their aim was to support continuous improvement and refinement of the methodologies used to conduct these rankings. Given the heterogeneity of methodologies of rankings, the Berlin Principles address the following: the purpose and goals of rankings the design and weighting of indicators the collection and processing of data the presentation of ranking results In 2010, the IREG announced a rankings audit exercise that delivered its first result in 2013: among the major global rankings, the QS World University Ranking became IREG Approved. While university rankings would appear to quench some people s legitimate thirst for comparative information in an important global sector, others find them barely digestible. Set out below is a simple introduction to four rankings to shed light on their composition: the Shanghai Ranking the Times Higher Education World University Rankings the QS World University Rankings the QS Stars University Ratings 4 - A global provider of specialist higher education and careers information and solutions 5 - Soh Kay Cheng (2011) Mirror, mirror on the wall: a closer look at the top ten in university rankings, European Journal of Higher Education, 1:1, 77-83 6 - Soh Kay Cheng (2011) Don t read university rankings like reading football league tables: taking a close look at the indicators, Higher Education Review, 44(1), 15-29 7 - See www.ireg-observatory.org 1

The Shanghai Ranking (8) The Shanghai Ranking (also known as the Academic Ranking of World Universities) was first published in June 2003 by Shanghai Jiao Tong University and is updated on an annual basis. It uses objective indicators to rank over 1,000 world universities and publishes the top 500 online. Universities are ranked by several indicators of academic or research performance, including alumni and staff winning Nobel Prizes and Fields Medals, highly cited researchers, papers published in the journals Nature and Science, papers indexed in major citation indices, and the per capita academic performance of an institution. For each indicator, the highest scoring institution is assigned a score of 100, and other institutions are calculated as a percentage of the top score. Scores for each indicator are weighted to produce an overall score for each university ranked. Indicators (9) and weights of the Shanghai Index Indicator Weight Quality of Education Alumni of an institution winning Nobel Prizes and 10% Fields Medals Quality of Faculty Staff of an institution winning Nobel Prizes and 20% Fields Medals Highly cited researchers in 21 broad subject 20% categories (10) Research Output Papers published in the journals Nature and 20% Science Papers indexed in Science Citation Index-expanded 20% and Social Science Citation Index Per Capita Performance Per capita academic performance of an institution 10% Total 100% The Times Higher Education World University Rankings (11) (the THEWUR ) In collaboration with Thomson Reuters since 2010, the THEWUR judges research-led universities across their core missions. Its 13 performance indicators are grouped into five areas: teaching: the learning environment (30% of the overall ranking score) research: volume, income and reputation (30%) citations: research influence (30%) 8 - www.shanghairanking.com 9 - See Appendix 1 for the definitions of indicators 10 - These are all maths and science-based other than Economics & Business 11 - http://www.timeshighereducation.co.uk/world-university-rankings/2013-14/world-ranking 2

industry income: innovation (2.5%) international outlook: staff, students and research (7.5%). Universities are excluded from the THEWUR if they do not teach undergraduates, if they teach only a single narrow subject or if their research output amounted to fewer than 1,000 articles between 2007 and 2011 (200 a year). Institutions provide and sign off their institutional data for use in the rankings. Whereas the Shanghai Ranking focuses on universities research outputs, the THEWUR includes a subjective worldwide survey of academics views. The indicators that make up each of the five areas covered by the THEWUR are: Teaching (30% of ranking score) Five indicators are used to provide a sense of the teaching and learning environment of each institution from the student and the academic perspective: 1. an invitation-only academic reputation survey that examines the perceived prestige of institutions in research and teaching (over 10,000 responses in 2013) 2. the staff-to-student ratio as a proxy for teaching quality 3. the ratio of doctoral to bachelor s degrees awarded by each institution in the belief that institutions with a high density of research students are more knowledge-intensive and the presence of an active postgraduate community is a marker of a research-led teaching environment valued by undergraduates and postgraduates alike 4. the number of doctorates awarded scaled against institution size as measured by the number of academic staff employed 5. institutional income to indicate the general status of an institution and give a broad sense of the infrastructure and facilities available Research: volume, income, reputation (30% of ranking score) Three indicators are used: 1. universities reputation for research excellence among their peers, based on the responses to the ranking s annual academic reputation survey 2. university research income 3. research productivity based on the number of papers published in the academic journals indexed by Thomson Reuters per academic Citations: research influence (30% of ranking score) This indicator looks at the role of universities in spreading new knowledge and ideas by capturing the number of times a university s published work is cited by scholars globally. The data is drawn from 12,000 academic journals indexed 3

by Thomson Reuters Web of Science database and includes all indexed journals published since 2007. Industry income: innovation (2.5% of ranking score) The ability to support industry with innovations, inventions and consultancy has become a core mission of many universities. This category looks at how much research income an institution earns from industry, scaled against the number of academic staff it employs. The extent to which businesses are willing to pay for research and a university s ability to attract funding is considered a useful indicator of institutional quality. International outlook: people, research (7.5% of ranking score) This category looks at diversity on campus and to what degree academics collaborate with international colleagues on research projects as signs of how global an institution is in its outlook. The three indicators are: 1. the ratio of international to domestic students 2. the ratio of international to domestic staff 3. the proportion of a university s total research journal publications that have at least one international co-author The five categories of the THEWUR and their indicators are summarised in the table below. Category / indicator Weighting Teaching 30% Academic reputation survey 15% Staff to student ratio 4.5% Ratio of doctoral to bachelor s degrees awarded 2.25% Number of doctorates awarded 6% Institutional income 2.25% Research 30% Academic reputation survey 18% Research income 6% Research productivity 6% Citations 30% Industry income 2.5% International outlook 7.5% Ratio of international to domestic students 2.5% Ratio of international to domestic staff 2.5% Proportion of university s total research journal publications that have at least one 2.5% international co-author 4

The QS World University Rankings (12) (the QSWUR ) First published in 2010, the QSWUR is made up of six categories, two of which accounting for half of the ranking score are based on a survey: academic reputation (40% of ranking score) employer reputation (10%) faculty to student ratio (20%) citations (20%) international faculty (5%) international students (5%) Academic reputation Respondents are asked to comment on their area of subject expertise and the geographic region of their experience in a survey that adapts based on their responses. Survey respondents are asked to identify up to ten domestic institutions they consider best for research in each of the faculty areas selected by them with their own institution, if it would otherwise be included, excluded from the presented list. They are also asked to identify up to 30 international institutions they consider best for research in each of the same faculty areas but drawn only from the geographic region of their experience. Employer reputation To gauge the views of employers on graduate hires, this category focuses on employability using a process similar to the one outlined above in relation to academic reputation. Student faculty ratio Used as a proxy to evaluate teaching quality, this category is taken to shed light on the notion of commitment to teaching and to correlate with the level of teaching quality. Citations This category is based on a per faculty member approach. International faculty / students These categories are based on the proportion of faculty members / students who are international. 12 - http://www.topuniversities.com/university-rankings/world-university-rankings/2013 5

In addition to the main QSWUR, there are also rankings in five broad faculty areas: Arts & Humanities Engineering & Technology Life Sciences & Medicine Natural Sciences Social Sciences & Management. QS Stars University Ratings (13) ( QS Stars ) Given the discussion on which attributes form a world-class university, the growing popularity of university rankings and interest in the criteria used, some have gone further / broader than others. As an example, the QS Stars awards universities a rating of one to five stars (14) depending on their performance in a broader range of criteria than those used by the rankings above, namely: Core criteria* Learning environment criteria** Advanced criteria*** Specialist criteria* Research Facilities Culture Discipline ranking Teaching Distance / online learning Innovation Accreditation Employability - Engagement - Internationalisation - Access - * Mandatory ranking criteria ** One criterion chosen by participating institutions *** Two criteria chosen According to QS Stars, these criteria were selected as the key pillars of what makes a world class university, taking into account a number of factors that are often overlooked in university rankings and other assessments. Over 50 different criteria indicators contribute towards the overall assessment. They are grouped into twelve categories from which each institution is evaluated in a total of nine (15). 13 - http://www.topuniversities.com/qs-stars/home 14 - See Appendix 2 for an explanation of the star ratings 15 - See Appendix 3 for the full list of indicators and descriptions 6

Conclusion University rankings have developed significantly since China s decision to evaluate world-class research-led universities in the late 1990s and they have made their presence felt. The decision resulted in the first publication in 2003 of the Shanghai Ranking which is science dominated, research output-driven, and based on a relatively small number of objective criteria. However, just over a decade later, even a brief introduction to other rankings reveals important differences between them. While there is continued reliance on objective Shanghai Ranking-style research-led outputs, heavy reliance is now also placed on subjective surveys and criteria, including elements of the university experience. This is reflected in a simple high level summary presentation of the rankings considered above. High level presentation of rankings Categories No. of indicators in category Category weight (%) Shanghai Ranking* Quality of education 1 10 Quality of faculty 2 40 Research output 2 40 Research per capita 1 10 THEWUR Teaching 5 30 Research 4 30 Citations 1 30 Industry income 1 7.5 International outlook 2 2.5 QSWUR Academic reputation 2 40 Employer reputation 2 10 Faculty to student ratio 1 20 Citations 1 20 International faculty 1 5 International students 1 5 QS Stars Research 2 or 3 12.5 Teaching 2 12.5 Employability 3 12.5 Internationalisation 7 12.5 Facilities 6 8.3 Distance / online learning 6 8.3 Culture 3 4.2 Innovation 3 4.2 Engagement 4 4.2 Access 4 4.2 Discipline ranking 2 12.5 Accreditation 1 4.2 *No subjective survey element included 7

From this report it is clear that the performance of universities is assessed in a variety of ways that include: objective criteria (e.g. number of Nobel Prize-winning faculty members) subjective methods (e.g. reputation surveys) purely academic criteria (e.g. PhDs in faculty) criteria that tend towards the university experience (e.g. student accommodation) or universities interaction with others (e.g. low-income outreach) As a result, the teaching category in THEWUR is not the same as the teaching category in QS Stars and it would be wrong to treat them as directly comparable (see above in relation to the THEWUR and Appendix 3 below in relation to QS Stars). However, despite the variety of indicators, methodologies and weightings behind the categories, there are key themes, in particular: teaching research output internationalisation Teaching and research output come as no surprise but the place of internationalisation begs questions as to cause and effect. For example, does a university s teaching and research output performance drive internationalisation? Or, is internationalisation, based on other factors such as Quality of Life, a key driver of teaching and research output performance as it entails access to a broader talent pool? Considering the higher education revolution referred to in the introduction to this report, and the consequent pressures on universities, how improvements in Quality of Life contribute to their performance in particular these three themes - should be the subject of further research. 8

Appendix 1 Shanghai Ranking definitions of indicators Indicator Alumni Award HiCi N&S PUB PCP Definition The total number of the alumni of an institution winning Nobel Prizes and Fields Medals. Alumni are defined as those who obtain bachelor, Master s or doctoral degrees from the institution. Different weights are set according to the periods of obtaining degrees. The weight is 100% for alumni obtaining degrees in 2001-2010, 90% for alumni obtaining degrees in 1991-2000, 80% for alumni obtaining degrees in 1981-1990, and so on, and finally 10% for alumni obtaining degrees in 1911-1920. If a person obtains more than one degrees from an institution, the institution is considered once only. The total number of the staff of an institution winning Nobel Prizes in Physics, Chemistry, Medicine and Economics and Fields Medal in Mathematics. Staff is defined as those who work at an institution at the time of winning the prize. Different weights are set according to the periods of winning the prizes. The weight is 100% for winners after 2011, 90% for winners in 2001-2010, 80% for winners in 1991-2000, 70% for winners in 1981-1990, and so on, and finally 10% for winners in 1921-1930. If a winner is affiliated with more than one institution, each institution is assigned the reciprocal of the number of institutions. For Nobel Prizes, if a prize is shared by more than one person, weights are set for winners according to their proportion of the prize. The number of Highly Cited Researchers in 21 subject categories. These individuals are the most cited within each category. If a Highly Cited Researcher has two or more affiliations, he/she was asked to estimate his/ her weights (or number of weeks) for each affiliation. More than 2/3 of those multi-affiliated Highly Cited Researchers provided such estimations and their affiliations receive the weights accordingly. For those who did not answer, their first affiliation is given a weight of 84% (average weight of the first affiliations for those who replied) and the rest affiliations share the remaining 16% equally. The number of papers published in the journals Nature and Science between 2008 and 2012. To distinguish the order of author affiliation, a weight of 100% is assigned for corresponding author affiliation, 50% for first author affiliation (second author affiliation if the first author affiliation is the same as corresponding author affiliation), 25% for the next author affiliation, and 10% for other author affiliations. Only publications of Article and Proceedings Paper types are considered. Total number of papers indexed in Science Citation Index-Expanded and Social Science Citation Index in 2012. Only publications of Article and Proceedings Paper types are considered. When calculating the total number of papers of an institution, a special weight of two was introduced for papers indexed in Social Science Citation Index. The weighted scores of the above five indicators divided by the number of full-time equivalent academic staff. If the number of academic staff for institutions of a country cannot be obtained, the weighted scores of the above five indicators is used. For ARWU 2013, the numbers of full-time equivalent academic staff are obtained for institutions in the USA, UK, France, Canada, Japan, Italy, China, Australia, Netherlands, Sweden, Switzerland, Belgium, South Korea, Czech, Slovenia, New Zealand etc. 9

Appendix 2 QS Stars University Ratings What the star ratings mean: One star universities A typical one star university is considered to have established all the key components required to provide a quality service to its students and, in many cases, the foundations upon which to build a stronger domestic reputation. A one star institution will often have been established within the last twenty years and will be putting in place the leadership and ambition to develop quickly. Two star universities A typical two star university is active in research and has an established domestic reputation. The institution is a key part of its local community and will often have begun to consider international opportunities. Three star universities A typical three star university is nationally well recognized, and may have also begun to attract international recognition. This institution maintains a reputable level of research and its graduates are attractive to employers. Four star universities A typical four star university is highly international, demonstrating excellence in both research and teaching. The institution provides an excellent environment for students and faculty. Five star universities A typical five stars university is generally world class in a broad range of areas, enjoys a high reputation and has cutting edge facilities and internationally renowned research and teaching faculty. Five star plus universities A typical five star plus institution is not just world-class, but an elite destination to which the very best students and faculty worldwide will aspire. Its brand name will transform the résumé of anyone connected with it. Five stars plus can apply equally to the world s foremost comprehensive and specialist institutions. 10

Appendix 3 QS Stars University Ratings criteria indicators and descriptions Research Indicator Academic reputation Citations per paper Papers per faculty Or Arts-related output Prolific academic experts Number of academic referees endorsing institution in the QS Global Academic Survey Citations per paper adjusted for institutions with negligible activity in medicine, science Research papers (Scopus) per faculty member Recognition of academic outputs in disciplines where journal articles are not pervasive Number of faculty members achieving international recognition through awards (e.g. Nobel Prize) Teaching Indicator Faculty student ratio Overall student satisfaction Or Completion Satisfaction with teaching Or Faculty with PhD Further study The ratio of faculty to students Proportion of students expressing satisfaction with overall experience in student survey Proportion of students scheduled to graduate succeeding in doing so Proportion of students expressing satisfaction with teaching through student survey Proportion of faculty with PhD or equivalent terminal degree Proportion of students pursuing further study within 12 months of graduation Employability Indicator Employer reputation Graduate employment rate Career service support Number of employer referees endorsing institution in QS Global Employer Survey Proportion of students employed within 12 months of graduation (excluding those not actively seeking work e.g. pursuing further study) Number of full-time careers advisors 11

Internationalisation Institutional research collaborations International faculty International students Inbound exchange students Outbound exchange students Religious facilities International diversity Facilities Sporting facilities Student accommodation IT infrastructure Library facilities Medical facilities Student societies University research collaborations or joint degree programs with QS top 500 universities in previous three years published rankings Proportion of international faculty Proportion of international students Proportion of inbound exchange students Proportion of outbound exchange students Provision of appropriate facilities for international students of different religions Number of nationalities represented in student body Provision of key, popular sporting facilities (e.g. swimming pool, gym, track, sports courts etc ) Number of student rooms relative to size of student body Number of computers on campus; proportion of wired student rooms or campus Wi-fi coverage Value or number of acquisitions with the past 12 months Provision of medical facilities appropriate to the size of the student body Number of student administered organisations Online / distance learning Latest technology Track record Student faculty engagement Student interaction Commitment to online Use of new technology for program deployment Length of time the institution had been operating successful online programs Regular access to faculty for one on one support Opportunities to meet and interact with fellow students Number/proportion of programs made available online Culture Concerts and exhibitions Credits and cultural awards Cultural investment Concerts and exhibitions featuring the work or performances of students and faculty Credits or cultural awards for students, faculty members or the institution as a whole publically accessible work Financial contribution to arts projects outside the university OR investment in cultural preservation 12

Innovation Patents Spin-off companies Industrial research Active patents registered with national or international patent offices Spin-off companies established in the last five years still operating and no longer requiring support from the university Joint research projects with distinct research corporations (non-university), yielding publications in Scopus in the last 5 years Engagement Community investment and development Charity work and disaster relief Regional human capital development Environmental impact Financial or in-kind contributions to community projects within 200km of any campus or affiliated facility Financial or in-kind contributions to national or international causes and charities Proportion of graduates employed in the region OR proportion of students from the region Performance against a checklist of environmental indicators Access Scholarships and bursaries Disabled access Gender balance Low-income outreach Financial value of, or number of students granted access through, scholarships Proportion of campus with wheelchair access Recognition of gender parity Proportion of students identified as low-income OR investment in promotion to low-income families Discipline ranking Faculty area ranking Specific subject ranking Faculty area ranking Ranking performance in specific disciplines in rankings conducted by QS or respected alternate publisher Accreditation Internationally recognized accreditations Or Nationally recognized accreditations Programs accredited by internationally recognized and applicable standards Programs accredited by internationally recognized and applicable standards 13

qualityoflifeinstitute@sodexo.com