Benchmarking Tertiary Education Systems for Improved Performance

Transcription

1 Introduction Benchmarking Tertiary Education Systems for Improved Performance Education is one of the most powerful instruments for reducing poverty and for laying the basis for sustained economic growth. Tertiary education, in particular, is a key factor in a nation s efforts to develop a highly skilled and competitive workforce. It can contribute to reduced inequality and increased productivity, through higher private and social returns, which include increased earnings and savings, a well-informed and engaged citizenry, and highly educated business, science and technology, and political leaders who can influence and expand opportunities for development. Tertiary education systems vary widely in performance, however, often despite similar funding levels and common country characteristics, demonstrating that certain tertiary education systems consistently out-perform others in many critical areas. Brazil, for example, spends.8% of its GDP on tertiary education, achieving an overall enrollment level of only 24%, lower that the Latin American average. Chile, on the other hand, has a 38% enrollment rate while public resources for tertiary education represent only.3% of GDP. For policymakers and stakeholders alike, such differences raise questions as to why and what can be done to understand and improve their systems. From these questions has emerged a proliferation of rankings national and international of universities. And, while these rankings allow for comparison of individual institutions across countries have proliferated, there have been very few attempts to benchmark entire tertiary education systems. There is strong demand for a reliable benchmarking tool, with comprehensive and consistent methodological components, and a more comprehensive scope than those in the rankings marketplace today. A growing number of World Bank client countries are expressing a desire to benchmark their tertiary education systems against international norms, though, to date, the mechanisms for comparison lack the depth of analysis or consistency of methodology to instill sufficient confidence. Benchmarks provide a baseline from which the performance and effectiveness of new programs can be evaluated. They provide a sensitive measure of performance degradation and improvement and can be used for funding and other resource allocation decisions. A benchmarking system for tertiary education needs to incorporate a mix of input, process, output, and outcome indicators. Such an approach would present several advantages: (a) contributing to consensus-building on the priorities for improving tertiary education systems and institutions; (b) establishing priorities for reforms; (c) providing policy makers with a tool to compare their tertiary education system internationally, or to compare state or regional sub-systems within large federal systems; and (d) allowing stakeholders to play a more active role in improving institutional and system-wide outcomes. To build such a benchmarking tool requires a background understanding of the myriad elements embedded in any benchmarking effort, including defining terms, explaining the numerous

2 approaches to benchmarking higher education, and building upon the benchmarking discussion with an expanded analysis of tertiary education rankings. Ultimately, this foundation provides the basis for thoughtful decision-making about the direction and design of a tertiary education systems benchmarking tool. What is Benchmarking of Tertiary Education? Establishing a clear definition of what is meant by the generic term benchmarking is a task fraught with the ambiguities that come with nearly three decades of appropriation of the term by divergent groups with divergent goals. In addition to the challenge of understanding benchmarking broadly, the benchmarking of tertiary education faces numerous, significant obstacles to creating any simple or concise definition of what benchmarking is. The rise of league tables and rankings of tertiary institutions, nationally and internationally, has focused a spotlight on benchmarking as hierarchical competition. Appropriately applied benchmarking, however, need not be zero-sum or a matter of determining who is the best among a set group. Benchmarking can, and likely should, provide nuanced and organic interpretations of a range of activities, which need then to be contextualized and examined with an understanding that the end result should not be a determining a winner but, instead, should be the development and installation of knowledge. With regard to the specific tertiary education environment, benchmarking should exist to help institutions answer questions like: How well are our institutions doing compared to others? Who is doing it better than we are? How? Should we be doing better than we are? If yes, how? If no, how to we justify our current quality and standards? Who is the best? How do we define ourselves against the performance of the best? What, if any, lessons can we learn from the best, to apply those lessons to our organization? What is unique about our department/school/faculty/institution and how do we sustain our existing excellence and mission while seeking constant improvements? In seeking to benchmark tertiary education systems, as opposed to individual institutions or institutional sub-sectors, the task of asking the right questions to provide useful and defensible information becomes even more complex. Approaches/methods Begun as a corporate exercise at Xerox in the early 1980s, to examine the competition and strategize appropriate responses, benchmarking was one of several industry initiatives that tertiary education institutions and, to a lesser extent, systems began to adopt in the face of calls for accountability and transparency in governance. Very simply, the purpose of benchmarking in tertiary education has been to develop best practices, based on the successes at peer institutions, through regular and continued examination. According to Alstete (1995), there are four main types of benchmarking:

3 Internal: where sub-groups within a single organization can conduct simultaneous and comparative examinations of their peer groups across the same campus. Competitive: involves peer institutions and examining the similar operations to show which overall institution is the best within similar markets. Functional/industry: coalitions acting within the general atmosphere of industry norms, to support collaborative learning and improvements Generic (best-in-class): uses the widest sweep of data available to determine best practices and measures all institutions/operations against those practices, regardless of similarities of mission, size, finances, etc. Schofield (1998, 18) adds a fifth type of benchmarking to this list: Implicit benchmarking: where data is produced and collected outside of any specific benchmarking exercise but with the knowledge that its usage in comparative benchmarking is inevitable and where data is used by institutions for transformative change across the institutions and not merely for learning and information. Within these five types of benchmarking, Schofield (1998, 22) articulates five specific benchmarking methodologies: Ideal type (gold) standards: promotes the potential of creating best practices using existing data from across systems/institutions to invent a composite of an ideal system/institution. Activity-based: focuses on a selective set of activities either across institutions or systems or within, and uses the comparisons for either a comparison on those activities alone or as a proxy for institutional comparisons. Vertical benchmarking: looks in depth at the comprehensive operations of a single area Horizontal benchmarking: looks at a single issue or activity across operational areas Externally comparative benchmarking: uses data and judgments made by external sources about systems/institutions/activities to influence internal judgments At the systems level of tertiary education, at least, it is this latter methodology, particularly the use of rankings and league tables in the development of institutional mission statements and strategic planning, which has come to define the benchmarking norm. In the newest and still evolving systems rankings (QS and Lisbon, both described below), the systems comparison emerges from counting the number of universities from one national system, with that country with the highest number (the US) by definition in this methodology, determined to have the best system. Mok and Chan (2008) describe this as the mechanism China and Taiwan have used to direct their policy initiatives toward higher education, though there is no consensus that this process produces a truly representative outcome that merits such influence. Strengths and limitations The strengths of benchmarking tertiary systems come from the capacity to gain insights into current operations as well as instituting procedures that allow for continued monitoring and organic evolution of the data used to inform policy development, strategies, and planning. By creating either internal benchmarks for comparison with outcomes or applying external

4 benchmarks by which to gauge the effectiveness of initiatives at different points in time, benchmarking can provide highly informative data that policymakers and other stakeholders can understand and apply in thoughtful ways. Limitations of benchmarking largely rest in the misuse of either the term benchmarking or the activity of benchmarking. When focused on rankings, policymakers (politicians, campus leaders, or other influential actors) may miss the learning available and instead become too fixated on their relative successes vis-à-vis other countries (and, usually, skewed by their most famous world-class institutions). Benchmarking may also lead to a muddying of contextual nuances within and across tertiary education systems, potentially leading to inaccurate readings or misinterpretation of the information being provided. High quality benchmarking focuses on learning for improvement, while rankings-based benchmarking focuses on static representations of hierarchy. Because of the propensity to equate benchmarking with rankings, particularly in tertiary education, it is imperative to understand how (and how not) to utilize rankings as mechanisms for benchmarking, particularly at the systems level. What is the place of rankings in all of this? Definition Rankings of universities, also called league tables, have flourished over the past twenty-five plus years, since US News and World Report s first ranking of US higher education institutions was published in At its inception, the US News ranking was to serve as a source of information for the consumer of American higher education students and their parents. Inevitably, however, the rankings became a tool of marketing for institutions and a source of quantifiable and understandable data, particularly as institutional accountability became more significant to tertiary education stakeholders including government policymakers, students, and private funding bodies. As the ranking of institutions within a single national context expanded with the publication of Shanghai Jiao Tung University s Academic Ranking of World Universities in 2003 and the Times Higher Education-QS World University Rankings, the power and significance of rankings went global. Media attention began pitting nations against each other based on the rankings. Most recently, QS (Quacquarelli Symonds) expanded its rankings work from its annual collaboration with the UK s Times Higher Education world institution rankings publication to include a ranking of national systems. In addition, the Lisbon Council has also developed its own mechanism for ranking national tertiary systems. These two rankings have different methodologies which shape their outputs and deserve further examination. QS SAFE: National System Strength Rankings QS SAFE is the first attempt to use rankings results, in concert with other indicators, not to evaluate the relative strength of individual institutions, but of countries higher education system

5 strengths as a whole (QS, 2009). The acronym SAFE represents the four equally-weighted indicators QS has selected to represent systems strength: system, access, flagship, and economic. System An evaluation of the overall strength of the system determined by taking the number of institutions ranked 500 or higher, in the given country, divided by the average position of those institutions. Access Calculated based on the number of places at top 500 universities from the subject country (the total number of FTE students at THES/QS top 500 universities in the country) divided by the square root of the population) Flagship Assuming that the performance of a country's leading university is a credit to the system from whence it comes, this is a normalized score based on the global performance of the leading university from the country in question. Economic This indicator recognizes two key factors: the relative fiscal emphasis that the given government places on higher education and the impact or effectiveness of that investment: essentially recognizing performance relative to investment. The indicator takes an indexed score (5 points for a university in the top 100, 4 points for , 3 points for , 2 for and 1 for ) and factors it against the GDP per capita for the country in question. All four outcomes per country are rating equally, then, and the country systems ranked. In 2008, based on this methodology, the ten strongest university systems in the world are: the United States, the United Kingdom, Australia, Germany, Canada, Japan, France, the Netherlands, South Korea, and Sweden. The Lisbon Council: University Systems Ranking Based on the premise that our universities and tertiary education systems exist to education and prepare people to be fully-functioning, well-developed members of our advanced post industrial society which means that we must look at the tertiary education system not simply as a mechanism for churning out a handful of elites and perpetuating social inequality, the Lisbon ranking seeks to evaluate tertiary education systems on a broader spectrum of characteristics than the QS ranking. Lisbon utilizes six indicators to develop its ranking: inclusiveness, access, effectiveness, attractiveness, age-range, and responsiveness. Inclusiveness: The ability of a country s tertiary education system to graduate large numbers of students relative to the size of its population, with the number of graduates a country produces as a percentage of the population theoretically available for advanced study. Access: The ability of a country s tertiary system to accept and help advance students with low levels of scholastic aptitude from secondary schools, comparing countries based on the skill threshold of students entering universities derived from recent OECD data. Effectiveness: The ability of a country s educational system to produce graduates with skills relevant for the country s labor market, calculating the average wage premia a

6 university graduate can expect (adjusting for labor-market characteristics which might affect wage premia independent of university education). Attractiveness: The ability of a country s system to attract a diverse range of foreign students, measured by the percentage of foreign students coming to each country from their 10 top source countries, to determine whether the system is regional or truly international. Age-Range: The ability of a country s tertiary system to function as a lifelong learning institution, as illustrated by the share of year olds enrolled in tertiary education institutions. Responsiveness: The system s ability to adapt as measured by the speed and effectiveness with which countries have adjusted their education system to the criteria laid down in the Bologna Declaration, signed in 1999,(with the United States and Australia exempt from this criteria as non-signatories of Bologna) Acknowledging the limitations to their methodology and the limited pool of countries examined (OECD countries only), the Lisbon Council point out that this process of examining systems is too new and untested to make any definitive statements about system quality. Their goal in developing such a ranking is to expand the point of comparison outside singular institutions to examine how tertiary education benefits society at large. With these limitations acknowledged up-front, the top ten systems in the Lisbon ranking in 2008 are: Australia, the United Kingdom, Denmark, Finland, the United States, Sweden, Ireland, Portugal, Italy, and France. Both of these systems rankings are on the right track regarding those elements that underpin a high quality, effective, and accessible tertiary education system. Whether they each have enough too many, too few indicators, whether the indicators used are the most illustrative ones, and how best to weigh the chosen indicators to represent the ideals of the ranking remain issues deserving repeated evaluation as the systems evolve. As these and other systems-wide rankings emerge, global rankings will do more than pit institutions against each other but should provide guidance and information to policymakers about the components of a comprehensive tertiary system that collectively support the diverse needs of their national and international constituencies. Conclusions and implications for World Bank benchmarking of tertiary education Benchmarking tertiary education effectively requires more than simply allowing others to determine the relative station of individual institutions or national systems of higher education. Institutions benefit more from examining the quality of their operations in light of their own contexts, stakeholders, and chartered expectations. Systems benefit from examining the impact of tertiary systems on their communities socially, economically, culturally. International benchmarking may provide access to information that can lead to improvements and innovations, but rankings should not be assumed to be the best source of that information. Instead, global benchmarking should allow for specific comparators to contexts that are similar and, therefore, illustrative of possibilities.

7 Where rankings provide access to data that can be culled from the whole and used to localized purposes, then their inputs can be very useful. The challenge for stakeholders is not to be complacent about the benefits of rankings and simply accept that the rankings methodologies reflect their own priorities. Benchmarking should specifically focus on a system s own priorities and, therefore, should have methodological foundations grounded in those priorities. Rankings done by third parties, particularly for-profit groups, though interesting intellectually, cannot be presumed to serve these ends. Instead, tools that provide policymakers and stakeholders access to the most comprehensive benchmarking data available would be by far the most useful agents for allowing bespoke comparative analyses that are relevant and appropriate for localized strategic development. The World Bank s efforts regarding benchmarking tertiary education will seek to be exactly this tool. Ideally, the Bank s tool will provide comprehensive, adaptable data, which can be used to create comparisons that inform individual client country needs. Such data will be as diverse as social (equity related to race, gender, religion, for instance), economic (representation of students across income levels, employment levels related to tertiary education, R&D), political (legislation related to tertiary education, public policy initiatives in support of tertiary education), and institutional (diversity of institutions available, enrollments rates, graduation rates, faculty outputs related to teaching and research). This benchmarking initiative will explore the implications of inputs (students, academic staff, administrative leadership) as well as outputs (graduates, research, publications) on quality tertiary education, to present the most comprehensive database available. The next steps, then, are to define which data the Bank s initiative will incorporate and the best sources for such data. Once those indicators are defined and collected, the tool itself will then be constructed for ease and efficiency of use by a myriad of interested parties. With this tool, located within the HDNED anchor website and available to the general public, ideally, researchers would be able to collect and examine whichever data they deem most important, creating weightings based on their own needs and goals. The Bank s contribution, then, will be both the aggregation of the largest available pool of data regarding tertiary education, defined as widely as reasonably possible, and our own utilization of the data to provide informed direction and assistance to our clients within our projects. In both instances and likely in as yet to be imagined ways this benchmarking tool will be a significant and hopefully game-changing contribution to the exploding environment of tertiary education rankings.