Selections. Ranking THE BUSINESS OF BUSINESS SCHOOLS. The Magazine of the Graduate Management Admission Council IN THIS ISSUE

Transcription

1 FALL 2001 Selections The Magazine of the Graduate Management Admission Council THE BUSINESS OF Ranking BUSINESS SCHOOLS IN THIS ISSUE A look behind the numbers The people who compile the surveys The impact: gloaters, growlers, and skeptics

2 FALL 2001 Volume 1, Issue 2 Selections The Magazine of the Graduate Management Admission Council F E A T U R E S Page 8 8 A LOOK BEHIND THE NUMBERS by Gail Tyson Illustration by Gene Greif Editors and number crunchers explain how their b-school rankings are determined and what they do to ensure the outcome is fair. Page THE PEOPLE BEHIND THE RANKINGS by Carlotta Mast From the creators of the rankings, an explanation of their genesis, evolution, and the need to quantify the value of a given school s MBA. THE IMPACT OF THE RANKINGS: MULTIPLE PERSPECTIVES by Jeff Wuorio Photography by Eugenie Uhl Though schools may or may not enjoy high ratings and students may welcome a simple answer, some researchers wonder about accuracy and undue emphasis. Page 26 D E P A R T M E N T S To Start 2 What Works 3 Keeping Up 38 The Last Word 39

3 TO Start Making Sense of the Rankings Selections EDITOR IN CHIEF James W. Schmotter, PhD Western Michigan University MANAGING EDITOR Nancy L. Freireich EDITOR Jacqueline M. Olkin EDITORIAL DIRECTOR Daphne Atkinson DESIGN Zehno Cross Media Communications PUBLISHED BY Graduate Management Admission Council 1750 Tysons Boulevard, Suite 1100 McLean, Virginia USA Phone The Graduate Management Admission Council (GMAC), composed of representatives of leading business schools worldwide, is a not-for-profit education association dedicated to creating access to graduate management and professional education. Opinions expressed are those of the authors and editors and do not reflect official positions of GMAC. Whenever groups of business school deans, admissions directors, or potential MBA applicants gather, the topic of business school rankings invariably arises. No one, it seems, is completely satisfied with these judgments by the media. Although some observers do acknowledge that rankings can be useful in motivating schools to improve, others find the results less beneficial. Certainly, alumni, students, and applicants demand to know what s wrong and who s responsible when a school drops a few places. And even those who rise in the rankings worry because they know that next year, it could be their turn to fall. Hence, few topics are more emotional and anxiety producing for the leaders and stakeholders of graduate management education. As the mini-industry of business school rankings continues to expand, with new publications entering the fray, the emotional intensity of our responses only increases, often to the point of obsession. Such emotion often prevents us from comprehending what the rankings purport to accomplish, what their role in our industry is, and how we should respond. We at Selections believe it s time for a dispassionate examination. That s what this issue provides. Since the first Business Week ranking of full-time MBA programs in 1988, media assessments have become an inescapable aspect of the landscape of graduate management education. However much some might hope, expecting such assessments to disappear is wishful thinking. Rankings are here to stay, and it s important for us all to accept and deal with that reality. We hope that the following pages will help in that effort. They include a look at how rankings differ in what they value and measure; interviews with representatives of the major publications that do rankings to learn how they define their purpose and constituents; and an examination of the impact of the rankings on business schools and other audiences. To comment on this issue of Selections or to get interview transcripts, visit We d like to know what you think. James W. Schmotter Editor in Chief James.schmotter@wmich.edu 2 Selections: Fall

4 What WORKS Setting the Standard How U.S. News & World Report and the Financial Times Fine-Tune Their Data The increasing standardization of data spurred in large part by the need for accurate data in business school rankings is changing the way schools think about, handle, and report the data they collect. As part of her article for Selections (see The People behind the Rankings, p. 16), Carlotta Mast interviewed Bob Morse, director of data research at U.S. News & World Report; Gayle Garrett, project director for Best Graduate Schools, also published by U.S. News; Parminder Bahra, senior adviser on education projects for the Financial Times; and Della Bradshaw, editor of the Financial Times business school rankings. They spoke about the standardization and auditing of data and related improvements they are making to their surveys. Here are excerpts from the interviews, followed by an explanation of the standards to which Morse and Garrett refer: GMAC s MBA Reporting Criteria and the MBA Career Services Council s (MBA CSC s) MBA Employment Standards. (For more information on the methodologies of the four major publication rankings, see A Look behind the Numbers, by Gail Tyson, p. 8. For complete transcripts of all the interviews conducted for The People behind the Rankings, visit SELECTIONS: In what particular ways have you seen the business school rankings in general and the U.S. News & World Report rankings in particular impact the schools and management education? [ Selections: BOB MORSE (U.S. News): I would say the thing I am most proud of in the rankings, and this came out at the AACSB [AACSB International the Association to Advance Collegiate Schools of Business] meeting in New York this year I was on a media panel. I think the schools believe even though it makes them madder than the dickens that U.S. News & World Report, and maybe Business Week to some degree, have been responsible for causing data standardization. And if it wasn t for U.S. News given ] Bob Morse, director of data research, U.S. News & World Report I think the schools believe even though it makes them madder than the dickens that U.S. News & World Report, and maybe Business Week to some degree, have been responsible for causing data standardization. Fall

5 What WORKS Gayle Garrett U.S. News & World Report that we are the only ones to use the data in the ranking, the other ones just publish it, right? then there wouldn t have been the standardization of placement data [by MBA CSC], and there wouldn t be the move by GMAC to try to implement auditing standards and put in rigorous definitions and schools signing up for it. I think we have caused that. Or, as a result of the rankings and the need for better consumer data as a result of that the schools have realized they have had to do that [adopt GMAC s MBA Reporting Criteria and MBA CSC s MBA Employment Standards]. And it s also helping the MBA education at least to seem more professional by having better data. SELECTIONS: How should schools be using the information that is published in the rankings? GAYLE GARRETT (U.S. News): What I hope the schools are using the information for is... well, I hope they are being very, very careful about the information they provide us, so that when it comes out... they will be taking ownership of it at the beginning of the process, rather than at the end. [ All the time, we are trying to get clearer and clearer salary data that we publish, and precisely what it means and to which programs it applies and for how many graduates it is applicable. As a matter of fact, I just got through revising our survey for this coming fall, and we have always used, in terms of the admissions data we have always tried to follow the GMAC criteria. There are also standards for reporting salary data, and those are put out by the MBA Career Services Council. I just made the decision in doing our new survey that we would not ask for any percentages to be provided to us by the schools, rather only exact counts. We will compute our own percents. There has been some misinterpretation in providing percents, in terms of what is the denominator.... They are asked to provide mean, median, high, and low, and I want to know the actual numbers that those statistics are computed on. Eventually, we will be redesigning the Web site, and every time you see a list of those statistics, you will see exactly how many students are involved there. That is very important. Right now, the instructions I just made the decision in doing our new survey that we would not ask for any percentages to be provided to us by the schools, rather only exact counts. ] Gayle Garrett, project director, Best Graduate Schools (U.S. News & World Report) 4 Selections: Fall

6 What WORKS Parminder Bahra, Financial Times [ ] There is a lot of talk about schools that may not submit accurate data. The only way to deal with that is to audit it. Della Bradshaw, editor, Financial Times business school rankings say, At least three students, or 1 percent of the graduates. Well, if you have a high, low, and median for three students, that is very different [from having] it for 30 students. And so by having that right alongside, it will immediately say to whoever looks at it, Oh, that is interesting information, but it is not based on a very big group. SELECTIONS: To whom do you find yourself accountable in your work of publishing business school rankings? GARRETT: Obviously, I am accountable to my editor because the editor makes decisions about what gets included in the table. I am accountable to U.S. News to make certain that whatever we publish is accurate. That when data comes to us, we have all kinds of ways of comparing it against data that was given to us the previous year to catch errors. There is a minimum of two separate processes by which we send information back to the schools to be verified. I am constantly trying to make what we do more and more transparent. That is not always easy because we really have a brutal publication schedule. There is not a lot of time between the time the data comes in and the time that it has to be put out for publication.... But every year, we keep getting better at it. There is an information feedback loop with the schools... as deans come in, and particularly as they come in and are willing to work their requests through the appropriate bodies in the discipline, the AACSB [AACSB International] or the [MBA] Career Services Council specifically, or the GMAC people. U.S. News tries very hard and I try very hard to look at those standards and implement them as carefully as we can, so that we re lifting the best standards from the profession standards of accountability. SELECTIONS: Are there any steps you take to audit [information supplied by schools] or make sure that everyone is supplying standardized data? PARMINDER BAHRA (Financial Times): This year, we will be looking into auditing the figures and maybe getting one of the consultancies to check with the business schools and actually audit the information formally. At the moment, the process is that we do some comparisons with previous years, just to make sure that they are in line. If any figures look like they could possibly be out of line, then we will go back to the business school to check to make sure it is correct. That is our formal check at the moment. But there is no formal auditing procedure at the moment. We are looking to introduce it in the next year or so. SELECTIONS: How do you audit the data the schools supply? DELLA BRADSHAW (Financial Times): We are hoping to use an auditing firm to audit the data for our January MBA 2002 rankings. We are still dealing with that because it is very expensive. I don t think the schools like the idea of it at all, to be honest. There are some schools that already audit their data. Queen s [University at Kingston] in Canada always sends audited data, and they were trying to get other Canadian schools to do this. They believe this is a good thing. But nobody else took them up on it. There is a lot of talk about schools that may not submit accurate data. The only way to deal with that is to audit it.// Selections: Fall

7 What WORKS Setting the Standard (continued) Judging B-Schools with One Yardstick GMAC s MBA Reporting Criteria and the MBA Pathfinder School Search Database The Graduate Management Admission Council originally released standards for reporting admissions statistics in 1993; these standards were intended for use by the Council s member business schools and others who used its services. In 1998, in response to a need it saw in the greater marketplace for consistent, comparable, accurate MBA program data, the Graduate Management Admission Council established an Industry Information Task Force. The task force was charged with proposing data definitions and criteria for schools to use in reporting data on MBA admissions, enrollment, and programs. In the spring of 2000, after releasing a draft and considering comments from 85 schools of graduate management, GMAC released the MBA Reporting Criteria, which enable schools to provide prospective students, the media, and other interested parties with consistent, comparable data. They make it easier to accurately assess and compare graduate schools of business and management and their offerings. So far, the MBA Reporting Criteria have been adopted by 109 business schools, as well as by Business Week and the Financial Times. Although U.S. News & World Report has been considering adopting the MBA Reporting Criteria, it has not yet done so. This fall, GMAC will launch the MBA Pathfinder school search database, a collection of data submitted by schools in accordance with the MBA Reporting Criteria. MBA Pathfinder will enable prospective students to compare schools on the basis of the characteristics that they personally regard as most important. It will allow business schools to post complete data and information for all their programs, and it will offer them the option of providing one or two levels of data: Level 1 data will include such basic program features as school location, program duration, concentrations offered, and availability of financial aid. Level 2 data will include more detailed admissions and such program statistics as average GMAT scores and admitted student demographics. For prospective students, MBA Pathfinder will offer a way of navigating through information on graduate management education programs in order to make informed decisions. It will be an alternative to rankings, allowing students to create their own lists of schools to target when they apply for a graduate management degree. For further information about MBA Reporting Criteria and MBA Pathfinder, visit then click on MBA Reporting Criteria. MBA Reporting Criteria: Highlights The MBA Reporting Criteria apply to reports of full-time, part-time, and executive MBA program information. Reports submitted to GMAC for MBA Pathfinder will be subject to an audit by an independent accounting firm (starting in 2003) to assess compliance with the criteria. One-third of the reports each year will be selected randomly for audit. Each reporting entity will be subject to audit at least once every five years. Schools submitting data must provide the following: Twelve months worth of information about new students and the related offers of admission and completed applications associated with those new students; Enrollment information, including address, citizenship, sex, age, work experience, GMAT scores, undergraduate degree (first degrees), and, for U.S. schools, additional information; and 6

8 What WORKS The cost of education, the availability of financial aid, concentrations and joint degrees offered, and information about class sizes. The Employment Standards of the MBA Career Services Council Jackie Wilbur, director of the career development office at MIT s Sloan School of Management, has extensive experience with the MBA CSC, having been a founding board member, vice president, and president of the organization, as well as a cochair and chair of the standards committee. Wilbur spoke to Selections about the MBA CSC s MBA Employment Standards. SELECTIONS: When and why did the MBA CSC develop standards for reporting MBA employment statistics? What are the goals behind the standards? WILBUR: The MBA CSC began the development of reporting standards in 1995 after conducting a current practices survey of more than 75 U.S. [business schools]. The survey measured how schools reported employment data at that time. The results were so inconsistent across schools that the standards committee was formed. In addition, at that time in the MBA industry, the media rankings were generating a lot of discomfort among career services professionals. As the MBA CSC was just beginning to form at this time too, the career directors involved in the early formation of the organization and the standards were concerned that others, within and without the industry, were defining what the MBA career services profession should be. The MBA CSC s mission addresses this issue: Our mission is to set standards for excellence within MBA career services and to serve as the voice for the profession inside and outside graduate management education, as does the standards mission statement: The MBA Employment [ ] The development of uniform standards for reporting employment has served as a valuable benchmark for the industry. Standards Committee was formed to develop reporting standards appropriate for the MBA career services profession. These standards are to be used primarily as internal Jackie Wilbur (to our industry) benchmarking indices to support our profession. SELECTIONS: How did the MBA CSC go about developing standards? WILBUR: A committee of between 10 and 15 schools first developed a set of definitions, including what is a job offer, what is a job acceptance, what is a salary, and when should schools collect employment data. A full list of definitions and participating standards committee schools can be found within the standards at SELECTIONS: Has the MBA CSC achieved its goals with its reporting standards? WILBUR: Almost. Since the employment market continues to evolve, we anticipate having to make minor edits as time goes on. We are also still focused on assisting schools on their implementation of the standards. Five regional training programs were held on members campuses [Rice, University of Southern California, Northwestern, University of North Carolina at Chapel Hill, and MIT] this summer; I believe between 75 and 100 schools participated in the training. SELECTIONS: Of those publications that publish business school rankings, which require schools to submit data following the MBA CSC standards? WILBUR: U.S. News was the first to adopt our reporting standards. Since that time, Business Week, the Financial Times, the Wall Street Journal, and others have adopted the standards. SELECTIONS: What do you see for the future of MBA employment data reporting standards? WILBUR: The committee has discussed the ongoing need for school training. The non-u.s. schools have requested that the standards be adapted to their needs. We are building a comprehensive Web site capable of [handling] detailed queries. [I also anticipate] further collaboration with GMAC and AACSB [AACSB International] on their data collection and dissemination efforts.// Selections: Fall

9 A Look behind the NUMBERS Yet another magazine or newspaper has come out with its rankings, or is about to. Will it be time to break out the champagne or will it be Wait till next year? Selections sets out to compare and explain four major business school rankings, compiled by Business Week, U.S. News & World Report, the Wall Street Journal, and the Financial Times. 8 Selections: Fall

10 Think of a business school as a feature film. Audiences may flock to a film because of the story and the stars or because the reviews say it s good. According to GMAC s Global MBA Survey 2001, most graduating MBAs paid a great deal of attention to the reviews when deciding where to attend business school. Of the nearly 5,000 respondents, 95 percent said that school rankings had more influence on their decision-making process than any other media source. Furthermore, rankings have global reach, being an influential media source for 95 to 96 percent of respondents worldwide. But, in some respects, rankings are not quite like movie reviews: We only spend several dollars per ticket and a few hours at a movie; a business school student invests a great deal of money, time, and effort in business school and, to some extent, banks a career on choosing the right school. Believing that rankings can be useful only if we know what they mean and how they are determined, Selections set out to elucidate and compare the four major rankings of business schools those compiled by Business Week (BW), U.S. News & World Report (USN), the Wall Street Journal (WSJ), and the Financial Times (FT). BY GAIL TYSON ILLUSTRATION BY GENE GREIF Selections: Fall

11 In addition to ranking MBA programs, BW comments on trends it perceives in both the student market and the corporate market, with regard to, for example, the demand for technology, desired class offerings, employment opportunities, and salary packages. BusinessWeek Goals of the Major Rankings In addition to ranking MBA programs, here is what each publication does. BW: Comments on trends it perceives in both the student market and the corporate market, with regard to, for example, the demand for technology, desired class offerings, employment opportunities, and salary packages. FT: Offers resources to prospective MBA students to help them choose the program that best fits their interests, plus information on financing the degree, preparing for admission assessments, and applying. Who Is Surveyed BW (2000): Distributed survey completely online to 419 companies that recruit and invited 16,843 MBAs from the class of 2000 to participate. Received responses from 247 corporate recruiters and 10,039 students at 82 business schools worldwide (the largest pool ever surveyed by the magazine). FT (2001): Sent one questionnaire to 137 schools, and a second to all alumni in the class of 1997 to chart their progress; of a total of 20,700 questionnaires, 5,918 were returned. (FT does not provide a breakdown of school versus alumni responses.) To produce the rankings, the Financial Times compiled these data and an independent assessment of the research. USN (2002): Computed in January 2001, on the basis of data from surveys sent out the previous fall. Assessed 341 schools with master s programs, of which 105 with full-time programs provided all data. Methodology included two surveys, one sent to corporate recruiters who hired from previous top-ranked programs and one sent to deans and directors of accredited programs. WSJ (2001): Surveyed 1,600 corporate recruiters for ratings and rankings of 50 schools. Between August 23 and December 15, 2000, the 1,600 recruiters completed the survey online. 10 Selections: Fall

12 USN: Provides two types of data to help make comparisons: expert opinion about reputation and statistical indicators (both input and output measures; i.e., student data and placement data). WSJ: Identifies which business schools produce the most marketable students, helps recruiters identify additional schools, and offers insights about the strengths and shortcomings of MBA programs and graduates. Differences among the Rankings BW: Describes itself as the first to produce rankings (in 1988). (U.S. News & World Report published results of a limited survey in 1987.) FT: Provides the first global, integrated ranking of the world s top full-time MBA programs and says it is the first business school ranking to include research as a criterion. USN: Ranks MBA programs as part of a broader index of top graduate programs. WSJ: Bills itself as the first to focus exclusively on the opinions of corporate recruiters and says, We are not rating the business schools, but rather reporting on the opinions of corporate recruiters. Non-U.S. Schools Included BW: Named the top seven non-u.s. schools (in addition to 30 in the United States). FT: Included 32 European, 9 Canadian, 3 Australian, 2 Asian, and 3 Central and South American (in addition to 51 U.S. schools). We review our statistical surveys each year to sharpen our questions so that our indicators yield results as closely comparable as possible over all schools. U.S.News USN: Consults the Association to Advance Collegiate Schools of Business (AACSB International) for accredited master s programs in business in the United States. WSJ: The final sample comprised 188 U.S. and 56 foreign schools. Meetings and discussions with experts in the field of MBA recruiting identified the initial group of non-u.s. schools considered. What the Rankings Evaluate BW: Grades 30 U.S. schools and 7 non- U.S. schools. FT: Ranks the top 100 schools worldwide and describes the central measurements as the career progression of alumni, international experience and mobility, and the school s track record in producing new ideas in management. USN: Uses GMAT scores to determine student selectivity and strengths of the full-time students entering a university. The surveys of corporate recruiters and deans/directors use three broad categories (reputation, placement success, and student selectivity), with eight quality indicators to measure the schools performance. Selections: Fall

13 HOW THE RESULTS COMPARE TOP 10 BUSINESS SCHOOLS U.S. WALL STREET JOURNAL (2001) BUSINESS WEEK (2000) U.S. NEWS (2002) 1 Dartmouth (Tuck) U. Penn (Wharton) Stanford 2 Carnegie Mellon Northwestern (Kellogg) Harvard 3 Yale Harvard Northwestern (Kellogg) 4 U. Michigan MIT (Sloan) U. Penn (Wharton) 5 Northwestern (Kellogg) Duke (Fuqua) MIT (Sloan) 6 Purdue (Krannert) U. Michigan Columbia 7 U. Chicago Columbia U.C. Berkeley (Haas) 8 Harvard Cornell (Johnson) Duke (Fuqua) 9 Southern Methodist (Cox) U. Virginia (Darden) U. Chicago 10 U. Texas Austin (McCombs) U. Chicago U. Michigan TOP BUSINESS SCHOOLS INTERNATIONAL WALL STREET JOURNAL (2001)* BUSINESS WEEK (2000)** FINANCIAL TIMES (2001) 22 U. Western Ontario (Ivey) 1 INSEAD 1 U. Penn (Wharton) 26 ESADE 2 London Business School 2 Harvard 28 INSEAD 3 U. Navarra (IESE) 3 Stanford 31 IPADE 4 IMD 4 U. Chicago 33 IMD 5 U. Western Ontario (Ivey) 5 Columbia 39 London Business School 6 Erasmus (Rotterdam) 6 MIT (Sloan) 48 York 7 U. Toronto (Rotman) 7 INSEAD 49 Instituto de Empresa (IE) 8 London Business School 50 U. Navarra (IESE) 9 Northwestern (Kellogg) *Wall Street Journal s list includes U.S. and international schools, but no international school ranks higher than 22. ** Business Week created a list of European and Canadian schools that is separate from its U.S. list. 10 NYU (Stern) 12 Selections: Fall

14 First global, integrated ranking of the WSJ: At least 20 recruiters ranked each of the 50 schools included in the results; 27 attributes, including such student attributes as interpersonal and teamwork skills, are used. How They Weight the Data BW: Combines the student, recruiter, and intellectual capital scores, weighted 45 percent/45 percent/10 percent. FT: Refers to 20 weighted criteria, although the Web site describes only 19, as follows: The first 8 reflect the performance of the MBA program (55 percent). The next 8 measure diversity and international exposure (20 percent). The final 3 criteria measure the school s performance in research (25 percent). The DVD Version, or Analyzing the Analyses Although the ranking surveys explain their different methodologies, Crain s Detroit Business says that universities suspect students don t read those. What if, like the DVD version of a favorite film, business schools put the rankings in a larger context? By making a case study of the rankings, current MBA students could analyze both their quantitative and qualitative impact. Those analyses, in turn, could help prospective students think critically about the annual business school rankings and how they influence one of the most important investments the students will ever make. world s top full-time MBA programs... [and] the first business school ranking to include research as a criterion. FINANCIAL TIMES USN: Weighted average of eight quality indicators (specialty rankings are based solely on reputation): Reputation (40 percent) was based on: - Responses from 53 percent of business school deans and directors surveyed account for one quarter of the overall score. - Opinions of 26 percent of corporate recruiters who responded are weighted by 15 percent. Placement success (35 percent) was measured by: - Mean starting salary and bonus and employment rates for year 2000 grads (40 percent) - Salary and bonus and employment rates computed at graduation (20 percent) - Salary and bonus and employment rates computed three months after graduation (40 percent) Student selectivity (25 percent) measured the strength of full-time students entering in fall 2000 by: - Mean GMAT score (65 percent) - Mean undergraduate GPA (30 percent) - Percentage of applicants accepted by the school (5 percent) Selections: Fall

15 WSJ: Rating is based on three components: Perceptions of the school on the 12 school and 13 student attributes (75 percent). Perceptions of the school on the two overall attributes: ability to meet your overall recruiting needs in terms of the number and quality of students and overall value for the money invested in the recruiting effort (5 percent). Mass appeal of the school (20 percent), as defined by the total number of recruiters choosing to rate a school. History of the Methodologies BW: Since it began ranking MBA programs in 1988, BW says it has increased the number of surveys it disseminates, as well as the number of schools and companies that receive them. For the 2000 poll, BW introduced its first completely online survey and an intellectual-capital component, which measured a school s influence and prominence in the world of ideas. FT: Although FT s 2001 survey asked alumni to itemize their shareholdings and stock options, there was no common We are not rating the business schools, but rather reporting on the opinions of corporate recruiters. benchmark for measuring their responses and using the measure in the rankings. Additionally, FT s use of data from the class of 1997 made some schools ineligible to participate, and its reliance on the salaries alumni earn makes it impossible to use the same criteria to measure some non-u.s. schools that accept students directly from undergraduate programs. USN: The magazine states, We constantly strive to improve our data processing procedures to spot errors. We review our statistical surveys each year to sharpen our questions so that our indicators yield results as closely comparable as possible over all schools. USN s Web site does not reveal whether it conducts the survey online, on paper, or by phone. WSJ: The results published in 2001 were based on the first online interviews conducted by WSJ/Harris Interactive Business School Survey from August 23 to December 15, Other Business Programs the Rankings Assess BW: Publishes more than 200 profiles of part-time MBA programs around the world and a database of distance-learning MBA programs. FT: Ranks international, nondegree executive education programs (custom programs and open-enrollment programs). Financial Times compiles these rankings from the responses of executive education course participants, corporate purchasers, and a THE WALL STREET JOURNAL 14 Selections: Fall

16 survey of business schools. Schools with more than $2 million worth of business in each of the two categories are eligible to participate in either or both rankings. USN: None. WSJ: None. What Does Each Methodology Capture? Because each methodology is so different, wise consumers will consult the publication that best suits their needs. People will also find the publications more or less satisfying according to the broad categories of information provided, which are as follows. BW: Trends in business, education, and the rankings themselves. FT: Complex analysis of career progression (especially internationally), diversity of experience, and research. USN: Insiders perspective (deans/program heads), standardized test scores, and placement success. WSJ: Outcomes data, including salary trends and opinions of recruiters.// Gail Tyson, a frequent contributor to Selections and GMAC s Graduate Management News, is a freelance writer who serves clients in business, education, and healthcare. Selections: Fall

17 The PEOPLE behind the Rankings The creators of the influential b-school surveys explain why they do what they do BY CARLOTTA MAST JULIANA THOMAS BILL AUTH Joy Sever, senior vice president and director of the reputation practice at Harris Interactive. Bob Morse, director of data research, U.S. News & World Report 16 Selections: Fall

18 Ranking business schools has become its own industry, with four major publications now in the market and others jumping into the fray now and then. The methodologies and interpretations of what makes a b-school "the best" differ so much that the same school can be ranked number 1, 11, or 45 in different surveys. It depends on who is being asked what and how much weight is assigned to various answers. Why do these publications put so much time and effort into ranking MBA programs? How sound are their methodologies? Do their motives go beyond boosting their bottom lines? And to whom do they hold themselves accountable? Selections took these and other questions to the editors and statisticians behind the most influential b-school rankings. Among other things, their responses prove that, just like graduate management education, the business of ranking business schools is growing increasingly competitive and complex. JULIANA THOMAS PHOTO COURTESY OF FINANCIAL TIMES Jennifer Merritt, editor, Business Week Della Bradshaw, editor of the Financial Times s annual MBA survey Selections: Fall

19 The output at almost all of the schools... is much better. I think there is almost total universal agreement on that, [and] one of the reasons is schools were fearful of having a poor showing in the survey. John Byrne, senior writer, Business Week JULIANA THOMAS It was in 1988 that John Byrne, then a new reporter at Business Week, began surveying thousands of MBA graduates and corporate recruiters about their business school experiences. Though he wasn t sure how he would use the information, he did know that what he was doing was important. I have to tell you what a labor of love this was, says Byrne, now a senior writer at Business Week. In the early days, I actually made up the surveys on my Macintosh at home. My wife and I stuffed envelopes in front of the television set. I used every connection I had to make the corporations respond to the survey oftentimes begging them on the phone and in person to do so. But once published, the Business Week survey quickly became one of the most widely known and anticipated outside assessments of MBA programs in the United States, and it opened the floodgates for ratings by U.S. News & World Report, the Financial Times, and the Wall Street Journal. 18 Selections: Fall

20 Holding Schools Accountable Byrne doesn t mince words when explaining why he felt it was important to create a ranking when he did. With a few exceptions, the schools didn t seem to care about the perceptions of their customers, the people who actually buy their product, he says. Byrne thought a ranking would create a market where schools could be rewarded and punished for failing to be responsive to their two primary constituents: the students and the corporations. Byrne says he felt the b-schools of the late 80s were too heavily focused on research at the severe expense of teaching quality and weren t producing graduates who could work well with others. So, with the help of a statistician, Matthew Goldstein, and using feedback from students, Byrne developed a survey methodology that measured the judgments of recent MBA graduates and recruiters to evaluate teaching quality, placement services, emphasis on teamwork, and other elements he felt were important to a quality management education. The first Business Week survey results, published in November 1988, ranked Northwestern s Kellogg School of Management ahead of two traditional MBA powerhouses, Harvard and Stanford. That ranking, he says, surprised the b-school community and sent some deans scrambling to copy what Kellogg seemed to be doing right: pleasing students and recruiters with a team-based approach to management education. In Byrne s view, reaction to the survey has resulted in improvements across the board. The output at almost all of the schools even those that won t acknowledge that they have responded to the marketplace is much better, he says. I think there is almost total universal agreement on that, [and] one of the reasons is schools were fearful of having a poor showing in the survey. Holding the schools accountable has forced them to serve their customers better, says Jennifer Merritt, the Business Week editor now responsible for the survey. But Merritt is quick to acknowledge that some schools have become obsessed with the rankings. We started hearing a couple years ago about people being fired after the rankings came out, she says. That is the most ridiculous thing I have ever heard. That is not the role of the Business Week rankings. Adding to the List of Contenders Alongtime publisher of the annual report on America s Best Colleges and Universities, U.S. News was already heavily entrenched in the business of ranking schools when in 1990 it added MBA programs to the list of graduate disciplines it surveys annually. (U.S. News did publish an abbreviated version of its b-school ranking in 1987.) With so many people eager to earn an MBA, U.S. News wanted to provide applicants with information to help them make a choice, explains Bob Morse, director of data research. Morse says a good dose of his own judgment went into developing the methodology including the weighting system for the U.S. News ranking, which is based on expert opinion about program quality (gauged by surveying deans and department heads) and such statistical indicators as GMAT scores and salary data. As Morse explains it, he did a lot of research and solicited input from b-school administration and faculty when developing the U.S. News methodology. If you were a student and you read our methodology, you would have fewer questions about how it is done than the others, he says. If you were to ask any dean, they would say that ours is the most transparent by far. Selections: Fall

21 The Financial Times entered the b-school ranking scene in 1997 with the goal of creating the first international MBA ranking. (Business Week recently supplemented its rankings with a separate list of international schools.) There was a lot of concern that all of the rankings were solely U.S. rankings, says Della Bradshaw, editor of the Financial Times s annual MBA survey. What our ranking has done is give prominence to schools outside of the U.S. The Financial Times formed an advisory board of deans and faculty members from different b-schools to guide the process of creating its ranking. Bradshaw says research was one of the elements that they were very keen on, which is why the Financial Times measures research quality along with career progression and diversity in its survey. The Financial Times also interviewed students to determine their primary motivations for earning an MBA and learned that students look at b-school mainly as a means for switching careers or landing a higher-paying job. This revelation is the reason behind the publication s decision to weigh input from MBA graduates who are three years out of school, Bradshaw says. We are trying to measure the degree of career progression that an MBA gives you. The Wall Street Journal wanted to do something completely different with its ranking, conducted in partnership with Harris Interactive, a marketing research firm, and published in April The idea was to survey only recruiters to determine which schools produced the most marketable graduates. The results of the survey, which took two years to create and conduct, certainly offered a new perspective. At the top of the Journal s list was Dartmouth s Amos Tuck School of Business Administration (ranked number 11 in the 2002 U.S. News poll and number 16 in the 2000 Business Week survey). Stanford, ranked number 1 by U.S. News and number 11 by Business Week, was all the way down at number 45 in the Journal survey. In terms of the quality of the school and its professors and academic contributions, this survey is not meant to look at that, explains Joy Sever, senior vice president and director of the reputation practice at Harris Interactive. This is a survey purely about recruiters perceptions. Stanford did poorly in the Wall Street Journal ranking, Sever adds, not because recruiters don t believe it is a quality school but because they didn t have success hiring from Stanford s small class of elite students. While the Wall Street Journal methodology clearly stands out as different, each of the publications uses a unique system for ranking MBA programs and that, argues every editor interviewed for this story, must be taken into account when looking at and comparing the rankings. Rather than saying, Well, Business Week, the Wall Street Journal, and U.S. News don t come up with the same results, people should realize these are enterprises that ask different questions and have different perspectives, says Gayle Garrett, project director for the U.S. News Best Graduate Schools guide. The Quest for Unbiased Results As Morse of U.S. News points out, Each of the rankings has its weaknesses. And the rankings creators aren t shy about pointing out what is wrong with their competitors surveys. One complaint about the methodologies used by Business Week and U.S. News is that they open the door to bias by using the judgments of students and administrators to rank the schools. In fact, Sever of Harris Interactive says one reason the Wall Street Journal ranking doesn t include input from either of these constituents is that she couldn t figure out a way to include their input without introducing bias. Those two groups are ones that are most frequently felt to introduce responses that definitely have a kind of selfserving aspect to them, Sever says. 20 Selections: Fall

22 Students can be coached as to how to answer the questions in the Business Week survey. And even if they are not officially prodded to give their school high marks, Morse argues, students understand the benefits that come with having graduated from a top school. They know that the value of their degree determines salaries in the marketplace, he says. Let s not be naïve. Evidence of student coaching did surface in 1998, acknowledges Merritt of Business Week, and the schools involved were penalized. If that were to happen again, the offending schools would be thrown out of the ranking. We ve gotten very strict on it, Merritt says. Business Week forbids school administrators to communicate with their students about the rankings and has hired a statistician to examine the survey results for any evidence of untoward influence. In addition, students are encouraged to include written comments on their surveys, and Byrne says students often use this as an opportunity to blow the whistle. You will find out about the professors who just don t give a damn by name. You ll find out about the placement offices How to assure untainted results? Business Week forbids schools to communicate with students about rankings and has a statistician examine results. JULIANA THOMAS Selections: Fall

23 To help prevent bias, U.S. News eliminates the two highest and the two lowest values when weighing responses about a school s reputation. Gayle Garrett, project director for Best Graduate Schools, U.S. News & World Report BILL AUTH that drop the ball. And you ll find out if there was any sort of effort at all on campus to rig the results. As to whether school loyalty colors students responses, Merritt dismisses the notion. We feel they are smart enough and savvy enough to be fair. But given that students attend only one school, can they really be expected to compare their MBA experience objectively to that of students at other schools? Of course not, Merritt says. You can t judge where you haven t been, so we ask them only about their schools. If we take out the student criteria, then we ve basically got a corporate survey, and we don t think that is appropriate. To help prevent bias from tainting its results, Garrett says, U.S. News eliminates the two highest and the two lowest values when weighing responses from deans and department heads to calculate the score for the reputation portion of its survey. We do that to take out the effect of people who vote for themselves as a perfect 5. Still, Byrne argues that deans and department heads can t be counted on to assess their peer schools accurately. Most 22 Selections: Fall

24 deans don t know what the hell is going on on their own campus, let alone what the hell is going on on other campuses, he says. That, contends Morse, is nonsense particularly given the increasingly competitive nature of the b-school landscape. Deans and program directors know about what is going on, Morse says. They rub shoulders with each other. They are competing over faculty. They are competing over research proposals and other things. They have a high level of knowledge of what are the better programs. The debate over methodology isn t limited to Business Week and U.S. News. The Wall Street Journal s CareerJournal.com site, for example, published a lengthy criticism of the Journal s survey methodology by John G. Lynch, Jr., a marketing professor at Duke s Fuqua School. Lynch argued that the Journal used a set of recruiters that was biased and open to manipulation, in part because rather than sampling randomly from all recruiters who visited a school, they allowed schools to choose which recruiters from a company would be invited to participate in the survey. CareerJournal.com published an equally lengthy response from Sever. Whatever may not be perfect about our survey, at least we are very open so that the schools can evaluate what we ve done, Sever comments, adding: We would never have published results if we didn t think that we could stand behind them. Agreeing to Definitions Alack of data auditing procedures is another common criticism of the rankings particularly of the U.S. News survey, which factors into its ranking system a good deal of hard data collected from the schools, including GMAT scores and starting salaries. I can tell you, Byrne says, the schools lie. To help prevent deception and to ensure that schools submit accurate, consistent, and comparable data, in 2000, GMAC released commonly accepted definitions for admission and MBA program data elements through its MBA Reporting Criteria initiative. (See p. 6 for more information on the MBA Reporting Criteria.) Business Week and the Financial Times have adopted the MBA Reporting Criteria; discussions are currently under way with U.S. News. Back in 1995, after completing a current practices survey of more than 75 U.S. schools offering MBAs, the MBA Career Services Council (MBA CSC) created its own standards for reporting job placement and salary numbers. U.S. News was the first to adopt these standards, and Business Week, the Financial Times, and the Wall Street Journal have now adopted them, too. In addition to working with such organizations as the MBA CSC and GMAC, Garrett says U.S. News is constantly trying to make its survey methodology more transparent and improve the integrity of the data it publishes. She says she is accountable for making certain that whatever we publish is accurate. When data comes back to us... there is a minimum of two separate processes by which we send information back to the schools to be verified. While Morse says he hasn t seen a school outright lie in its survey responses, he does say the standards have greatly improved the quality of the data being supplied by the schools. U.S. News, and maybe Business Week to some degree, have been responsible for causing data standardization, he says. Selections: Fall

25 With business schools, because there isn t any licensing degree in the marketplace, the reputation of the school is what is determining the value of a degree. Bob Morse, director of data research, U.S. News & World Report BILL AUTH The Value of Rankings Two undeniable benefits of the rankings are that they compel schools to share data that might otherwise have been kept confidential, and they provide an abundance of useful information to prospective MBA students and to recruiters. But why put so much effort into distilling a school s overall performance into a ranking number particularly when doing so opens the door to so much external criticism? In Morse s view, if any field in higher education understands assessments, ratings, judgment, and competition, it is business. With business schools, because there isn t any licensing degree in the marketplace, the reputation of the school is what is determining the value of a degree, he says. Of course, rankings are also a good way to make headlines and catch the public s attention. Rankings are one way to increase your profile, says Parminder Bahra, senior adviser on education projects at the Financial Times. Given the margins for error within the calculation of figures, we could probably have a system where we just tier the schools. But the current system provides for a more interesting read. Bahra s point leads to an important question: Just how much do the rankings affect a publication s bottom line? 24 Selections: Fall