How To Rank A Journal

Transcription

1 Vol. 35, No. 4, July August 2005, pp issn eissn X informs doi /inte INFORMS Top-25-Business-School Professors Rate Journals in Operations Management and Related Fields Josephine E. Olson Joseph M. Katz Graduate School of Business, University of Pittsburgh, Pittsburgh, Pennsylvania 15260, I obtained quality ratings and rankings of 39 journals in operations management and related disciplines through surveys of faculty members at top-25 US business schools in 2000 and in I also computed five-year impact factors for 29 of these journals and developed a ranking based on these impact factors. I found evidence of some change in journal quality ratings over the two-year period. Ratings also differed by research area but not by professorial level. In addition, I ranked the journals based on the number of academics who rated their quality, calling this a visibility measure. Finally, I compared my ratings to ratings in earlier survey and citation studies. The quality ratings were more consistent than the citation ratings. Key words: professional: OR/MS education; communications. History: This paper was refereed. Whether a faculty member is being considered for promotion or tenure or is just facing an annual review, his or her published research papers are often the most important factor in the evaluation. For top research universities and those with aspirations, quality is generally more important than quantity. Because the impact of a specific article is usually difficult to determine until years later, the quality of the journal is frequently used as a proxy for article quality. Many universities now expect their schools and departments to develop lists of appropriate target journals and to rate them by quality. Universities also require external referees for promotion and tenure cases. Unless they are intimately familiar with all the publications of the faculty member under review, these external experts often rely in part on their opinions of the journals that published the faculty member s papers. Practitioners and students also use such ratings to determine where to look for high-quality academic research. Because journal ratings are important in evaluating faculty, authors have conducted many studies of journals in various management-related disciplines. Although people in operations management were slower to evaluate their journals than those in some other disciplines (Barman et al. 1991; Goh et al. 1996, 1997; Soteriou et al. 1999), at least seven studies of journals related to operations management have been published since Even so, we need continuing research. Operations management is a diverse and changing discipline. The authors of previous studies have had different objectives, have used different lists of journals, have employed different methodologies and, in survey studies, have targeted different populations, and have therefore obtained different findings. There is little consensus on journal ratings except at the very top. Moreover, because of the diversity in operations management and its changing nature, new journals have appeared and older journals have changed in status. Thus, we regularly require new studies rating the journals in the field. I obtained quality ratings and rankings on journals in operations management and related disciplines by faculty members in operations management and related fields at top US business schools. Although several other researchers employed surveys to rate and rank journals, most surveyed members of professional societies; I am the first to use faculty at top business schools. I also obtained ratings based on the visibility of journals in operations management and based on citation impact factors, and I compared ratings of journals in this and earlier studies. 323

2 324 Interfaces 35(4), pp , 2005 INFORMS Literature Review In earlier studies of operations management journals, researchers used opinion surveys of experts and citation studies to assess journals. Their goals also differed; some focused on quality and others, on influence or relevance (Table 1). In four of the studies, the researchers relied primarily on opinion surveys. Saladin (1985) asked members of the Operations Management Association (now defunct) to list operations management journals that they would classify as Category I/Class A journals and to list journals they would consider to be the most appropriate publication outlets for operations management research. Barman et al. (1991) had members of the Decision Science Institute (DSI) in the US rate 20 journals with respect to the quality of POMrelated articles in each journal and the relevance of the journal to POM. They also asked respondents to indicate their institution s view of these journals. Soteriou et al. (1999) later conducted a survey of Europeans based on Barman et al. s (1991) survey to see if they differed in their perception of journal quality from the US respondents. They asked respondents to evaluate 35 journals with respect to quality and relevance. Finally, Barman et al. (2001) repeated the 1991 survey 10 years later on members of the Production and Operations Management Society using the original 20 journals plus Production and Operations Management, which was first published in In their survey, they asked additional questions regarding respondents roles on editorial boards and the factors that define journal quality. In all three studies based on the Barman methodology, the researchers found that the correlations between rankings of journal quality and relevance were not particularly high. The correlation was 0.56 for Barman et al. (1991), 0.65 for Soteriou et al. (1999), and 0.40 for Barman et al. (2001). The Europeans (Soteriou et al.) tended to rate international journals more highly than did the Americans (Barman et al. 1991, 2001). In both Barman studies, the ratings did not vary significantly by professorial level but the professors believed that their administrators ratings of journals were quite different from theirs. Barman et al. (2001) found that respondents perceptions of a journal s relevance and quality appeared to be somewhat influenced by service on its editorial board, usually in a positive manner. Of eight factors that might be critical in assessing the quality of a journal, methodological rigor ranked highest. In three studies, the authors relied primarily on frequencies of citations to a journal to assess its status. Arguing that many of the operations management journals were not included in the Social Science Citation Index SSCI or the Science Citation Index SCI, these authors developed their own citation data to make their assessments. Goh et al. (1996) chose as base journals the five journals of Barman et al. s (1991) top 10 for quality that focused exclusively on production and operations management. They then collected all the citations for all articles published in these five journals from 1989 to There were almost 20,000 citations from 1,296 journals. They developed two rankings of these journal citations. They based their unnormalized objective rank on the total number of times a given journal appeared as a citation in the five base journals over the five-year period. They determined the other ranking, their normalized objective rank, by weighting each of the five base journals equally. This meant first dividing the number of citations to a specific journal from each base journal by the total number of citations in the base journal. Then they used the sums of these numbers over the five base journals to rank the journals. The purpose of this normalization was to deal, at least partially, with the fact that a base journal with a large number of citations (namely, The International Journal for Production Research) would tend to dominate the other four journals. According to Goh et al. (1996), their rankings indicated the relative importance (p. 338) or influence (p. 342), rather than the quality, of the various journals in the field of production and operations management. Vokurka (1996) also developed his own citation data to determine the relative importance of academic journals to OM research (p. 345). He used three journals as his base journals, those high in the rankings of both Saladin (1985) and Barman et al. (1991). Because two of these journals publish articles outside of operations management, Vokurka had to eliminate articles that he judged to be outside the field. Vokurka looked at articles published by the three journals from 1992 through Like Goh et al. (1996), he counted citations to all journals in

3 Interfaces 35(4), pp , 2005 INFORMS 325 Opinion surveys Citation studies Group surveyed Criterion for journals Criterion for journals Criterion for journals Criterion for journals (and number and top 10 journals and top 10 journals Base journals and top 10 journals and top 10 journals Author(s) of responses) by rank by rank Author(s) and years by rank by rank Saladin Academic members Category I/Class A Most appropriate Goh et al. Top five journals Normalized objective Unnormalized (1985) of Operations journals (only outlets for OM (1996) focusingexclusively rank (ratio of objective rank (total Management three journals research (only on POM from number of citations number of times a Association (61) were ranked) three journals Barman et al. to a journal divided given journal was were ranked) (1991) JOM, by the total number cited over the five- IIET, IJPR, of citations in the year period) JMOM, IJOPM base journal and ( ) then summed over the five journals) JOM, MS, DS JOM, MS, DS MS, IJPR, IIET, OR, IJPR, MS, IIET, OR, HBR, PIM, JOM, HBR, PIM, JOM, DS, IJOPM, EJOR DS, EJOR, JMS Barman US academic Quality of 20 journals Relevance of Vokurka Three high ranking Total citations Citations per OM et al. members of with respect to 20 journals with (1996) journals from article (1991) Decision Science POM-related respect to POM Saladin (1985) Institute with articles and Barman et al. POM as primary (1991) DS, JOM, interest (225) MS ( ) MS, DS, JOM, IIET, JOM, IJPR, IJOPM, MS, IJPR, OR, DS, MS, DS, HBR, JOM, IJPR, HBR, OR, JMOM, DS, PIM, IIET, JOM, IIET, HBR, OR, IIET, INFCS, NRL, JMOM, EJOR INFCS, MS, JPMM PIM, NRL, EJOR IJPR, NRL, PIM Soteriou European academic Quality of 35 journals Relevance of Citations per OM et al. members of with respect to 35 journals with words published (1999) INFORMS and POM-related respect to POM academic members articles of Euro with OM focus (106) MS, OR, JOM, IJOPM, JOM, IJOPM, POM, MS, DS, HBR, OR, IJPR, POM, EJOR, IJPR, IJPE, PIM, JOM, PIM, IIET, HBR, AMJ, DS MS, HBR, EJOR, INFCS, NRL, IJPR SMR Barman US academic Quality of 21 journals Relevance of Goh et al. Three journals Journals assigned as et al. members of the with respect to 21 journals with (1997) from their prior elite, major, (2001) Production and POM-related respect to POM study JOM, important, and Operations articles IJPR, IJOPM notable based on Management ( ) breadth, consistency, Society (223) trend, and intensity of recognition. MS, OR, JOM, DS, JOM, POM, IJOPM, Elite journals in POM, HBR, IIET, PIM, IJPR, DS, alphabetical order: NRL, EJOR, INFCS INFCS, MS, IIET, DS, HBR, JOM, HBR IJOPM, IJPR, MS, PIM Table 1: This table shows a brief summary of previous studies ranking OM-related journals. The journal abbreviations are as follows: AMJ (Academy of Management Journal), DS (Decision Sciences), EJOR (European Journal of Operational Research), HBR (Harvard Business Review), IIET (IIE Transactions), IJOPM (International Journal of Operations and Production Management), IJPE (International Journal of Production Economics), IJPR (International Journal of Production Research), INFCS (Interfaces), JMOM (Journal of Manufacturing & Operations Management, now IJPE), JMS (Journal of Manufacturing Systems), JOM (Journal of Operations Management), JPMM (Journal of Purchasing and Materials Management), MS (Management Science), NRL (Naval Research Logistics), OR (Operations Research), PIM (Production and Inventory Management Journal), POM (Production and Operations Management), SMR (Sloan Management Review).

4 326 Interfaces 35(4), pp , 2005 INFORMS the OM articles in the three base journals published over this three-year period. He had 2,270 citations to 332 different journals. He then ranked the top 25 journals according to the number of citations to them in the three base journals. Where possible, he also ranked the journals by citations per article published and citations per words published. He made an interesting finding in his study: there was a tendency for a larger than average number of citations in each base journal to be from articles published in that base journal. In another article on POM journals, Goh et al. (1997) used a subset of the citations in the earlier study (Goh et al. 1996) and applied multiple criteria to assign journals to several tiers of influence in the POM discipline: elite, major, important, and notable. Although the authors of these three studies claimed that citation studies are more objective than surveys of experts, these studies have their own subjectivities. The choice of base journals can cause a possible bias because, as Vokurka (1996) found, journals may lean towards self-citation. In addition, the researchers choice of years in all three studies, Vokurka s choice of articles, and Goh et al. s (1997) choice of additional attributes and their hurdles potentially affected their results. New journals, specialized journals, and journals with small circulations are also at a disadvantage in citation studies (Goh et al. 1996). Methodology I took a survey approach to assessing journals relevant to operations management, but I also developed impact factors for comparative purposes. Instead of relying on members of a specific professional society, such as DSI, INFORMS, or EURO, I based my study on two surveys of the faculty of top-25 US business schools. I carried out the first survey in 2000, and I carried out the second in 2002 to update the findings of the first survey and to extend it to a wider list of journals. Because I used essentially the same methods for the two surveys, I describe them together. The operations, decision science, and artificial intelligence (ODSAI) faculty interest group of my school designed the first instrument. (I am not a member of this group.) It developed the list of journals as follows. The group s coordinator asked members of the ODSAI faculty to identify journals in their areas. They identified over 100 journals. Then, six faculty members rated these journals using a survey instrument similar to the one I used in this study. I included only the 30 journals that three or more of these faculty members rated in my first survey. Two of these 30 journals, however, were no longer published (AIIE Transactions [now IEE Transactions] and Mathematical Programming Study) and thus the actual number of journals was 28. In the second survey, I excluded two additional journals that the coordinator of the ODSAI faculty group felt were not in the area of operations management and included 11 new journals for a total of 37 (Table 2). In both surveys, I gave respondents a list of journals in alphabetical order and asked them to rate them for quality on a seven-point scale from 1 (denotes the very top journals) to 7 (denotes the lowest quality journals). In contrast to Barman et al. (1991, 2001) and Soteriou et al. (1999), I did not modify quality with the additional phrase with respect to POMrelated articles. I told the respondents not to rate the journals they did not know. When it was available, I provided information on circulation, acceptance rate, number of external and internal referees, and frequency of publication, and I gave the sources of this information, usually Cabell s Directory of Publishing Opportunities in Management and Marketing (1997) or Ulrich s Periodicals Directory On-Line (2002). I also asked respondents to rate the journals for audience and to give their primary research area(s), their faculty rank, and (in the second survey only) their tenure status. Because the surveys are nearly identical, I reproduce only the second survey in the appendix. I chose the target respondents for the surveys as follows. For the first survey, I chose 25 of the top 27 American business schools listed in U.S. News and World Report: Best Graduate Schools, 2001 Edition (2000). (Because of ties, it often has more than 25 schools in its top 25. ) For the second survey, I used the top 27 business schools reported in the April 15, 2002 edition of the magazine. The schools surveyed are listed in the appendix. I chose U.S. News primarily because it was one of the oldest rankings and was published every year. I identified faculty members with assistant, associate, or full professor rank from the schools Web sites by searching under

5 Interfaces 35(4), pp , 2005 INFORMS 327 Number of Mean Sidebars Journals year Ranking respondents quality Standard indicatingduncan in the impact based on Ranking journal rating rating deviation Duncan grouping groupings grouping factors impact factor 1 Management Science Operations Research Mathematics of Operations Research 4 Mathematical Programming Journal of the American Statistical Association # 6 Manufacturing & Service na Operations Management 7 Naval Research Logistics Transportation Science IIE Transactions SIAM Review # Interfaces INFORMS Journal on & 6 Computing 13 Operations Research Letters Networks European Journal of Operational Research 16 Annals of Operations Research 17 Production and Operations & 25 Management 18 Journal of Operations Management 19 Journal of Combinatorial na Optimization ## 20.5 Decision Sciences Journal of the Operational Research Society 22 Journal of Global Optimization ## 23 Journal of Business na Logistics ## 24 Journal of Scheduling ## International Journal of Production Research 26 Journal of Supply na Chain Management ## 27 Mathematical and Computer Modelling 28 Journal of Heuristics ## Computers and Operations Research Table 2: This table lists the 39 published journals included in one or both surveys. The first column shows the ranking of the journal based on the mean quality rating (with ties shown as the average of the two). The second column shows the number of persons who rated the quality, the third column shows the mean quality rating, and the fourth column shows the standard deviations of the ratings. The sidebars next to the fourth column show groupings resulting from Duncan s multiple range test. The fifth column shows the grouping number, and the sixth column indicates the journals in the grouping, shown by their rank number. The impact factors and the resulting ranking are shown in the last two columns. The symbols and their meanings follow: tied in ranking; # journal appeared only in the first survey; ## journal appeared only in the second survey; na not available; & 2003 four-year impact 2003 one-year impact factor.

6 328 Interfaces 35(4), pp , 2005 INFORMS Number of Mean Sidebars Journals year Ranking respondents quality Standard indicatingduncan in impact based on Ranking journal rating rating deviation Duncan grouping groupings the grouping factors impact factor 30 International Journal of Production Economics 31 International Journal of na Operations and Production Management ## 32 International Journal of Flexible Manufacturing Systems ## 33 Decision Support Systems na and Electronic Commerce 34 Journal of Manufacturing Systems ## 35 Omega Production and Inventory na Management Journal ## 37 Computers and Industrial na Engineering 38 American Journal of na Mathematical and Management Science 39 International Journal of na Operations and Quantitative Management ## Table 2: (Continued) departments or research areas, such as operations, operations and technology management, decisions sciences, quantitative methods, statistics and operations research, management science, information and operations management, operations and manufacturing, technology and innovation, manufacturing, and the like. When departments were broader than purely operations management and operations research, I tried to limit the sample to those in the relevant areas but erred on the side of being inclusive. For the second survey, I followed a similar method. For the first survey, I obtained the names and addresses of 254 faculty members. For the second survey, I obtained 226. I sent the surveys via as attached Excel files, the first in May 2000 and the second in June In both cases, I sent out two followups. For the first survey, 12 surveys were returned because of bad addresses. Of the 242 surveys that reached their destinations, I had 111 responses, of which 85 sent completed surveys, giving a usable response rate of 35 percent. At least one faculty member responded from 24 of the 25 schools. For the second survey, five had bad addresses. Of the 221 surveys that reached their destinations, I had 105 responses, of which 92 sent completed surveys, giving a usable response rate of 42 percent. At least one faculty member responded from each of the 27 schools. Some of the reasons 39 respondents did not include completed surveys were that they were not in the field (15), not active researchers (6), and did not want to participate or were too busy (8). Six respondents had trouble accessing the survey or forgot to attach the file. In the combined study, I obtained 177 usable responses; 28 persons responded to both surveys. In addition to the surveys, I computed five-year impact factors for as many of the 39 journals included in one or both surveys as possible. Goh et al. (1996) and Vokurka (1996) developed their own citation information because they argued that the automated referencing systems did not include many of the journals relevant for operations management. However, the situation appears to have improved since I found 29 of the 39 journals in Table 2 in either the science or social science editions of the 2003 ISI Journal Citations Report JCR, including 18 of those ranked in

7 Interfaces 35(4), pp , 2005 INFORMS 329 my top 20. Using information from the 2003 JCR and earlier reports, I computed five-year impact factors for These show the average number of citations in 2003 to articles published in the specific journal from 1998 to Five-year impact factors seem more appropriate than the two-year factors found in JCR because Gupta (1997) determined that three years elapsed before most operations research articles were cited. I could compute only a four-year impact factor for INFORMS Journal on Computing and for Production and Operations Management. There was only one year of data for the Journal of Scheduling. Results Most of the respondents in both surveys appeared to be tenured professors (Table 3). About two thirds in both surveys listed their primary research area as operations management, 16 percent listed their primary area as operations research, and seven percent listed it as decision analysis. The other fields listed were primarily statistics or logistics. I ranked the 39 journals included in one or both surveys based on their mean quality rating (with ties shown as the average of the two) (Table 2). I also show the number of persons who rated the quality of the journal and the standard deviation of the First survey Second survey Number Percent (%) Number Percent (%) Professorial rank Full professor Associate professor Assistant professor Missing1 1 0 Tenure stream Tenured na Tenure track na Missingna 1 1 Primary research area Operations management Operations research Decision analysis Other fields Missing Total Table 3: This table shows faculty rank and research areas for respondents of both surveys. I asked for tenure stream status only in the second survey. ratings. Stem and leaf plots for each journal did not show any bimodal distributions; and, except for the most highly rated journals, the distributions around the mode were fairly symmetric. Because the difference in means was small for closely ranked journals, I applied Duncan s multiple range test to see if journals could be grouped, and I obtained 14 groupings. The first group includes Management Science, Operations Research, and Mathematics of Operations Research. The second group includes Mathematics of Operations Research (again), Mathematical Programming, Journal of the American Statistical Association, and Manufacturing & Service Operations Management. There is no overlap between the second and third group, but the later groupings overlap so extensively that it would be arbitrary to form clusters of journals with distinct quality ratings. I obtained impact factors and their resulting rankings (Table 2). (I did not include Journal of Scheduling in the ranking because there was only one year of data.) The highest impact factors are for SIAM Review, Journal of the American Statistical Association, Journal of Operations Management, and Management Science. The correlation coefficient between the quality ratings and the impact factors for the 28 journals is Although this is statistically significant at the 0.05 level for a two-tail test, it does not indicate a close correlation between quality ratings and impact factors. Because I administered the two surveys two years apart, I next conducted t-tests to determine whether there were significant differences in the mean quality ratings of journals over the two-year period for the 26 journals that were included in both surveys (Table 4). Mean ratings for six of the 26 journals were statistically significantly different between the two surveys at the 0.05 level or better. The binomial probability of finding at least six significant results in 26 tests at the 0.05 level is close to zero, and thus I concluded that perceptions of the quality of some journals did change over the two-year period. Because the rankings are based on ordering the mean ratings from smallest to largest, any difference in mean can cause a change in the rankings, even if the change is not statistically significant. Although the two rankings differ somewhat, their correlation coefficient is and the correlation for the mean ratings

8 330 Interfaces 35(4), pp , 2005 INFORMS Number of Quality Number of Quality Rankings of Rankings of raters in mean in raters in mean in 26 journals 26 journals Journal survey 1 survey 1 survey 2 survey 2 in survey 1 in survey 2 Management Science Operations Research Mathematics of Operations Research Mathematical Programming Manufacturing & Service Operations Management Naval Research Logistics Transportation Science IIE Transactions Interfaces INFORMS Journal on Computing Operations Research Letters European Journal of Operational Research Networks Production and Operations Management Journal of Operations Management Decision Sciences Annals of Operations Research Journal of the Operational Research Society International Journal of Production Research International Journal of Production Economics Decision Support Systems and Electronic Commerce Computers and Operations Research Mathematics and Computer Modelling Omega Computers and Industrial Engineering American Journal of Mathematical and Management Science Table 4: This table compares the mean quality ratings and ranking by survey for the 26 published journals included in both surveys. Also shown are the number of respondents rating the journals in each survey. The symbols and their meanings follow: the means are statistically significantly different at the 0.01 confidence level; the means are statistically significantly different at the 0.05 confidence level. is Both correlations are statistically significant at the confidence level. Barman et al. (1991) tested whether ratings varied by faculty rank. Following their example, I conducted a one-way analysis of variance on the ratings of the 39 journals by the three categories of faculty. Only two journals showed statistically significant betweengroup differences. The binomial probability of finding at least two significant results in 39 tests at the 0.05 level is 0.691; therefore, the results do not support the hypothesis that ratings vary by professorial level. These findings are consistent with those of Barman et al. (1991). Next, I conducted a one-way analysis of variance on the quality ratings of each journal to determine whether there were differences in ratings by the primary research field (Table 5). Because of the small number of respondents in decision sciences, I recoded it as Other. Other now includes decision sciences as well as such fields as statistics and logistics. The maximum number of potential ratings for a journal in both surveys is 119 for operations management, 29 for operations research, and 24 for other; however, not everyone rated all 39 journals and not all journals were in both surveys, and for six journals the number of raters was too small to allow ANOVA. There were 13 statistically significant between-group differences at the 0.05 level or higher (Table 5). Because the binomial probability of finding 13 or more significant results in 33 tests is close to zero, the evidence is strong that journal quality ratings vary with the research field. Post hoc Bonferroni tests indicated which mean ratings were significantly different from others. Mathematics of Operations Research was rated very highly by operations-management researchers and by

9 Interfaces 35(4), pp , 2005 INFORMS 331 Operations Operations management research Other fields Journal rankings by field Number OM Number OR Number of Other of OM quality of OR quality other quality OM OR Other Journal raters means raters means raters means rankingrankingranking Management Science Operations Research Mathematics of Operations Research Manufacturing & Service Operations Management Mathematical Programming Journal of the American Statistical Association IIE Transactions Naval Research Logistics Transportation Science SIAM Review Interfaces Operations Research Letters European Journal of Operational Research Networks INFORMS Journal on Computing Annals of Operations Research Production and Operations Management Journal of Combinatorial Optimization Journal of Operations Management Decision Sciences Journal of the Operational Research Society International Journal of Production Research Journal of Supply Chain Management Journal of Business Logistics International Journal of Production Economics Mathematics and Computer Modelling Computers and Operations Research International Journal of Operations and Production Management Decision Support Systems and Electronic Commerce Omega Production and Inventory Management Computers and Industrial Engineering American Journal of Mathematical and Management Science Table 5: This table shows the mean quality ratings of 33 journals by three academic disciplines: operations management, operations research, and other (which now includes decision analysis as well as other fields) along with the number of raters in each discipline. Six journals were not included because of the small number of respondents rating them. The last three columns show the rankings by discipline derived from the mean ratings by discipline. I tested for statistically significant differences among the three using analysis of variance and the Bonferroni post hoc test. The symbols and their meanings follow: the means are signficantly different at the 0.01 confidence level; the means are significantly different at the 0.05 confidence level; 1 Bonferroni post hoc test indicates OM mean is significantly different from other mean; 2 Bonferroni post hoc test indicates other mean is significantly different from both OM and OR means; 3 Bonferroni post hoc test indicates OM mean is significantly different from both OR and other means; 4 Bonferroni post hoc test indicates OR mean is significantly different from other mean; and 5 Bonferroni post hoc test indicates OR mean is signficantly different from both OM and other means. those in operations research and other areas. Manufacturing & Service Operations Management, a fairly new (1997) INFORMS journal, is ranked fourth by operations management faculty but below the top 10 by those in operations research and other fields. IIE Transactions is also ranked much higher (seventh) by operations-management researchers than by those in operations research and other areas (14th and tied for 16th). For all three research areas, the top two journals are Management Science and Operations Research.

10 332 Interfaces 35(4), pp , 2005 INFORMS Despite some noticeable differences in mean ratings and rankings by research area, the correlation between the OM and OR ratings is 0.887, the correlation between OM and Other ratings is 0.848, and the correlation between OR and Other ratings is All three correlations are statistically significant at the level. These high correlations may reflect the fact that I asked respondents to rate the quality of the journals independent of their relevance to specific research areas. In addition to providing quality ratings and fiveyear impact factors and their corresponding rankings, this study provides a measure of the visibility of the 39 journals among professors in operations management and related disciplines at top-25 business schools. Because I asked respondents to rate only those journals with which they were familiar, the number who rated the quality of a particular journal provides an idea of its visibility among academics in top US business schools (Table 6). I ranked journals by the number of respondents who assessed their quality. For journals listed in only one of the surveys, I doubled the number of respondents rating the journal to approximate the responses they might have received had they appeared in both studies. The four most highly visible journals are Management Science, Operations Research, the European Journal of Operational Research, and Interfaces. Among the top 10 are such journals as Decision Sciences and the European Journal of Operational Research, which rank much higher on visibility than on quality. Nevertheless, there is a significant p < correlation of between quality ratings and visibility measures. In general, the higher quality journals are also the better known ones. The correlation between the visibility measures and five-year impact factors is only and is not statistically significant. Comparison of Ratings The usefulness of journal ratings should be related to their agreement across studies. Thus, I compare my quality and visibility ratings with quality and relevance ratings from three of the earlier survey studies. I also compare my 2003 five-year impact factors with citation measures from two of the earlier citation studies. Because analysts assessed somewhat different sets of journals in the various studies, I identified 23 journals that I rated in the present study that were rated in one or more of the other studies, listing them in alphabetical order and giving ratings and scores from the various studies for each journal (Table 7). I recomputed rankings for journals from each study to reflect the smaller sets presented. Management Science placed first in seven of the 11 rankings. The three cases in which it was not first or third are ratings measuring relevance to operations management. Operations Research placed second in four of the rankings, and its lowest rankings are also for relevance. Below the very top, ratings and derived rankings differ more over time and by the type of rating used. The derived rankings also differ because all the studies did not include all 23 journals. The greatest disparity in quality rankings between my study and the earlier opinion surveys seems to be in the lower rankings given to the Journal of Operations Management and Decision Sciences. I next computed correlations between all the pairs of ratings or scores (Table 8). I correlated ratings or scores rather than rankings, because ratings are the basic units of analysis and rankings are sensitive to small differences in scores. In the four quality ratings, the correlation coefficients between pairs of ratings are positive and very statistically significant. The quality ratings I obtained are the mostly highly correlated with those of Barman et al. (2001). My study s visibility measure is also highly and significantly correlated with the four quality ratings. The four quality ratings are also positively correlated with the citation scores, but the coefficients are generally smaller and the significance levels are lower than the correlations between quality ratings. The quality ratings for my study have the lowest correlation coefficients with the citation measures of any of the studies, and their correlation with Goh et al. s (1996) citation scores is not statistically significant. The impact factors in my study are positively correlated with other citation measures, but the correlation with Goh et al. s (1996) citation measures is not statistically significant. The relevance measures found by Barman et al. (1991, 2001) and Soteriou et al. (1999) are highly correlated with each other but are not significantly correlated with the other measures in my study and in other studies.

11 Interfaces 35(4), pp , 2005 INFORMS 333 Number of respondents Adjusted rankingnumber of quality over respondents Overall Quality Visibility rankingjournal both surveys rankingquality quality mean ranking 1 Management Science Operations Research European Journal of Operational Research Interfaces Naval Research Logistics IIE Transactions Decision Sciences Mathematics of Operations Research Manufacturing & Service Operations Management Production and Operations Management Operations Research Letters Journal of Operations Management Journal of the Operational Research Society Transportation Science Mathematical Programming Journal of the American Statistical Association # International Journal of Production Research International Journal of Operations and Production Management ## 19 Omega International Journal of Production Economics SIAM Review # Computers and Operations Research INFORMS Journal on Computing Production and Inventory Management Journal ## Networks Annals of Operations Research Computers and Industrial Engineering Journal of Business Logistics ## Decision Support Systems and Electronic Commerce Journal of Supply Chain Management ## International Journal of Flexible Manufacturing Systems ## International Journal of Operations and Quantitative Management ## 33 Mathematical and Computer Modelling Journal of Combinatorial Optimization ## Journal of Scheduling ## American Journal of Mathematical and Management Science Journal of Manufacturing Systems ## Journal of Heuristics ## Journal of Global Optimization ## Table 6: This table lists journals ranked by visibility, where visibility is the adjusted number of respondents who rated the journal. For journals that appeared in only one of the surveys, the adjusted number is double the actual number. The last two columns show the quality ratings and rankings for purposes of comparison. The symbols and their meanings follow: # this journal appeared only in the first survey, and ## this journal appeared only in the second survey.

12 334 Interfaces 35(4), pp , 2005 INFORMS Quality studies Relevance and visibility studies Citation studies Barman Soteriou Barman Barman Soteriou Barman My study s Goh et al. Vokurka My study s et al. (2001) et al. (1999) et al. (1991) My study s et al. (2001) et al. (1999) et al. (1991) year (1996) (1996) quality quality quality quality visibility relevance relevance relevance impact normalized total Journal mean 1 mean 2 mean 3 mean 2 score mean 2 mean 3 mean 2 factor citations citations Annals of Operations Research 2.97 (10) NR NR NR 71 (20) NR NR NR (19) (16) NR Computers and Industrial Engineering 4.46 (23) 5.46 (18) 4.71 (19) 5.09 (15) 71 (21) 5.46 (18) 5.03 (18) 5.14 (16) NR (15) 13 (18) Computers and Operations Research 4.05 (17) 5.14 (16) 5.54 (16) 4.85 (14) 88 (18) 5.33 (17) 5.35 (17) 5.34 (17) (10) (19) 21 (16) Decision Sciences 3.27 (13.5) 2.81 (4) 6.35 (8) 2.58 (2) 142 (7) 3.18 (7) 5.47 (16) 2.68 (5) (4) (7) 201 (4) European Journal of Operational 2.83 (9) 3.60 (8) 6.61 (7) 3.74 (9) 166 (3) 4.05 (12) 6.18 (8) 4.32 (11) (7) (9) 64 (9) Research IIE Transactions 2.44 (6) 3.18 (6) 6.13 (12) 2.87 (4) 142 (6) 3.63 (9) 6.06 (9) 3.33 (7) (11) (3) 125 (6) Interfaces 2.53 (7) 3.67 (9) 5.75 (15) 3.87 (10) 164 (4) 3.18 (6) 5.93 (10) 3.48 (8) (12) (13) 45 (10) International Journal of Operations 4.10 (19) 4.43 (13) 6.79 (4) 3.94 (11) 98 (15) 2.69 (3) 7.60 (2) 2.46 (3) NR (8) 36 (11) and Production Management International Journal of Production 4.06 (18) 4.51 (14) 6.17 (11) 3.25 (8) 96 (17) 3.73 (10) 6.66 (5) 2.49 (4) (18) NR NR Economics # International Journal of Production 3.88 (15) 3.82 (10) 6.74 (5) 2.94 (5) 105 (14) 3.07 (5) 6.73 (4) 2.34 (2) (8) (2) 229 (2) Research Journal of Manufacturing Systems 4.36 (20) NR NR NR 44 (23) NR NR NR (13) (12) 17 (17) Journal of Operations Management 3.02 (12) 2.66 (3) 6.86 (3) 2.60 (3) 122 (11) 2.12 (1) 7.62 (1) 1.68 (1) (2) (6) 152 (5) Journal of Supply Chain 3.90 (16) 5.38 (17) 4.77 (18) 5.27 (17) 62 (22) 3.97 (11) 5.68 (14) 3.91 (10) NR (17) 28 (14) Management ## Journal of the American Statistical 1.68 (4) NR NR NR 106 (13) NR NR NR (1) (18) NR Association Journal of the Operational Research 3.27 (13.5) 4.19 (11) 6.27 (9) 4.30 (12) 122 (12) 4.69 (16) 5.72 (13) 4.77 (15) (16) (10) 32 (13) Society Management Science 1.10 (1) 2.09 (1) 7.67 (1) 2.34 (1) 176 (1) 3.52 (8) 6.43 (7) 3.55 (9) (3) (1) 554 (1) Mathematics of Operations Research 1.41 (3) NR NR NR 138 (8) NR NR NR (5) NR 27 (15) Naval Research Logistics 2.38 (5) 3.26 (7) 6.19 (10) 3.23 (7) 149 (5) 4.29 (14) 5.54 (15) 4.34 (12) (15) (11) 66 (8) Omega 4.37 (21) 4.33 (12) 5.94 (14) 4.40 (13) 97 (16) 4.40 (15) 5.75 (12) 4.39 (13) (9) (14) 35 (12) Operations Research 1.12 (2) 2.58 (2) 7.56 (2) 3.16 (6) 169 (2) 4.2 (13) 5.92 (11) 4.59 (14) (6) (4) 225 (3) Operations Research Letters 2.65 (8) NR 6.04 (13) NR 125 (10) NR 4.80 (19) NR (14) (20) NR Production and Inventory 4.42 (22) 5.07 (15) 5.19 (17) 5.18 (16) 86 (19) 2.81 (4) 6.65 (6) 2.88 (6) NR (5) 80 (7) Management Journal Production and Operations 2.99 (11) 2.84 (5) 6.62 (6) NR 133 (9) 2.21 (2) 7.44 (3) NR (17) NR NR Management Total in category Table 7: This table compares ratings or scores of 23 OM-related journals that appeared in two or more of the studies (including mine) that rate OM and OM-related journals. The Saladin (1985) and Goh et al. (1997) studies are not included because they did not have a formal ranking. The table also does not include some of the new journals, such as Manufacturing & Service Operations Management, which ranked highly in Table 2. The journals are listed alphabetically. I show the mean rating or score and, in parentheses, the resulting ranking of the journal in each study. The symbols and their meanings follow: NR not rated in that study; # Journal of Manufacturing and Operations Management merged into this journal; ## formerly called Journal of Purchasing and Materials Management; 1 seven-point scale with lower number representing higher quality; 2 nine-point scale with lower number representing higher quality; and 3 nine-point scale with higher number representing higher quality.

13 Interfaces 35(4), pp , 2005 INFORMS 335 Quality studies Relevance and visibility studies Citation studies Barman Soteriou Barman Barman Soteriou Barman My Goh et al. Vokurka My et al. et al. et al. My et al. et al. et al. study s (1996) (1996) study s (2001) (1999) (1991) study s (2001) (1999) (1991) impact normalized total quality quality quality quality visibility relevance relevance relevance factors citations citations Quality studies My study s quality Barman et al. (2001) quality Soteriou et al. (1999) quality Barman et al. (1991) quality Relevance and visibility studies My study s visibility Barman et al. (2001) relevance Soteriou et al. (1999) relevance Barman et al. (1991) relevance Citation studies My study s impact factors Goh et al. (1996) normalized citations Vokurka (1996) total citations Table 8: This table shows the correlations between the various ratings or scores shown in Table 7. The numbers above the diagonal are the correlations between two sets of ratings. Signs have been adjusted to reflect the fact that in some cases the best ratings are low numbers and in other cases the best ratings are large numbers. The numbers below the diagonal represent the number of journals compared in the correlations. The symbols and their meanings follow: indicates the correlation is significant at the 0.05 level; indicates the correlation is significant at the 0.01 level; and indicates the correlation is significant at the level. Conclusions I surveyed faculty at top US business schools rather than members of professional organizations, targeting respondents who are likely to be academic leaders of operations management and those likely to serve as references for promotion and tenure decisions. A drawback of my study is that it excludes leaders in operations management at other fine business schools and in industrial engineering departments inside and outside the United States. Most researchers, including me, do not define quality, but Barman et al. (2001) tried to determine what are the most important factors affecting quality. Their respondents indicated that the most important criterion is methodological rigor of published works, followed by authors who publish in the journal, acceptance rate, and editor and editorial board members. As Barman et al. (2001) note, this leaves unanswered whether rigor is more closely associated with sophisticated mathematical and statistical models or with triangulation in methodologies (p. 381). The authors of the four studies of quality used somewhat different lists and numbers of journals and included journals that may not be relevant to operations management. I asked respondents to rate quality, and the other researchers asked them to rate quality with respect to POM articles. This distinction may account for some differences in ratings between my study and the others. Discipline-based journals, such as Mathematics of Operations Research, Mathematical Programming, and Journal of the American Statistical Association, are very highly rated in my study even though they are unlikely to be major targets for operations-management research. In contrast, the Academy of Management Journal and the Academy of Management Review, generally considered to be top management journals, ranked in the lower half of the quality ratings obtained by Barman et al. (2001), presumably because they publish very few operations management articles. A subset of the journals in the various studies that focus more on operations management tend to be similar in rankings, with a couple of exceptions.

14 336 Interfaces 35(4), pp , 2005 INFORMS I also studied citations for journals in operations management. Goh et al. (1996) and Vokurka (1996) developed their own citation numbers based on a limited number of journals; I relied on information from the online ISI Journal Citation Reports JCR for 2003 and earlier years. Although the citation scores are positively correlated with each other and with the quality ratings, they are not all statistically significant. In particular, the impact factors I calculated are not highly correlated with other measures, which were estimated in different ways and which represent different time periods. Now that JCR is increasing the number of journals relevant to operations management that it includes, further studies could determine whether impact factors are fairly stable over time and how closely they correlate with quality and relevance measures. Clearly, small differences in ratings can lead to very different rankings. Analysts use different sets of journals and different numbers of journals; all affect rankings. Instead of focusing on rank, we would do better to focus on ratings and loosely group journals based on those ratings. Appendix Survey Instrument Below is the instrument used in the second survey. The first instrument was nearly identicial, but the second asked for tenure status of respondents and included 11 journals that were not in the first survey instrument. The first survey instrument had two journals that were no longer published and two journals that were dropped in the second survey instrument Survey of OM/OR Journals Please indicate your primary research area(s) in column F: 1 Operations Management; 2 Operations Research; 3 Statistics; 4 Artificial Intelligence; 5 Decision Analysis; 6 Other (please specify). Please indicate your faculty rank in column F: 1 Full Professor; 2 Associate Professor; 3 Assistant Professor; 4 Other (please specify). Please indicate in column F whether you are: 1 Tenured; 2 Tenure Track; 3 Adjunct or Other. Please rate the journals below first for audience and then for quality. Audience (column B): AG Academic General; AS Academic Specialized; and P Practice. Quality on a 7 point scale (column C): 1 denotes A or the very top journals; 2 denotes A journals; 3 denotes B+ journals; 4 denotes B journals; 5 denotes B journals; 6 denotes C+ journals; 7 denotes C or lower. Leave blank if you do not know the journal. B C D E F G H I Number of Number of Audience Quality external internal Acceptance Journal ratingratingcirculation referees referees Frequency rate (%) Sources American Journal of Mathematical Quarterly Ulrich s and Management Sciences Annals of Operations Research 8 volumes per year Ulrich s Computers and Industrial Engineering 1,000 8 times a year Ulrich s Computers and Operations Research 1, times a year 40 Cabell s, Ulrich s Decision Sciences 3,001 4, Quarterly Cabell s Decision Support Systems and 8 times a year Ulrich s Electronic Commerce formerly Decision Support Systems European Journal of Operational 1,001 2, times a year Cabell s, Ulrich s Research IIE Transactions Monthly Ulrich s INFORMS Journal on Computing 1,125 Quarterly INFORMS Interfaces 4, Bimonthly Cabell s, Ulrich s International Journal of Flexible 10,000 25, Quarterly Cabell s, Ulrich s Manufacturing Systems International Journal of Operations 2 0 Monthly Cabell s, Ulrich s and Production Management International Journal of Operations < times a year Cabell s, Ulrich s and Quantitative Management International Journal of times a year 45 Cabell s, Ulrich s Production Economics

15 Interfaces 35(4), pp , 2005 INFORMS 337 B C D E F G H I Number of Number of Audience Quality external internal Acceptance Journal ratingratingcirculation referees referees Frequency rate (%) Sources International Journal of times a year 50 Cabell s, Ulrich s Production Research Journal of Business Logistics 15, times a year Cabell s, Ulrich s Journal of Combinatorial Quarterly Ulrich s Optimization Journal of Global Optimization Monthly Ulrich s Journal of Heuristics Bi-monthly Ulrich s Journal of Manufacturing Systems 1,000 Bi-monthly Ulrich s Journal of Operations Management 1, times a year Cabell s, Ulrich s Journal of Scheduling 6 times a year Ulrich s Journal of Supply Chain Management 3,000 Quarterly Ulrich s (formerly J. of Purchasing & MM) Journal of the Operational Research Monthly Ulrich s Society Management Science 6,000 2 or more 0 Monthly Cabell s, Ulrich s, INFORMS Manufacturing & Service 2 2 Quarterly INFORMS Operations Management Mathematical and Computer 1,001 2, times a year Cabell s, Ulrich s Modelling Mathematics of Operations Research 3,001 4, Quarterly Cabell s, Ulrich s Mathematical Programming 9 times a year Ulrich s Naval Research Logistics 1,000 8 times a year Ulrich s (formerly NRLQ) Networks 8 times a year Ulrich s Omega 1, Bimonthly Cabell s, Ulrich s Operations Research 10, Bimonthly Cabell s Operations Research Letters 10 times a year Ulrich s Production and Inventory 72,000 Quarterly Ulrich s Management Journal Production and Operations 1, Quarterly Cabell s, Ulrich s Management Transportation Science 1, Quarterly Cabell s, Ulrich s Other journals (please specify) Information on journals, where available, comes from Cabell s Directory of Publishing Opportunities in Management and Marketing, 7th ed., ; Ulrich s Periodicals Directory On-Line; and occasionally from a journal Web site Survey of OM/OR Journals (continued). Business Schools Surveyed Carnegie Mellon University (Tepper) Columbia University Cornell University (Johnson) Dartmouth College (Tuck) Duke University (Fuqua) Emory University (Goizueta) Georgetown University (McDonough) Harvard University Indiana University Bloomington (Kelley) Massachusetts Institute of Technology (Sloan) Michigan State University (Broad) New York University (Stern) Northwestern University (Kellogg) Ohio State University (Fisher) Purdue University West Lafayette (Krannert) Stanford University University of California Berkeley (Haas) University of California Los Angeles (Anderson) University of Chicago University of Michigan Ann Arbor (Ross) University of Minnesota Twin Cities (Carlson) University of North Carolina (Kenan Flagler) University of Pennsylvania (Wharton) University of Rochester (Simon) University of Southern California (Marshall) University of Texas Austin (McCombs) University of Virginia (Darden) Vanderbilt University (Owen) Washington University at St. Louis (Olin) Yale University I list the business schools that I included in my surveys. Means the school was included only in the 2000 survey and means the school was included only in the 2002 survey. Acknowledgments I thank all the faculty members of the operations, decision science, and artificial intelligence interest group of the Katz

16 338 Interfaces 35(4), pp , 2005 INFORMS Graduate School of Business, University of Pittsburgh, for developing the questionnaire and the list of journals. I especially thank Prakash Mirchandani, Richard Wendell, and two anonymous reviewers for their very helpful comments on earlier drafts of this paper. References Barman, Samir, Mark D. Hanna, R. Lawrence LaForge Perceived relevance and quality of POM journals: A decade later. J. Oper. Management 19(3) Barman, Samir, Richard J. Tersine, M. Ronald Buckley An empirical assessment of the perceived relevance and quality of POM-related journals by academicians. J. Oper. Management 10(2) Cabell, D. W. E., ed Cabell s Directory of Publishing Opportunities in Management and Marketing, 7th ed Cabell Publishing, Beaumont, TX. Goh, C. H., C. W. Holsapple, L. E. Johnson, J. Tanner An empirical assessment of influences on POM research. Omega 24(3) Goh, Chon-Huat, Clyde W. Holsapple, Linda Ellis Johnson, John R. Tanner Evaluating and classifying POM journals. J. Oper. Management 15(2) Gupta, Uma G Using citation analysis to explore the intellectual base, knowledge dissemination, and research impact of Interfaces ( ). Interfaces 27(2) Journal Citation Reports. Science and social science editions to Thomson ISI Web of Knowledge. Retrieved October 2004 from jcr01.isiknowledge.com/jcr/jcr?rq=home. Saladin, Brooke Operations management research: Where should we publish? Oper. Management Rev. 3(4) 3 9. Science Citation Index Institute for Scientific Information, Philadelphia, PA. Social Science Citation Index Institute for Scientific Information, Philadelphia, PA. Soteriou, Andreas C., George C. Hadjinicola, Kalia Patsia Assessing production and operations management related journals: The European perspective. J. Oper. Management 17(2) Ulrich s Periodicals Directory On-Line R. R. Bowker LLC, New Providence, NJ. Retrieved May 2002 from www. ulbrichsweb.com. U.S. News and World Report Best Graduate Schools, 2001 ed. U.S. News and World Report, Inc., Washington, DC, 23. U.S. News and World Report Schools of business; The top schools. (April 15) 56. Vokurka, R. J The relative importance of journals used in operations management research: A citation analysis. J. Oper. Management 14(4)