Citation Benchmarks for Articles Published by Australian Marketing Academics. Geoffrey N. Soutar, University of Western Australia.

Transcription

1 Citation Benchmarks for Articles Published by Australian Marketing Academics Geoffrey N. Soutar, University of Western Australia Abstract Research quality is becoming increasingly important in the Australian university system and journal quality and citation rates are both seen as reasonable quality indicators. The present study used the citations received by articles published in a set of high and reasonable quality marketing journals in 2001, which is the first year of the intended Australian RQF exercise, to develop influence benchmarks. The analysis suggested a high quality benchmark of 16 citations and a reasonable quality benchmark of 6 citations. The influence of a sample of Australian marketing scholars recent publications was examined by comparing their citations with these benchmarks. Marketing scholars have produced a number of very influential papers in the past six years and the scholars who have published such articles can be found across a range of Australia s universities. Introduction The introduction of research quality frameworks (RQFs) in the education sectors in a number of countries (e.g. New Zealand) has seen an increased interest in issues of research quality. Australia is no exception to this trend as the Federal Government will undertake an initial evaluation of the quality of the research produced by the country s universities in 2008, including research produced by marketing academics. The Australian RQF process requires universities to define research active staff and to provide four pieces of research output produced from 2001 to 2006 for each of these academics to enable the evaluation to be made. While the Labor Opposition has indicated that it may not use the same approach, there is little doubt that some type of quality evaluation will be undertaken in the near future, even if there was a change of government. What is also clear is that however the evaluations are made, they will be influenced by the quality of the publication outlets (e.g. journals and publishers) and the influence the research outputs have had, as measured by the number of times they have been cited. The former issue (journal quality) has long been a concern to business academics and has been examined by a number of people in recent years; in particular by a number of Australian-based business academics (e.g. Sullivan Mort et al., 2005; Soutar et al., 2005; Polonsky et al., 1999). Some universities (e.g. the University of Queensland and the University of Melbourne) have even developed internal quality lists with a ranking system to provide guidance to staff members. From this activity, a number of very high quality marketing journals, which are shown in alphabetical order in Table 1, has emerged that most academics in the field would accept as reasonable. It seems likely Australian academics who have published in such journals will be seen as producing quality work. However, the issue of whether Australian business-related research has been influential, whether published in such journals or in other outlets, needs further investigation and the present paper was undertaken to examine this issue. In particular, evaluators might ask what the citation rates are for papers published in the very 3515

2 high quality journal set, as this would provide a benchmark against which influence could be assessed and it was this issue that was examined in the present study. ISI citations, obtained from Thomson Scientific s database of over 8,600 peer-reviewed journals ( are often used to measure this aspect of research performance (King, 2004), but there are issues in using this database as different fields have different numbers of journals included. Thus, while economics has 164 indexed journals (Yeung, 2002), marketing has only 23 (Baumgartner & Pieters, 2003). This has led some to argue that the recently launched Google Scholar (GS) may be a better source for citation data as it has a much wider footprint because it searches databases from a very wide range of academic publishers, professional societies, preprint repositories, universities and other scholarly organizations (Murphy et al., 2007). It is also apparent that GS provides a better indicator of actual citations for recent articles (Bakkalbasi et al., 2006). As the RQF will assess output produced from 2001 to 2006, GS seems likely to be an appropriate source of such data. GS s advanced search capability provides a way to determine the number of times an article has been cited across this range of online sources. Harzing s (2007) Publish or Perish software enables GS results to be obtained reasonably easily and was used to obtain the citations obtained by all of the articles published in the 12 very high quality journals shown in Table 1 that were published in As noted earlier, this is the earliest year in which research outputs will be considered in the Australian RQF exercise. Articles published in 2001 are likely to have more citations than articles published later and these citation counts should provide a useful benchmark against which an article s influence can be assessed. The results of this analysis are discussed in the next section. A High Quality Benchmark The Results As was noted earlier, the GS citations of each of the articles published in the 2001 issues of the 12 journals shown in Table 1 were obtained. However, the process revealed a significant issue as a number of the citations were incorrect, which was expected as prior research has noted such a tendency (Harzing, 2002). As examples, some citations had authors in incorrect order, others seemed to have titles that suited the citing researchers own needs and journals and years of publication were occasionally wrong. Consequently, every citation permutation was checked and incorrect citations were added to the correct article, which was determined by a check of the relevant table of contents. In some cases this process more than doubled the number of citations attached to an article. The analysis also made clear a further issue as the number of articles published by each of the 12 journals in 2001 varied considerably (ranging from 17 for the International Journal of Research in Marketing to 65 for the European Journal of Marketing), which suggests corrections might be needed if a journal level analysis was being undertaken with the data. However, as the analysis was undertaken at an article level, this was not an issue in the present case. Twelve very high quality management journals were also analysed, although not discussed here. Considerably more articles were published in these journals (625) than were published in the twelve marketing journals (392), which has some implications for cross-discipline comparisons. The citation distributions were calculated for each journal, and for the total set. In each case, as expected, article citations were often skewed. As the citations did not follow a normal 3516

3 distribution in a number of cases, other distributions were estimated. A generalised Pareto distribution fitted the various non-normal data sets reasonably well, suggesting that, in these cases, relatively few articles were highly cited. Given this result, the median scores, rather than the mean scores, both of which are shown in the Table 1, should be used as a measure of central tendency. The 75, 90 and 95 percentiles, as well as the maximum number of citations obtained by an article published in the relevant journal, are also shown in the Table 1. As would be expected given the comments about the generalised Pareto distribution being appropriate, the medians were often considerably lower than the means and the higher percentiles were often very high. Clearly, some 2001 articles have already had considerable influence and such citation rates should be seen as strong evidence of research quality. Table 1: Citations by Journal * Marketing Journal Median Journal of Marketing Journal of Marketing Research Journal of Retailing Journal of Consumer Research Marketing Science International Journal of Research in Marketing Journal of Advertising Research Journal of the Academy of Marketing Science Journal of Advertising European Journal of Marketing Journal of Service Research Marketing Letters Overall * The median and percentile figures are the number of citations for an article in each journal It is also apparent there is considerable disparity in the influence of even the very high quality journals. Thus, Journal of Marketing articles have been more influential than Marketing Letters articles. Interestingly, citations were higher for the management articles than for the marketing articles, but whether this is a real difference or merely a function of the number of articles published in each area cannot be determined by the present data set. It does, however, seems undesirable to assume citation rates in different business areas can be compared as a Mann-Whitney U test that on the two data sets was significant well beyond the one percent level (z=3.64), suggesting real differences. As was noted earlier, the influence of the major marketing journals was clear. Interestingly, while the Journal of Marketing, the Journal of Marketing Research and the Journal of Consumer Research are clearly in this top group, as would be expected, the Journal of Retailing also appears to be about as influential, at least in terms of the papers that were published in Indeed, these four were the only marketing journals for which the median citation count exceeded 20. The median number of citations over the twelve marketing journals was 16, suggesting any paper published during the RQF period that achieved such a score would provide strong evidence of influence, especially as the benchmark was developed from the elite group of marketing journals. 3517

4 A Reasonable Quality Benchmark To further examine this issue, the number of citations obtained by articles published in less well regarded marketing journals was also examined. In this case, six marketing journals that were not ranked at the top of the Sullivan-Mort et al. (2005) list, were chosen. As can be seen in Table 2, the citation rates for these journals were considerably lower than for the very high quality journals, which was confirmed by a Mann-Whitney U test that was significant well beyond the one percent level (z=7.45). The high quality benchmark may be too high and a more reasonable hurdle may be the median of the citations obtained by the articles published in the journals contained in Table 2. Table 2: The Second List * Journal Median Max. Journal of Business & Industrial Marketing Journal of Consumer Marketing Journal of Interactive Marketing Journal of Marketing Theory & Practice Journal of Services Marketing Psychology & Marketing Second List Combined * The median and percentile figures are the number of citations for an article in each journal Do Australian Marketing Academics meet these Benchmarks? To see whether either benchmark (i.e. the median number of citations from the high and reasonable quality journals, which were 16 citations and 6 citations respectively) was reasonable, the citations of the research published during the 2001 to 2006 period by fifty senior Australian marketing scholars from across a range of Australian universities were examined to see if a reasonable number would be classified as influential. In order to ensure there was an appropriate range of universities, no more than four scholars were chosen from any one institution, although some institutions were not included as they did not have a senior marketing scholar. As the RQF will require four outputs to be provided, the four outputs with the most citations were used in the present analysis and the highest citation count and the average number of citations were used as influence measures. The results obtained in this case are shown in Table 3 and compared across the high quality and reasonable benchmarks that were suggested. As can be seen from the Table, less than a third of the scholars would meet the high quality median benchmark and none would meet a ninety percentile benchmark, but almost three quarters would meet the reasonable median benchmark, while 14% would meet a ninety percentile reasonable benchmark. Interestingly, the seven scholars who met that benchmark came from five different universities, suggesting influence among Australia s marketing academics is spread across a number of institutions. Information on the maximum number of citations obtained from a single article is also provided in the Table. As would be expected, many more scholars meet the benchmark when their only their best article is considered. Indeed, a majority of the scholars met even the high quality median benchmark in this case, while ten percent had published an article that met the 95 percentile benchmark and twenty precent met the 95 percentile benchmark for the 3518

5 reasonable quality benchmark. Interestingly, one article has received more than 100 citations, while another has received more than 90 citations. Table 3:A Comparison of Fifty Marketing Scholars Citations with the Benchmarks. Benchmark Mean Median 75 Percentile 90 Percentile 95 Percentile High Quality Proportion meeting Benchmark Proportion (with Best Article) Reasonable Proportion meeting Benchmark Proportion (with Best Article) Averaged Citations Conclusions The present study examined the citations obtained by articles published in a number of highly regarded and reasonably regarded journals in 2001, the first year of the intended Australian RQF exercise. The data suggested citation distributions are skewed, as relatively few articles are highly cited, at least in the five years after their publication. However, it was also clear that some marketing articles have become very influential, with a number being cited more than 200 times. Two benchmarks were suggested (16 citations for a high quality benchmark and 6 citations for a reasonable quality benchmark) and an examination of the citations obtained by articles published by 50 senior Australian marketing academics during from 2001 to 2006 found a small number met the high quality test when their best four articles were considered, although considerably more met the reasonable quality test. When only their best article was considered, it could be seen that many of the Australian scholars have produced articles in recent years that have influenced a considerable number of other marketing scholars. While further research is needed to correct for such factors as the recency of publication, it seems that at least some Australian marketing scholars have had an influence in the present century and that Australian research in marketing is making a contribution that has had an impact around the world. 3519

6 References Bakkalbasi, N., Bauer, K., Glover, J., Wang, L., Three options for citation tracking: Google Scholar, Scopus and Web of Science. Biomedical Digital Libraries, 3 (7), Baumgartner, H., Pieters, R., The structural influence of marketing journals: A citation Analysis of the Discipline and its sub-areas over time. Journal of Marketing, 67 (2), Harzing, A. W., Are our referencing errors undermining our scholarship and credibility? The case of expatriate failure rates. Journal of Organizational Behavior 23 (1), Harzing, A. W., Publish or Perish Version (Software). King, D. A., The scientific impact of nations. Nature, 430 (6997), Murphy, J., Soutar, G. N., Lee, R., Online visibility: A complementary way to rank marketing journals. Paper presented at the AMA Summer Conference. Polonsky, M. J., Jones, G., Kearsley, M. J., Accessibility: An alternative method of ranking marketing journals? Journal of Marketing Education, 21 (3), Soutar, G. N., Fulop, L., Brocklesby J., Senior management academics perceptions of management journals, A Report Prepared for the Australian and New Zealand Academy of Management. Sullivan-Mort, G., McColl-Kennedy, J. R., Kiel, G., Soutar, G. N., Perceptions of marketing journals by senior academics in Australia and New Zealand. Australasian Marketing Journal, 12 (2), Yeung, T. A., Deciphering citations. Environment and Planning A, 34 (12),