Current Institutional Trends in Research Productivity in Counseling Psychology Journals

Current Institutional Trends in Research Productivity in Counseling Psychology Journals 10.1177/0011000004274130 THE Diegelman COUNSELING et al. / COUNSELING PSYCHOLOGIST PSYCHOLOGY / May 2005RESEARCH PRODUCTIVITY Nathan M. Diegelman Rachel A. Uffelman Kimberly S. Wagner Sally A. Diegelman University of Akron This study investigated institutional publication activity in counseling psychology journals for the 10-year period from 1993 to 2002. Four journals reported by counseling psychology training directors as prime publication outlets for the field of counseling psychology were used: Journal of Counseling Psychology, The Counseling Psychologist, Journal of Vocational Behavior, and Journal of Counseling and Development. Comparisons with the most recent previous ranking study revealed that institutions in the top 20 remained fairly stable. However, 13 of the top 40 institutions ranked in this study were not ranked in the previous top 40. Implications for the field are discussed, and suggestions for how to improve publication activity trend analyses are made. The quality of counseling psychology training has been assessed continuously throughout the history of the field. Initial efforts to evaluate graduate psychology programs often focused on reputation (e.g., Roose & Anderson, 1970). Because of concerns about the objectivity of using reputation scores in ranking psychology programs, however, subsequent examinations focused on published research productivity (Cox & Catt, 1977; Thoreson, Cox, & Krauskopf, 1975). Although research productivity of counseling psychology programs has been evaluated using a wide range of criteria, including individual author productivity and impact (e.g., Goodyear, Abadie, & Walsh, 1983; Heesacker, Heppner, & Rogers, 1982; Howard & Curtin, 1993), most recent evaluations have also focused on research productivity at the institutional level to provide a broader overall evaluation (e.g., Buboltz, Miller, & Williams, 1999; Delgado & Howard, 1994; Howard, 1983; Perez, Constantine, & Gerard, 2000; Williams & Buboltz, 1999). An earlier draft of this research was presented at the Great Lakes Conference April 6 to 7, 2001, at the University of Akron. The second and third authors provided equal contributions to this study; in this case, author order was determined alphabetically. The authors wish to thank Drs. Linda M. Subich and David M. Tokar for ideas and comments on earlier drafts of the article. Correspondence concerning this research should be addressed to Nathan M. Diegelman, Department of Psychology, University of Akron, Akron, Ohio 44325-4301; phone: (330) 972-7280; fax: (330) 972-5174; e-mail: BL891@bfn.org. THE COUNSELING PSYCHOLOGIST, Vol. 33 No. 3, May 2005 327-339 DOI: 10.1177/0011000004274130 2005 by the Society of Counseling Psychology 327

328 THE COUNSELING PSYCHOLOGIST / May 2005 Initial examinations of institutional-level research productivity in counseling psychology tended to focus solely on articles published in the Journal of Counseling Psychology (JCP) (Bohn, 1966; Goodstein, 1963; Howard, 1983; Katz & Brophy, 1975; Tinsley & Tinsley, 1979). To more completely portray counseling psychology research activity, Howard (1983) selected four other journals based on the nominations of 10 unidentified counseling psychologists: The Counseling Psychologist (TCP), Journal of Vocational Behavior (JVB), Personnel and Guidance Journal (now Journal of Counseling and Development; JCD), and Journal of Consulting and Clinical Psychology (JCCP). The most recent multijournal ranking by Delgado and Howard (1994) replicated Howard (1983) with the inclusion of the same journals (JCP, TCP, JVB, JCD, and JCCP) for the 10 years from 1983 through 1992. Analysis of the interrelations among research productivity in these five journals revealed that while TCP, JCD, and JVB were at least moderately interrelated to JCP, the relation for JCCP was weaker. In addition, JCCP had the weakest loading of all five journals to the derived overall productivity index. Despite this, all five journals were retained by Delgado and Howard to ensure comparability with the previous ranking of general psychology programs conducted by Cox and Catt (1977). Although each journal considered by Delgado and Howard (1994) represents a noteworthy avenue for scientific contributions, the continued relevance of each to the field of counseling psychology has not been reassessed. First, principal research outlets for a field, including counseling psychology, may change throughout time. Second, it is unclear who comprised Howard s (1983) sample of counseling psychologists, and the small size of the sample is a concern. The dimension(s) through which the journals were chosen in Howard s sample, including the specific instructions provided or questions asked, were also not provided. Incorporating a larger sample of clearly identified counseling psychologists, whose judgments will be viewed as credible, is one way to rectify these limitations. By identifying and using a more clearly defined and consistent subset, future investigations of institutional publication productivity would be enhanced by ensuring that current outlets for counseling psychology are included. Such an approach would also ensure consistency with past efforts and allow for greater strength to be placed in making comparisons across time. Toward this end, counseling psychologists most likely to be familiar with the status of contemporary publication outlets are training directors of counseling psychology programs. Training directors have substantial roles in training future counseling psychologists and can hold great influence over the current and future direction of the field. In addition, training directors typically are in senior positions in which their responsibilities can include

Diegelman et al. / COUNSELING PSYCHOLOGY RESEARCH PRODUCTIVITY 329 mentoring junior faculty and addressing issues related to promotion and tenure. Consistent with these foci, training directors as a group are likely tobe knowledgeable about research outlets in which faculty and students are encouraged to publish outlets that arguably may best reflect the current focus of the field. This study, then, had two purposes. First, provide more current information about valued research outlets within the field. Consequently, this study surveyed training directors of counseling psychology programs. Second, and foremost, provide current rankings of institutional research productivity in counseling psychology based on the number of publications in the nominated journals. Journal Evaluation METHOD To ascertain which journals most represent research productivity relevant to counseling psychology, training directors of all counseling psychology programs were surveyed via e-mail through the Council of Counseling Psychology Training Programs Listserv, which contains continually updated e-mail addresses of the current training directors of counseling psychology programs accredited by the American Psychological Association. In the e-mail, training directors were asked, What specific journals are faculty and/or graduate students encouraged to publish in? That is, does your department or program reinforce your publishing in specific journals? After responses were obtained from the first e-mail, a second attempt, to improve the response rate, was conducted roughly 2 months later. Although the response rates for the first attempt (18%) and second attempt (15%) were somewhat low, responses were eventually obtained from 26 of 77 directors surveyed, constituting a 34% response rate. Because of the diminishing response rate and the consistency in the journal rankings obtained, we decided to stop the nomination process at that point. The goal of the a priori decision to examine the top four journals nominated by the training directors was to focus more precisely on research most representative of the field. Specifically, we hoped that limiting the review to four journals most representative of the field would lead to a greater likelihood of capturing more research productivity of faculty and students in counseling psychology programs rather than programs in related fields. This decision was also based in part on the fact that the loadings for the first four journals of Delgado and Howard s (1994) list were the strongest and that the loading for the fifth journal (JCCP) was marginal. The top four journal

330 THE COUNSELING PSYCHOLOGIST / May 2005 endorsements were JCP (14 endorsements), TCP (12 endorsements), JCD (6 endorsements), and JVB (5 endorsements). Note that, except for JCCP, these four journals were the same journals incorporated by Delgado and Howard. In the current survey, JCCP only received one endorsement. Other journals mentioned by training directors included American Psychologist, Professional Psychology: Research & Practice, Psychological Science, Psychological Bulletin, Psychotherapy, Journal of Multicultural Counseling & Development, and Annual Review of Psychology. Most of these journals received only single nominations (e.g., Journal of Multicultural Counseling & Development), and none received more than two nominations. Thus, they did not warrant inclusion in the final rankings. In addition, rather than naming specific journals, some training directors stated that refereed journals, or those published by the American Psychological Association, were preferred. Institutional Ranking Procedure The 10-year period (1993 to 2002) available to the authors since the last review by Delgado and Howard (1994) was drawn on to obtain estimates reflective of the most current trends of counseling psychology research productivity and to provide continuity with the last published study. Authorship credit was derived and assigned using a data coding and scoring method analogous to that of Delgado and Howard. Specifically, a credit score of 1.0 was assigned to full-length articles, and a score of.50 was given for comments, reactions, and brief reports. Editor comments and proceedings from conference and committee meetings were excluded. In cases of multiple authorship, credit was partitioned according to the equation provided in Howard, Cole, and Maxwell (1987) and used in Delgado and Howard. For example, in a full-length article with three authors, the first author was given a weighting of.50, the second.30, and the third.20. Similarly, in a brief report with three authors, the credit scores would be halved (i.e.,.25,.15, and.10). The specific computational formula used to partition credit among multiple authors is provided in Howard et al. (1987). To compute overall institutional productivity scores, individual productivity scores were calculated separately for each journal and then summed to provide an overall composite score. Institutions were subsequently assigned a rank based on the obtained overall score. This ranking system differs somewhat from the work by Delgado and Howard (1994) in that institution scores are based on the summed scores for each individual journal as opposed to productivity factor scores. The decision not to base the current rankings on factor scores was based on prior critiques concerning numerous problems with using factor productivity scores (e.g., Hanish et al., 1995). In so doing, rankings for the current study were derived solely from numerical scores

Diegelman et al. / COUNSELING PSYCHOLOGY RESEARCH PRODUCTIVITY 331 based on summative research output and do not give stronger weights to journals that are lower on the rank list (e.g., JCD) provided by training directors. Nevertheless, comparisons of the institutional ranks obtained in this study with those provided by Delgado and Howard (1994) may be hindered by the incorporation of a different set of journals and different rankings methodology. Thus, the data provided in Delgado and Howard were reanalyzed using the methods incorporated in this study, and the institutional rankings were subsequently recalculated. Specifically, the institutional productivity scores for JCP, TCP, JVB, and JCD provided by Delgado and Howard were summed to derive overall productivity scores, as opposed to overall productivity factor scores. These new rankings, along with the original ranks provided by Delgado and Howard based on the productivity factor scores, are provided in the current study to facilitate cross-study comparisons. Finally, citation impact scores were also identified for each journal and included for further interpretation. Overall scores were computed by averaging the available citation impact factor scores for each journal to the other social science journals. Higher scores indicate higher rates of citation of the articles published in any one journal among articles published in different journals within a field (and thus may suggest higher journal prominence). These data were retrieved from the Thomson/ISI Web of Science database (http://www.isinet.com/ products/citation/wos/). RESULTS Rankings for each institution based on publication activity are provided in Table 1. In Table 1, the first two columns denote the institution and rank for the current study. The third column lists the original ranks provided by Delgado and Howard (1994) based on productivity factor scores. The fourth column provides the adjusted past ranks derived from the data provided by Delgado and Howard (1994). The overall institutional productivity totals for the current study, column five, were computed by summing the individual productivity scores for JCP, TCP, JVB, and JCD (the scores for which are provided in the remaining columns). The intercorrelations of the overall institutional productivity scores for each journal, and the relations of those scores to total score and overall institutional rank, are then presented in Table 2. Table 2 also lists citation impact data for each journal. Changes in the Delgado and Howard (1994) ranks for some institutions were based on the use of either the productivity factor scores or the revised summation scores. Overall, however, the obtained correlation between the ranks for both methods was significant (r =.83, p <.000). Thus, the overall

TABLE 1: Current and Past Institutional Rankings Based on Publication Activity in Selected Journals Current Prior Prior Total JCP TCP JVB JCD Institution Rank Rank Rank-R Score Score Score Score Score University of Missouri, Columbia 1 1 2 85.49 42.44 10.23 8.13 24.69 University of Maryland, College Park 2 2 1 80.77 46.83 11.61 10.83 11.50 University of Akron 3 18 21 39.42 10.51 6.03 13.99 8.89 State University of New York, Albany 4 4 3 36.01 14.11 8.90 5.50 7.50 Iowa State University 5 29 16 33.74 13.00 3.34 14.90 2.50 Arizona State University 6 9 14 33.19 9.33 7.58 5.95 10.33 University of Minnesota, Twin Cities 7 8 5 33.04 5.30 5.41 16.08 6.25 University of Illinois, Urbana-Champaign 8 12 7 32.16 16.60 0.87 12.03 2.66 Ohio State University 9 3 4 31.83 6.43 2.96 12.73 9.71 Pennsylvania State University 10 14 12 31.17 7.41 4.05 11.92 7.79 Ball State University 11 NR NR 29.38 5.66 4.77 1.50 17.45 Michigan State University 12 30 18 27.64 8.59 3.36 6.07 9.62 Virginia Commonwealth University 13 10 9 25.69 14.70 4.84 2.62 3.53 Southern Illinois University, Carbondale 14 15 6 24.07 8.83 1.63 9.81 3.80 University of Iowa 15 5 8 21.46 6.75 7.61 1.83 5.27 University of Florida 16 17 15 20.13 4.91 6.35 2.50 6.37 University of California, Santa Barbara a 17 7 11 20.10 8.88 6.56 4.33 0.33 Indiana University, Bloomington 18 NR NR 19.59 1.67 4.51 1.34 12.07 University of Southern Mississippi 19 NR NR 19.57 5.33 5.73 0.00 8.51 University of North Carolina, Greensboro b 20 NR NR 19.51 1.00 0.50 1.00 17.01 University of Notre Dame 21 11 17 19.41 4.50 10.53 2.70 1.68 University of Georgia 22 25 25 18.84 1.50 0.12 5.18 12.04 University of Wisconsin, Madison 23 36 32 18.74 6.75 4.78 0.00 7.21 University of North Texas 24 6 10 17.83 3.40 6.33 2.35 5.75 Loyola University of Chicago 25 NR NR 16.19 6.76 3.72 4.71 1.00 332

Teacher s College, Columbia University 26 32 30 16.11 4.11 5.83 2.17 4.00 Texas Tech University 27 NR NR 16.03 6.25 3.00 2.00 4.78 Boston College 28 NR NR 16.00 7.00 4.51 1.33 3.16 University of Wisconsin, Milwaukee 29 NR NR 15.45 5.80 3.00 4.62 2.03 University of Oregon 30 31 23 14.63 8.30 3.54 0.33 2.46 Lehigh University 31 NR NR 14.50 5.58 0.70 1.18 7.04 University of Southern California 32 33 28 14.41 6.32 4.98 0.00 3.11 University of Oklahoma 33 NR NR 13.22 0.83 3.91 2.86 5.62 University of British Columbia 34 22 26 13.11 6.12 0.00 1.00 5.99 Colorado State University 35 20 22 12.94 7.00 1.21 3.55 1.18 University of Nebraska, Lincoln 36 16 13 12.34 4.40 3.50 1.00 3.44 Georgia State University 37 NR NR 12.26 0.42 2.00 1.44 8.40 Hebrew University of Jerusalem b 38 NR NR 11.30 6.10 0.00 3.20 2.00 Illinois State University b 39 NR NR 11.17 7.67 0.83 1.00 1.67 University of Kansas 40 24 19 10.59 2.97 2.12 3.80 1.70 Other academic institutions 911.82 134.70 99.83 237.80 439.49 Nonacademic settings 124.12 12.39 16.14 24.38 71.21 Total 1994.97 487.15 287.42 449.66 770.74 NOTE: Prior rank = Delgado and Howard (1994) rank; prior rank-r = recalculated Delgado and Howard (1994) rank; JCP = Journal of Counseling Psychology; TCP = The Counseling Psychologist; JVB = Journal of Vocational Behavior; JCD = Journal of Counseling and Development; NR = not ranked. Institutional names that are not currently ranked but appear in Delgado and Howard (1994) include the following: University of California, Irvine; University of North Carolina, Chapel Hill; Stanford University; University of California, Los Angeles; University of Texas, Austin; Kent State University; University of Washington; University of Denver; University of Tennessee, Knoxville; Kansas State University; University of Western Ontario; University of Utah; and Syracuse University. University of North Texas changed its name in 1988 and was listed as North Texas State University in Delgado and Howard. Teacher s College, Columbia University changed its name in 1999 and was listed as Columbia University in Delgado and Howard. a. Institutions with combined programs. b. Institutions that do not house counseling psychology programs accredited by the American Psychological Association. 333

334 THE COUNSELING PSYCHOLOGIST / May 2005 TABLE 2: Journal Productivity Score Intercorrelations and Citation Impact Factors 1 2 3 4 5 6 1. JCP 1.00 2. TCP.54* 1.00 3. JVB.39*.14 1.00 4. JCD.37*.25.10 1.00 5. Total.91*.61*.57*.61* 1.00 6. Rank.54*.56*.68*.49*.77* 1.00 Overall citation impact factor 1.45 1.16 1.67 0.68 NOTE: JCP = Journal of Counseling Psychology; TCP = The Counseling Psychologist; JVB = Journal of Vocational Behavior; JCD = Journal of Counseling and Development; Total = total score; rank = institutional rank. Citation impact factor scores for these journals to other social science journals are based on data obtained from the Thomson/ISI Web of Science database (http:// www.isinet.com/products/citation/wos/) for 1998 through 2002. *p <.01. ranks based on both methods appeared to evidence at least a moderate level of correspondence. Considerable stability within the top 20 rankings can be seen between the current rankings and the original rankings of Delgado and Howard (1994); 14 of the current top 20 institutions were also ranked in the top 20 of Delgado and Howard. University of Maryland, College Park, and University of Missouri, Columbia, remained leaders in counseling psychology research productivity, and notable gains were evidenced for the University of Akron; Iowa State University; Ball State University; Michigan State University; University of Southern Mississippi; and University of Indiana, Bloomington, among others. Further variability was also found, such that 27 of the top 40 institutions had a net change of more than five places in rank order when compared with the original rankings of Delgado and Howard (1994). Similarly, 27 of the top 40 institutions had a net change of more than five places in rank order when compared with the revised Delgado and Howard rankings. Most of these changes (17 using the original rankings, 16 using the revised rankings) occurred among the institutions ranked 21 to 40, suggesting generally greater stability for those institutions occupying the upper half of the list. Note, however, that 13 of the current top 40 institutions were not found in the original ranking by Delgado and Howard (1994). Although this difference may be related to significant changes in research productivity, most of these 13 institutions were found in the lower half of the current list. The individual differences in productivity scores among institutions are much more minimal among these 13 than among those ranked higher in the list, consistent with the greater variability among the lower half of the list.

Diegelman et al. / COUNSELING PSYCHOLOGY RESEARCH PRODUCTIVITY 335 Three institutions without counseling psychology programs accredited by the American Psychological Association (University of North Carolina, Greensboro; Hebrew University of Jerusalem; and Illinois State University) were ranked within the current top 40. None of these institutions were included in the Delgado and Howard (1994) rankings, in which, however, six institutions appeared that did not house counseling psychology programs accredited by the American Psychological Association. DISCUSSION Comparing results from this study with the Delgado and Howard (1994) rankings yields interesting findings. Although there was some stability, considerable differences were found in the rankings across studies. Prior reviews have demonstrated similar patterns of general consistency overall but marked changes for specific institutions throughout time (Delgado & Howard, 1994; Howard, 1983). The differences between the original 1994 rankings and the current rankings may have resulted from the following, in any combination: Different journals were examined; summative instead of factor scores were used to calculate productivity; and publication activity for institutions changed since the last ranking. For example, the inclusion of JCCP (a primary research outlet for clinical psychology research) in previous reviews may have artificially increased the scores of institutions at universities that also house clinical psychology or psychiatry programs, because publication in JCCP by a clinical psychology or psychiatry faculty member would also have been included in the institution s overall ranking. Alternatively, members of these programs may have been more likely to engage in joint research, which is publishable in JCCP.In these cases, research activity not directly attributable to counseling psychology programs still would have been factored into scores, thus creating inflated estimates of research productivity for some programs. Therefore, excluding JCCP may have provided a more accurate evaluation of counseling psychology program publication activity. Although JCD and JVB were incorporated in this study based on the responses given by training directors, similar concerns could be raised that JCD may inflate program scores by drawing on research productivity of counselor education programs or that JVB draws on research productivity of industrial/organizational programs. In this study, JVB scores correlated significantly with JCP (r =.39, p <.01) but demonstrated only marginal relations to TCP (r =.14) and JCD (r =.10). Similarly, JCD scores correlated significantly with JCP (r =.37, p <.05) but only marginally with TCP (r =.25) and JVB (r =.10). Furthermore, the citation impact score for JCD was the

336 THE COUNSELING PSYCHOLOGIST / May 2005 lowest among all the journals. Nevertheless, productivity scores for all journals correlated highly and significantly with JCP. The relations of JCP productivity scores to total productivity scores were the highest for any journal (r =.91, p <.01). Given that JCP is considered a flagship journal of counseling psychology research, the overall results of the correlation matrix suggest that the four journals may have adequately represented institutional research productivity in journals that reflect counseling psychology but also represent more distinct areas within the field. These results may imply that counseling psychology may be changing throughout time. The four journals were recommended by training directors as being primary research outlets for the field. That JCCP was not included as one of the most important journals may indicate a shift in emphasis within the field since Howard s (1983) assessment. The lone nomination for JCCP suggests that this journal may not reflect the current zeitgeist of counseling psychology, although it may still represent an important outlet for related fields. That no specifically multicultural journals were ranked in the top four as being germane to counseling psychology is somewhat surprising, given the field s unique focus on multicultural issues. This absence may stem from several factors. First, no multicultural journal has yet distinguished itself as a flagship journal for counseling psychologists, although many multicultural journals may serve as publication outlets for counseling psychologists (e.g., Journal of Multicultural Counseling & Development or Cultural Diversity and Ethnic Minority Psychology). Alternatively, training directors may believe that research published in the journals they identified already sufficiently incorporates multicultural issues. Thus, they may have seen no need to specifically nominate a separate journal to more explicitly incorporate multicultural issues as they relate to counseling psychology. Consistent with this hypothesis, recent reviews (e.g., Buboltz et al., 1999; Koegel, Donin, Ponterotto, & Spitz, 1995; Perez et al., 2000) demonstrated that the percentage of articles incorporating a multicultural emphasis in both JCP and JVB has more than doubled in recent years. Regardless, the current authors determined to use the nominations of the training directors. As such, the journal selection process was not altered to arbitrarily incorporate other journals that were not specifically nominated. Note that three institutions ranked in this study do not house counseling psychology programs. Thus, only tentative conclusions should be drawn about program-specific research productivity from the available institutional-level data. Research productivity by individuals in noncounseling psychology programs likely led to these institutions being ranked (e.g., University of North Carolina, Greensboro, has a counselor education program and received its ranking almost exclusively based on its JCD score; JCD is published by the American Counseling Association). The overall number of such

Diegelman et al. / COUNSELING PSYCHOLOGY RESEARCH PRODUCTIVITY 337 institutions, however, is only half the number of such institutions that were included by Delgado and Howard (1994), suggesting that the current journal mix and ranking methodology may more reflect research productivity attributable to specific counseling psychology programs. A review of the 13 programs that were previously ranked by Delgado and Howard but not ranked in the current study found that 6 of these institutions did not house counseling psychology programs, and 2 institutions had phased out their programs. Future efforts to examine research productivity in counseling psychology may benefit from a greater focus on counseling psychology programs. Such efforts may further reduce the potential for inflated rankings resulting from research activity outside counseling psychology programs, such as activity from counselor education and/or business programs. In addition, such an approach might be more inclusive of active counseling psychology faculty working internationally. One potentially useful method might involve focusing specifically on research productivity of faculty who are members of counseling psychology programs (Hanish et al., 1995). This method would also redress those institutions housing two separate programs that might both be contributing to the overall institutional score. An additional benefit to this approach would be the ability to adjust rankings based on faculty size. Current ranking exercises may favor institutions with more faculty; focusing specifically on contributions of faculty employed in counseling psychology programs would allow an average number of publications per faculty member within a department to be calculated. Similarly, possible differences related to the characteristics of faculty within each program (i.e., junior vs. senior status) could be examined using such methods. However, significant downfalls accompany such an approach. Early attempts were made during the current study to examine more closely scholarly productivity that was attributable specifically to counseling psychology programs, as opposed to using more institutional-level data. Nevertheless, numerous drawbacks hindered these efforts, which may explain (in part) why previous similar reviews did not fully incorporate such an approach. Given that there were more than 4,000 author contributions, it soon became evident that tracking departmental affiliations for each author, or adding a correction for department size, was not pragmatic. This drawback was due to many factors, including the relative fluctuations in size and constitution of program faculty during a 10-year period given the regular changes resulting from tenure decisions and institutional affiliations. More importantly, in addition to pragmatic issues, several theoretical concerns support the use of broader inclusion criteria that comes with institutional-leveldata. Specifically, if ranking studies were to focus only on publications by faculty, credit would not be assigned for publications by graduate students in those same departments, despite the fact that publica-

338 THE COUNSELING PSYCHOLOGIST / May 2005 tions by graduate students also indicate programmatic research productivity and research training. The use of institutional-level data does not preclude tentative and cautious inferences pertaining to research productivity of specific counseling psychology programs. Specifically, more than 90% of the institutions publishing in journals related to counseling psychology that made the current ranking list, in fact, house counseling psychology programs accredited by the American Psychological Association. Furthermore, these findings do not appear to result solely from institution size, because many large academic universities nationally (and internationally) that do not house counseling psychology programs subsequently did not make the list. This absence suggests that research productivity of specific counseling psychology programs likely affected the overall productivity scores significantly, as would be expected. In addition, this finding further supports the utility of incorporating the specific journals nominated by training directors, who appear to have adequately captured research productivity related to counseling psychology. Nevertheless, note that these results should not be interpreted as a ranking of counseling psychology program quality, primarily because the data in this study (and much of the work preceding it) are institutional in nature. Realize that scholarly research productivity is only one aspect of the overall quality of counseling psychology programs and that the use of institutional-level data precludes definite conclusions about research productivity of specific counseling psychology programs. Avoid any tendency to make blanket judgments of overall counseling psychology program quality based on institutional ranking scores. The purpose of the current research was not to provide quality ratings but only to document the extent to which institutions as a whole publish in journals germane to counseling psychology. Impact of research, student internship and job placement, coursework quality and relevance, mentoring, infusion of diversity, and opportunities relevant to training and practicum experiences all represent additional aspects worthy of merit when considering the quality of counseling psychology programs. Thus, the current data may be most useful when considered with these and other factors with the goal of helping faculty, administrators, or prospective students ascertain the strengths of programs. Similarly, given the greater focus by many administrators in recent years on securing external grant funding for research, future investigations may obtain different results by incorporating a broader focus on additional outlets that may publish more grantfunded research than the traditional outlets identified in the current study and by previous authors (e.g., Delgado & Howard, 1994). Nevertheless, in doing so, such studies would be broadening the focus away from the more traditional research areas and publication outlets of counseling psychology identified by the training directors for the present study.

Diegelman et al. / COUNSELING PSYCHOLOGY RESEARCH PRODUCTIVITY 339 REFERENCES Bohn, M. J. (1966). Institutional sources of articles in this journal of counseling psychology Four years later. Journal of Counseling Psychology, 13, 489-490. Buboltz, W. C., Miller, M., & Williams, D. J. (1999). Content analysis of research in the Journal of Counseling Psychology. Journal of Counseling Psychology, 46, 496-503. Cox, W. M., & Catt, V. (1977). Productivity ratings of graduate programs in psychology based upon publication in the journals of the American Psychological Association. American Psychologist, 32, 793-813. Delgado, E. A., & Howard, G. S. (1994). Changes in research productivity in counseling psychology: Revisiting Howard (1983) a decade later. Journal of Counseling Psychology, 41, 69-73. Goodstein, L. D. (1963). The institutional sources of articles in the Journal of Counseling Psychology. Journal of Counseling Psychology, 10, 94-95. Goodyear, R. K., Abadie, P. D., & Walsh, W. B. (1983). Graduate school origins of Journal of Counseling Psychology authors: Volumes 15-28. Journal of Counseling Psychology, 30, 283-286. Hanish, C., Horan, J. J., Keen, B., St. Peter, C. C., Ceperich, S. D., & Beasley, J. F. (1995). The scientific stature of counseling psychology training programs: A still picture of a shifting scene. The Counseling Psychologist, 23, 82-101. Heesacker, M., Heppner, P. P., & Rogers, M. E. (1982). Classics and emerging classics in counseling psychology. Journal of Counseling Psychology, 29, 400-405. Howard, G. S. (1983). Research productivity in counseling psychology: An update and generalization study. Journal of Counseling Psychology, 30, 600-602. Howard, G. S., Cole, D. A., & Maxwell, S. E. (1987). Research productivity in psychology of American universities. American Psychologist, 42, 975-986. Howard, G. S., & Curtin, T. D. (1993). Individual productivity and impact in counseling psychology. The Counseling Psychologist, 21, 288-302. Katz, M., & Brophy, A. L. (1975). Institutional sources of articles in the Journal of Counseling Psychology, 1962-1973. Journal of Counseling Psychology, 22, 160-163. Koegel, H. M., Donin, I., Ponterotto, J. G., & Spitz, S. (1995). Multicultural career development: A methodological critique of 8 years of research in three leading career journals. Journal of Employment Counseling, 32, 50-63. Perez, R. M., Constantine, M. G., & Gerard, P. A. (2000). Individual and institutional productivity of racial and ethnic minority research in the Journal of Counseling Psychology. Journal of Counseling Psychology, 47, 223-228. Roose, K. D., & Anderson, C. J. (1970). A rating of graduate programs. Washington, DC: American Council on Education. Thoreson, R. W., Cox, J. G., & Krauskopf, C. J. (1975). Reputation, halo, and ratings of counseling programs. Journal of Counseling Psychology, 22, 446-450. Tinsley, D. J., & Tinsley, H. E. A. (1979). Trends in institutional contributions to the Journal of Counseling Psychology. Journal of Counseling Psychology, 26, 152-158. Williams, M. E., & Buboltz, W. C. (1999). Content analysis of the Journal of Counseling and Development: Volumes 67 74. Journal of Counseling and Development, 77, 344-349.