Research Productivity in Counseling Psychology: An Update

Professional Forum Research Productivity in Counseling Psychology: An Update 10.1177/0011000005277814 THE Buboltz COUNSELING et al. / RESEARCH PSYCHOLOGIST PRODUCTIVITY / September 2005 Walter C. Buboltz Jr. Louisiana Tech University Steve M. Jenkins Wagner College Adrian Thomas Auburn University Lori D. Lindley Jonathan P. Schwartz James M. Loveland Louisiana Tech University This article is an update and review of institutional research productivity in counseling psychology. Institutional research productivity is assessed by totaling credits for articles published from 1993 to 2002 in the following journals: Journal of Counseling Psychology, The Counseling Psychologist, Journal of Consulting and Clinical Psychology, Journal of Vocational Behavior, Journal of Counseling and Development, Journal of Multicultural Counseling and Development, and Cultural Diversity and Ethnic Minority Psychology. Results show that the rankings of many programs have remained fairly stable over the years, while others have substantially changed. Additionally, two factors were found to represent the research productivity of institutions in the field. These results are discussed in terms of the identity of counseling psychology and research productivity. When the field of counseling psychology was developed in the 1940s, its initial role was generally seen as one that would fill a perceived gap between industrial/organizational psychology and clinical psychology (Gelso & Fretz, 2001). A major emphasis of this new branch of psychology was to study and treat normal personality, functioning, and development. This emphasis continues to be integral to counseling psychology s identity today. However, over the past 60 years, counseling psychology has evolved into a distinct discipline that contains diverse areas of specialization and that continues to grow and further define its role within psychology. In fact, Division 17 (Society for Counseling Psychology) of the American Psychological Please address all correspondence to the first author at Department of Psychology, Louisiana Tech University, PO Box 10048, Ruston, LA 71272. THE COUNSELING PSYCHOLOGIST, Vol. 33 No. 5, September 2005 709-728 DOI: 10.1177/0011000005277814 2005 by the Society of Counseling Psychology 709

710 THE COUNSELING PSYCHOLOGIST / September 2005 Association has 9 sections and 11 special interest groups, including sections on ethnic and racial diversity, counseling health psychology, and teaching and research on qualitative methods (Division 17 of the American Psychological Association, 2003). Training programs in counseling psychology reflect the evolution of the field, with most institutions ascribing to the scientist-practitioner model. With this model, students are afforded the training and knowledge to become proficient practitioners and researchers. As the field continues to grow and diversify, its identity, and thus its outlets for research, will broaden accordingly. This broadening may give clues to the evolving identity of counseling psychology. By examining the primary journals in which counseling psychologists have published, we may be able to identify trends in the areas of research that are being pursued by the major academic counseling psychology training programs. Since each journal has a mission and publishes research that is consistent with that mission, changes observed in the number of publications indicate that researchers and training programs are either focusing more closely on the area covered by the journal or are focusing on areas not covered by the journal and therefore publishing in other journals. These potential changes in the research productivity of institutions for the journals over the years may indicate that other areas of counseling psychology or psychology in general are emerging as part of the identity of counseling psychology. For example, if a noticeable decrease was noted in the publication numbers for the Journal of Counseling Psychology across years, while an increase was noted in the Journal of Vocational Behavior, then the indication may be that career issues are becoming more salient than the areas covered by Journal of Counseling Psychology. We can further extrapolate that if new areas are emerging in training programs through research and productivity, these areas are being passed to trainees during their doctoral education. Through the process of examining research productivity one can better understand the identity of counseling psychology and gain insight into the future direction of the field. The quality of graduate training programs has traditionally been assessed in two ways: reputation (Roose & Anderson, 1970) or research productivity (Howard, Cole, & Maxwell, 1987). Over the years debate regarding which method is the most fruitful and accurate has been heard. Cox and Catt (1977) argue against the use of reputation ratings in determining the quality of graduate programs in the field of psychology. They argue that use of reputation may be fraught with numerous errors that can be misleading. Therefore there has been a long history of tracking institutional research productivity of programs by the frequency of publications in key journals. The quality of each article was not considered because this is beset with potential pitfalls. The

Buboltz et al. / RESEARCH PRODUCTIVITY 711 overall quality was considered to be at a minimum level as each article has undergone a review by an editorial board and editors for its relevance, quality, and contribution. However, relying solely on research productivity data or reputation may be misleading as to the quality of graduate training. The use of research productivity may not reflect the actual training and quality of graduates froma program. Several other markers may be used to assess the quality of graduate training in counseling psychology. For example, one could look at internship and employment placement, scores on the National Licensing Examination, and other potential outcome variables. Examining every marker to determine the quality of a graduate program was beyond the scope of this study. This study focused on research productivity as one of the many potential important aspects of program quality and identity. Additionally, in the current environment of accountability, national rankings (e.g., National Research Council), and cost structures, programs that do not add to the value and prestige of a university are overlooked and in some cases phased out. This point was made clear at the midwinter (2004) meeting of the Council of Counseling Psychology Training Programs as many counseling psychology doctoral programs would not be included in national rankings because they are housed in colleges of education and not in departments of psychology. The current study describes one way that programs can show their standing in the field and provide administrators with data about the productivity and value of the program. Additionally, the ranking of programs based on research productivity can provide a means by which programs can increase their reputation and value to a university. Initially, counseling psychology research productivity was defined by the institutional contributions to the Journal of Counseling Psychology (Cox & Catt, 1977) and was later expanded to five representative journals (cf. Howard, 1983). The most recent studies on productivity in counseling psychology that considered multiple journals as representing the field were by Delgado and Howard (1994) covering 1983-1992 and by Diegelman, Uffelman, Wagner, and Diegelman (2005) covering 1993-2002. However, note that Diegelman et al. employed only four of the journals from Delgado and Howard, omitting Journal of Consulting and Clinical Psychology (JCCP). The journals included in Diegelman et al. were determined based on the frequency of endorsement by training directors of counseling psychology programs. Diegelman et al. only included the top four most frequently endorsed journals in which counseling psychology faculty and students publish, as perceived by directors of training. The authors state that by limiting the review to four journals that they believed most represented the field would lead to a better capture of the research productivity of faculty and students in counseling psychology. Although we agree with this rationale to

712 THE COUNSELING PSYCHOLOGIST / September 2005 some degree, this may not be the case. Due to the changing nature of the field and the development of other substantive areas of research not covered by these journals, many potentially important areas may have been left out of their analysis. Thus, to correct for this potential problem, this study included two additional journals to better encompass the identity of counseling psychology and reflect its commitment to diversity: the Journal of Multicultural Counseling and Development and Cultural Diversity and Ethnic Minority Psychology. This study also updates and extends the current research productivity in counseling psychology by including two additional journals that are believed to represent the field. The inclusion of these two journals may help alleviate the potential shortcomings of previous reviews (Delgado & Howard, 1994; Diegelman et al., 2005) that focused on journals that have been traditionally believed to represent counseling psychology. Additionally, because the Journal of Counseling Psychology has been used as a primary journal for tracking research productivity over the past 35 years in counseling psychology, this journal is also examined individually for comparison purposes and examination of productivity changes over the years. The Diegelman et al. (2005) study used four of the original journals of the Delgado and Howard (1994) study, postulating JCCP no longer represented current counseling psychology research. This belief was based on responses from training directors rather than a representative sample of counseling psychologists. Since JCCP was not included in their analysis it cannot be empirically determined if JCCP represents counseling psychology. Because of this potential shortcoming, this study included all the original journals as well as the additional two. Inclusion of the all the original journals allows for replication and testing of the Delgado and Howard model to determine if that model is still current. Additionally, by examining the contributions to representative journals some insight can be gained into the identity of counseling psychology. By examining changes or lack thereof in the number of publications in journals that represent counseling psychology we can get an idea of the field s identity based on the missions of the journals. Finally, by examining the productivity across the seven journals we can better understand the research productivity of programs based on the current state of the field. Although the purpose was not to rank programs in terms of their overall quality of training these data may be used to gain some insighton one aspect of the quality of training. Institutions that are more active in research and publication may afford students more opportunities to learn research and publication skills. Additionally, the data of this study can be used by programs to increase their reputation and to show their value to their institutions.

Buboltz et al. / RESEARCH PRODUCTIVITY 713 METHOD To be consistent and to allow for comparisons with the previous productivity studies, the same methodology was employed in this study to obtain research productivity ratings. Howard (1983) identified the five top journals in which counseling psychologists publish (noted above). However, in addition to the original five journals two other journals were also included in this secondary analysis to better represent the field. The original five journals were included in the current study to allow for accurate comparisons and to test the model proposed by Delgado and Howard (1994). Additionally, four of the journals were found to represent the field based on responses from training directors in the Diegelman et al. (2005) study. The inclusion of the two new journals reflects the changing nature of the field and reflects opinions of editors and editorial board members. Additionally, as was noted by Buboltz, Williams, and Miller (1999), the content areas of psychology have been increasing and thus may not be reflected in the more traditional counseling psychology journals. Credit for authorship was assigned in the following manner: Articles received a total authorship institution credit of 1.0. Thus, the institution of an individual responsible for a single-authored article was credited with 1.0 points. The institution of the first author of a coauthored article received.67 points, and the second author s institution received.33 points. For articles with three authors, the breakdown follows:.50 for first author s institution,.30 for second author s institution, and.20 for third author s institution. Similar divisions of credit for articles having more than three authors were obtained by using Howard et al. s (1987) formula for determining productivity credit, on the basis of an article s number of authors and each author s ordinal position on the article. The period covered by the current analysis ranged from 1993 to 2002, inclusively. For each of the seven journals, all articles were included in the analysis except for letters to the editor, introductions to special sections, book reviews, presidential addresses, and test reviews. Comments, brief reports, and reactions were included in the analysis. All authors (including students) were given the appropriate institutional credit depending on their ordinal rank. Note that institutions that do not have doctoral programs in counseling psychology were included in these analyses to help determine the overall standing of counseling psychology programs in the field. A counseling psychology program was not teased apart from institutional affiliation because we were partially interested in publication trends in the journals to get some insight about the identity of the field. The exclusion of non counseling psychology program faculty would tend to overestimate the contribution of fac-

714 THE COUNSELING PSYCHOLOGIST / September 2005 ulty members while excluding the contribution to the field of counseling psychologists in other employment settings (e.g., counseling centers or master s program in counseling) that are clearly involved in training. Additionally, allowing other individuals who were not formally part of a counseling psychology program to contribute to an institution s rank would eliminate collaborative efforts despite the fact that the article was published in a journal examined in the study. The number of articles reviewed for each journal follow: Journal of Counseling Psychology, 496 articles; JCCP, 1,100 articles; Journal of Counseling and Development, 601 articles; Journal of Vocational Behavior, 445 articles; The Counseling Psychologist, 344 articles; Journal of Multicultural Counseling and Development, 205 articles; and Cultural Diversity and Ethnic Minority Psychology, 135 articles. RESULTS All results for this study employed the same methodology (except for the additional journals included in the second analysis) and statistical analyses as those employed by Delgado and Howard (1994). The same procedures and analyses were used to allow for accurate comparisons between the two studies. A confirmatory factor analysis was conducted on the five previous target journals to determine the degree of relationship between the latent variable (institutional productivity) and the observed variables (five measurements of the latent variable). A second factor analysis was conducted for all seven of the journals that were identified to currently represent the field. Due to the possible impact of faculty size on research productivity, productivity rankings were also adjusted for the number of faculty members in each counseling psychology program. The faculty size used was the current number of core counseling psychology faculty members listed for each program. Gathering information about the amount of time that each faculty member contributes to the program or about contributions of noncore or adjunct faculty members was impossible. This adjustment was made by simply dividing the overall productivity index for each university by the number of core faculty members at the time of preparation of the current manuscript. However, note that this adjustment was only made for counseling psychology programs as information was not available for other programs that contributed to the journals. Results of the first confirmatory factor analysis showed that the data did not satisfactorily fit a one-factor model, 2 (5, N = 1,133) = 13.32, p =.02. Removal of JCCP resulted in a one-factor model to which the data showed adequate fit, 2 (2, N = 1,133) =.258, p =.879. Note, however, that 2 is

Buboltz et al. / RESEARCH PRODUCTIVITY 715 dependent on sample size, and Delgado and Howard s (1994) reliance on it as their only measure of fit may be viewed as a limitation of their work. As such, the current study obtained additional measures of the fit of the data to the model. For the current one-factor model based on the four journals, the comparative fit index (CFI) was 1.0, the goodness-of-fit index (GFI) was.99, and the root mean square residual was.01. According to generally accepted practice, CFIs and GFIs exceeding.95 and root mean square residuals of less than.08 demonstrate good fit. Table 1 presents the productivity rating on the four journals for the institutions that ranked in the top 40 for overall productivity for 1993-2002. The four journals were included in the table for comparison purposes. The first four columns present the productivity for each institution across the four journals. The fifth column ( overall productivity index ) is the factor score derived from the confirmatory factor analysis on the productivity scores for the four journals, not including JCCP. The sixth column shows the current rank without faculty size adjustment. The seventh column shows the productivity index and rank adjusted for faculty size. The eighth column shows the rank from the Delgado and Howard (1994) study. However, note that Delgado and Howard used all five journals in their factor analysis. Examination of Table 1 shows that the University of Missouri Columbia ranks first in the present study and the previous study. Additionally, the University of Maryland College Park ranks second in the present and previous studies. Overall, examination of Table 1 shows that 7 of the institutions that were ranked in the top 10 in 1994 are still ranked in the top 10. However, note that a few institutions that were ranked outside the top 20 in 1994 are now ranked within the top 20. Additionally, 12 institutions that were not ranked in Delgado and Howard (1994) are ranked in the top 40 in the current study. Comparing the institutional contributions of each journal in the present study for top 10 ranked institutions with the institutional contributions in Delgado and Howard (1994) reveals interesting results. First, most of the leading institutions have increased their contributions in the Journal of Counseling Psychology. For JCCP, contributions from the top 10 ranked institutions have uniformly decreased. Finally, for the other three journals, contributions have been fairly stable with only minor changes in contributions between the two studies. When rankings were adjusted for faculty size, the top three institutions showed little change in rank. However, noticeable changes were observed for other institutions. Four institutions previously outside the top 10 in the nonadjusted rankings moved into the top 10. Note also that several institutions changed little in their rank (e.g., University of Mississippi moved from 16 to 17), while others showed large changes (e.g., University of North Texas moved from 19 to 33). However, these numbers should be interpreted with

TABLE 1: Productivity Ratings on the Four Journals for the 40 Top Institutions Ranked on Overall Productivity Factor Scores Overall Rank Faculty Rank of Rank of Productivity Present Adjusted Dieg 1994 Institution JCP JCD JVB TCP Index Study Index (Rank) Study Study University of Missouri Columbia 41.71 23.14 8.00 14.65 15.039 1 1.11 (2) 1 1 University of Maryland College Park 44.53 10.37 11.04 14.60 14.622 2 1.88 (1) 2 2 University of Akron 10.51 7.80 13.99 9.64 6.907 3 1.02 (4) 3 18 University at Albany, State University 13.77 7.33 6.00 9.69 6.658 4 0.77 (9) 4 4 of New York Arizona State University 9.50 9.46 5.95 8.95 5.844 5 0.61 (15) 7 9 Iowa State University 12.66 2.50 14.63 4.34 5.150 6 0.76 (10) 6 29 University of Iowa 6.87 5.56 1.99 9.17 4.662 7 0.81 (7) 15 5 Ohio State University 5.99 9.22 12.41 5.09 4.589 8 0.48 (20) 8 3 Virginia Commonwealth University 14.29 0.94 2.84 5.84 4.582 9 0.68 (11) 13 10 University of Illinois, Urbana-Champaign 15.51 3.00 10.54 2.08 4.573 10 0.67 (12) 9 12 University of Notre Dame 4.50 0.63 2.83 11.14 4.480 11 0.67 (13) 21 11 University of Minnesota Minneapolis 0.67 4.80 13.00 8.01 4.210 12 1.10 (3) 5 8 University of California Santa Barbara 7.67 0.33 4.50 7.17 3.919 13 N/A 16 7 Pennsylvania State University 7.33 4.15 9.46 3.82 3.649 14 0.95 (5) 10 14 Southern Illinois University Carbondale 9.33 1.70 11.29 2.72 3.609 15 0.62 (14) 14 15 University of Southern Mississippi 5.33 9.05 0.00 6.24 3.572 16 0.53 (17) 18 NR Michigan State University 8.76 6.65 6.12 3.06 3.564 17 0.92 (6) 12 30 University of Wisconsin Madison 7.01 6.52 0.00 5.53 3.406 18 0.39 (23) 22 36 University of North Texas 4.53 5.09 2.47 6.17 3.276 19 0.31 (33) 24 6 Boston College 8.00 2.66 1.33 5.17 3.236 20 0.37 (24) 27 NR 716

Columbia University Teachers College 4.18 3.50 2.01 6.83 3.226 21 0.48 (18) 25 32 University of Florida 4.83 2.64 2.50 6.55 3.222 22 0.48 (19) 17 17 Ball State University 5.57 4.20 1.33 5.84 3.158 23 0.27 (36) 11 NR Loyola University of Chicago 7.78 1.50 5.01 3.72 2.993 24 0.78 (8) 26 NR University of Wisconsin Milwaukee 5.83 1.50 4.53 4.50 2.841 25 0.37 (25) 30 NR Texas Tech University 6.30 4.65 2.00 3.94 2.790 26 0.36 (26) 28 NR University of Oregon 8.50 1.50 0.33 3.74 2.654 27 0.55 (16) 29 31 Indiana University 1.67 6.03 1.34 5.18 2.414 28 0.25 (37) 19 NR University of Southern California 5.83 1.61 0.00 4.50 2.391 29 0.31 (32) 31 33 Fordham University 2.87 2.00 0.00 5.97 2.364 30 0.41 (22) NR NR Colorado State University 7.00 1.18 3.51 1.88 2.076 31 0.18 (42) 35 20 University of Nebraska 4.53 2.86 1.20 3.50 2.075 32 0.43 (21) 36 16 University of Utah 2.84 1.20 2.00 4.50 2.013 33 0.35 (27) NR 39 Washington State University 2.00 0.50 4.00 4.32 1.938 34 0.29 (34) 39 NR University of Memphis 3.33 2.67 1.67 3.62 1.926 35 0.33 (28) NR NR University of Oklahoma 0.84 4.83 2.60 4.04 1.910 36 0.29 (35) 33 NR University of Kansas 3.07 1.95 3.67 3.26 1.897 37 0.33 (30) NR 24 University of Georgia 1.50 12.04 6.04 0.20 1.834 38 0.24 (39) 23 25 University of California Los Angeles 4.65 0.50 1.00 3.17 1.745 39 N/A NR 23 Illinois State University 7.67 1.67 1.00 0.83 1.652 40 N/A 40 NR NOTE: JCP = Journal of Counseling Psychology; JCD = Journal of Counseling and Development; JVB = Journal of Vocational Behavior; TCP = The Counseling Psychologist; Dieg = Diegelman et al. (2005); U = university; NR = not ranked; N/A = no doctoral program in counseling psychology. Higher productivity numbers indicate greater productivity; lower numerical rankings indicate greater productivity. The overall productivity index represents the sum across the four journals of the product of each institution s productivity total and that journal s loading on the productivity factor. 717

718 THE COUNSELING PSYCHOLOGIST / September 2005 caution, as an institution with 5 faculty members that produces a total of 10 articles a year would rank equally with an institution with 20 faculty members that produces a total of 40 articles a year. However, in overall productivity, universities with larger faculty numbers are clearly contributing more to the field even when the data are corrected for faculty size. Results of the second confirmatory factor analysis showed that a twofactor model represented a satisfactory fit for the data from the seven journals. Again, as 2 is dependent on sample size, more appropriate fit indices were computed. These were computed on the entire two-factor model (i.e., we did not compute different fit indices for each factor separately). The estimates of fit for the data to the two-factor model follow:.96 for the CFI,.97 for the GFI, and.08 for the root mean square residual. These numbers are again taken as evidence that the data provided sufficient fit to the two-factor model. The first factor was represented by the same four journals that were highlighted in Table 1. Results for the first factor have previously been discussed. The second factor was represented by JCCP, Cultural Diversity and Ethnic Minority Psychology, and Journal of Multicultural Counseling and Development. Table 2 presents the productivity rating on the three journals of factor 2 for the institutions that ranked in the top 40 for overall productivity during 1993-2002. Note that Table 2 presents data only for the institutions that have counseling psychology doctoral programs. However, the rankings reflect ranks when including all institutions. Examination of Table 2 shows that out of the top 40 institutions only 18 have doctoral programs in counseling psychology. This may indicate that the field as represented by doctoral programs in counseling psychology is not publishing in the areas covered by this cluster of journals as much as it publishes in the more traditional areas of counseling psychology (i.e., those in factor 1). The top two ranked institutions were the University of California Los Angeles and the University of Hawaii Manoa, both of which do not have doctoral programs in counseling psychology. The next two ranked institutions were the University of Maryland College Park and Columbia University Teachers College, both of which have doctoral programs in counseling psychology. Also note that examination of the contributions to each journal on which the rankings were based tends to decrease dramatically for institutions with counseling psychology programs once past the top 12 counseling psychology programs. These results tend to indicate that the areas of diversity and multiculturalism as well as other areas which are part of counseling psychology may not be as dominant in the research areas of faculty at the doctoral programs. Comparisons among counseling psychology doctoral programs after adjusting for the size of faculty show some changes in the rankings; however, the top institutions tend to remain high.

Buboltz et al. / RESEARCH PRODUCTIVITY 719 TABLE 2: Productivity Ratings for the Second Factor Analysis for the 40 Top Institutions Ranked on Overall Productivity Factor Scores Faculty Overall Adjusted Productivity Present Index Institution JCCP CDEMP JMCD Index Rank (Rank) University of Maryland 2.22 4.20 5.21 5.93 3.74 (2) College Park Columbia University 4.54 3.39 4.6 5.10 4.73 (3) Teachers College University of Wisconsin 5.68 2.14 5.52 4.53 6.5 (6) Madison Pennsylvania State 7.24 2.89 3.46 4.32 7 1.08 (1) University Ohio State University 10.8 1.79 1.92 2.89 10.29 (13) University of Minnesota 3.0 2.33 1.16 2.49 15.36 (7) Minneapolis University of Missouri 3.34 0 5.13 2.36 16.17 (24) Columbia Fordham University 0 0 5.08 2.12 18.35 (8) University of Southern 11.40 1.12 1.35 2.09 19.26 (15) California Michigan State University 4.33 0 4.27 2.03 20.51 (5) Temple University 15.57 0.21 2.23 1.98 21.33 (9) University of Memphis 10.95 0.38 2.47 1.94 23.32 (10) University of Louisville 1.0 0 4.28 1.82 29.30 (11) University of Illinois, 1.0 1.42 1.53 1.76 31.25 (16) Urbana-Champagne Arizona State University 5.2 0 3.56 1.75 32.18 (22) Oklahoma State 0.5 1.47 1.0 1.52 34.15 (30) Western Michigan 0 1.0 1.88 1.49 35.17 (26) University Howard University 0 0 3.64 1.45 37.29 (12) Southern Illinois University 1 1 1.38 1.33 41.22 (19) Carbondale University of Texas Austin 8.74 0.96 0.35 1.31 42.22 (20) Washington State University 1.37 0 3.0 1.25 47.18 (21) West Virginia University 2.33 1.0 1.0 1.24 48.16 (29) Seton Hall 0 1.28 0.79 1.23 49.25 (17) University of Florida 2.56 0 2.79 1.22 50.18 (23) Virginia Commonwealth 7.46 1.07 0 1.16 51.17 (25) University University of Georgia 5.41 0 2.20 1.13 52.13 (32) University of Colorado 0.52 0.08 2.10 0.84 60.02 (66) University of Oregon 2.01 0 2.0 0.82 61.17 (27) Stanford University 4.52 0.89 0 0.82 62.27 (14) Georgia State University 2.78 0 1.88 0.82 63.10 (37) (continued)

720 THE COUNSELING PSYCHOLOGIST / September 2005 TABLE 2 (continued) Faculty Overall Adjusted Productivity Present Index Institution JCCP CDEMP JMCD Index Rank (Rank) University of Oklahoma 0 0 2.25 0.81 64.12 (35) University of Nebraska 0.30 0 2.20 0.80 65.16 (28) Texas A&M University 0.84 1.0 0.26 0.80 67.13 (33) University of Pittsburgh 13.79 0 0 0.66 78.22 (18) University of Akron 0 1.0 0 0.62 82.09 (39) University of Iowa 3.93 0.16 1.0 0.62 89.10 (36) University of North Texas 1.0 0.64 0.40 0.56 95.05 (42) Loyola University of 5.64 0 0.74 0.47 102.12 (34) Chicago Boston College 0 0 1.47 0.45 104.05 (43) Lehigh University 0 0 1.46 0.44 106.09 (40) NOTE: JCCP = Journal of Consulting and Clinical Psychology; CDEMP = Cultural Diversity and Ethnic Minority Psychology; JMCD = Journal of Multicultural Counseling and Development; N/A = no doctoral program in counseling psychology. Higher productivity numbers indicate greater productivity; lower numerical rankings indicate greater productivity. The overall productivity index represents the sum across the three journals of the product of each institution s productivity total and that journal s loading on the productivity factor. The intercorrelations among the seven journals and each journal s loading on the overall productivity factor score are presented in Table 3. Examination of the table shows that most intercorrelations and productivity factor scores are high, except for the intercorrelations with JCCP and the other four more traditional journals that represent counseling psychology. The low intercorrelations between JCCP and the four original journals are not surprising given the fact that JCCP is not part of the first factor. Additionally, the correlation between the two factors was.36 (p <.01). Thus, despite the different rank order of the institutions, the two factors are still moderately correlated. Table 4 highlights more than 30 years of productivity research that is based on the Journal of Counseling Psychology. Several studies in the past have used the Journal of Counseling Psychology as the sole criterion or part of the criterion for research productivity (Cox & Catt, 1977; Delgado & Howard, 1994; Howard, 1983). Examination of the validity coefficients from the various factor analyses conducted have ranged from.68 (Howard, 1983) to.93 for the present study. Note that the first four columns of rankings in Table 4 are directly comparable since all the studies used similar methodologies for assigning author credit and are based on institutional productivity credits.

Buboltz et al. / RESEARCH PRODUCTIVITY 721 TABLE 3: Intercorrelations Among Journals Journal JCP JCD JVB TCP JCCP CDEMP JMCD JCP JCD.62** JVB.60**.48** TCP.78**.61**.60** JCCP.20**.17**.24**.21** CDEMP.22*.12**.15**.19**.34** JMCD.45**.43**.30**.44**.27**.49** Productivity.93**.74**.72**.93** Factor 1 Productivity.51**.92**.75** Factor 2 NOTE: JCP = Journal of Counseling Psychology; JCD = Journal of Counseling and Development; JVB = Journal of Vocational Behavior; TCP = The Counseling Psychologist; JCCP = Journal of Consulting and Clinical Psychology; CDEMP = Cultural Diversity and Ethnic Minority Psychology; JMCD = Journal of Multicultural Counseling and Development; Productivity Factor 1 = productivity score for Factor 1; Productivity Factor 2 = productivity score for Factor 2. *p <.05. **p <.01. Examination of Table 4 shows that the University of Maryland College Park has ranked number one across all studies. Additionally, the University of Missouri Columbia and Southern Illinois University Carbondale have both consistently ranked in the top 10 across all studies. Four schools that were not ranked in the top 10 in Delgado and Howard (1994) are now ranked in the top 10, with three (Iowa State University, University of Akron, and Arizona State University) rising in the rankings. Finally, seven schools that were not ranked in the top 20 in the Delgado and Howard or Howard et al. (1987) studies are ranked in the top 20 in the present study. DISCUSSION This study examined the institutional productivity rankings for 1993-2002 as represented in seven leading counseling psychology journals. The results indicate that two factors best represent the field based on publication in these journals. The results of the first factor were generally consistent across most journals, with 7 of the top 10 institutions in the current study having appeared in the top 10 in Delgado and Howard s (1994) study. Additionally, the results of the current study are consistent with Diegelman et al. (2005), with 8 common institutions in the top 10. Note that the rankings in Diegelman et al. are based

722 THE COUNSELING PSYCHOLOGIST / September 2005 TABLE 4: Productivity Rankings in Journal of Counseling Psychology for Five Studies During 1970-2002 Howard, Delgado Cole, & Present & Howard Maxwell Howard Cox & Catt Study 1983-1992 (1987) (1983) (1977) Institution 1993-2002 (1994) 1976-1985 1976-1982 1970-1975 University of Maryland 1 1 1 1 1 University of Missouri 2 3 3 4 4 University of Illinois, 3 7 17 26 16 Urbana-Champaign Virginia Commonwealth 4 4 10 11 NR University University of Albany 5 2 5 14 NR Iowa State University 6 23 8 6 29 University of Akron 7 NR 14 NR NR Arizona State University 8 25 NR NR NR Southern Illinois University 9 5 4 5 5 Carbondale Michigan State University 10 18 40 NR 7 University of Oregon 11 11 NR NR NR Boston College 12 NR NR NR NR Loyola University of Chicago 13 40 NR NR NR Illinois State University 14.5 NR NR NR NR University of California 14.5 6 9 10 NR Santa Barbara Pennsylvania State 16 15 11 25 13 University University of Wisconsin 17 34.5 NR NR 8 Madison Colorado State University 18 22 14 9 12 University of Iowa 19 12 13 18 18 Texas Tech University 20 NR NR 27 NR Ohio State University 21 10 2 2 3 University of Southern 22.5 24 NR NR NR California University of Wisconsin 22.5 NR NR NR NR Milwaukee Ball State University 24 NR NR NR NR University of Southern 25 NR NR 29 NR Mississippi University of Florida 26 21 32 33 NR University of California 27 32 NR NR NR Los Angeles University of Nebraska 28.5 8 6 7 NR University of North Texas 28.5 13 21 38 NR University of Notre Dame 30 9 12 13 NR (continued)

Buboltz et al. / RESEARCH PRODUCTIVITY 723 TABLE 4 (continued) Howard, Delgado Cole, & Present & Howard Maxwell Howard Cox & Catt Study 1983-1992 (1987) (1983) (1977) Institution 1993-2002 (1994) 1976-1985 1976-1982 1970-1975 Columbia University 31 NR NR NR NR Teachers College University of Memphis 32 NR NR NR NR University of Kansas 33 28.5 18 12 15 Fordham University 34 NR NR NR NR University of Utah 35 20 16 19 NR Washington State University 36 36.5 NR 40 NR Indiana University 37 NR 19 15 11 University of Georgia 38 NR NR NR NR University of Oklahoma 39 NR NR 37 NR University of Minnesota 40 17 7 8 2 Minneapolis NOTE: NR = not ranked. Lower numerical rankings indicated greater productivity. on summation of the contributions to each of the four journals, while the current study used an overall productivity score based on factor analysis. The differences in deriving the overall productivity scores between the current study and Diegelman et al. lead to several differences in the rankings. These differences are primarily due to the fact that institutions that published more in journals with higher factor loadings using the factor analysis approach would increase an institution s overall productivity index more than would be obtained by just adding contributions for each journal. The difference in rankings based on the two methodologies can be seen in Table 1. In contrast to the Delgado and Howard (1994) study, the results of this study indicated that JCCP does not constitute a large proportion of the research productivity of counseling psychology in the first factor. This finding may indicate that a proportion of the research is no longer appropriate for JCCP. Note that JCCP was significant in the second factor. Based on what were previously considered traditional journals in counseling psychology, this may reflect a change in the identity or focus of counseling psychology. The articles that in the past would have been published in JCCP are now being published in other journals that have a different mission and focus, such as multiculturalism. This is consistent with the focus on emphasizing the unique and dynamic identity of counseling psychology (Fouad et al., 2004; Sue, 2003).

724 THE COUNSELING PSYCHOLOGIST / September 2005 Results also showed that a few institutions have made great strides in terms of research productivity and have moved up in the rankings. For instance, the University of Akron has moved from 18th (Delgado & Howard, 1994) to 3rd; Iowa State University moved from 29th to 6th; and the University of Southern Mississippi moved from not being ranked to 16th. The overall change in rankings of the various programs in the current study is similar to the reported changes identified by Diegelman et al. (2005). Examination of the contributions for each journal across the various studies showed minimal changes in the contributions to each journal, except for JCCP. Results for faculty-adjusted rankings show that the top institutions changed modestly in the rankings. However, several institutions with smaller faculty moved up in the rankings. As was noted, these results should be taken with a degree of caution as institutions with fewer faculty may receive a higher ranking, despite an overall lower number of contributions. Results of the productivity ratings based solely on the Journal of Counseling Psychology show that 4 of the 5 top schools in the present study were also in the top 5 in the Delgado and Howard (1994) study. Also 6 of the top 10 in the present study were in the top 10 in the previous study. These results lend even more credence to the stability of the ratings over time. Despite the relative stability, a few noteworthy changes have occurred in productivity ratings based on contributions to the Journal of Counseling Psychology. Specifically, Iowa State, Arizona State, and Michigan State gained 17 places, 17 places, and 8 places, respectively, and the University of Akron went from being unranked to being ranked 7th. These institutions and programs should be commended for their achievement, and others may want to explore what measures and methods were adopted or operationalized to lead to this increase in research productivity. Examination of the institutional research productivity serves as one method of evaluating the work of graduate programs and may also lend some insight into the identity of counseling psychology. As can be seen, a change in the research contributed to JCCP has occurred over the past 10 years, with fewer contributions seen. This may indicate that a shift is occurring or has occurred in the identity of counseling psychology as members of the field publish in other journals that have a different focus and mission. At the same time, however, the contributions to the other four journals have remained fairly constant, which may indicate that a large portion of the identity of counseling psychology has remained fairly stable. Another possible reason for the decrease in contributions to JCCP is that counseling psychology has separated itself from the field of clinical psychology, so the idea that the two fields may be merging is not as true as believed (Beutler & Fisher, 1994). The addition of other journals into this study further illuminates the research productivity and identity of counseling psychology. The results of

Buboltz et al. / RESEARCH PRODUCTIVITY 725 the second factor indicate that some programs are publishing in journals that have a strong multicultural and diversity focus. However, as was previously noted, many of the leading institutions in the second factor do not have counseling psychology doctoral programs. For instance, institutions such as Columbia University Teachers College and University of Wisconsin Madison are highly ranked on factor 2 yet are not ranked among the top 10 institutions on factor 1. These results may reflect that doctoral programs may still be focusing on more traditional areas of counseling psychology based on the interests of faculty members and are only just beginning to branch out to newer journals that have a more multicultural/diversity mission. Senior faculty that already have an established area of research may be unlikely to change their interests and research focus. This may also reflect the particular interests of faculty members at various programs. Although these journals represent an area of counseling psychology, the area of multiculturalism/ diversity may also have broad implications for other disciplines (e.g., counselor education, clinical programs, and minority programs) that use these journals as outlets for professional work. This hypothesis is supported by the fact that the Journal of Multicultural Counseling and Development is the journal for the Association of Multicultural Counseling and Development of the American Counseling Association and that Cultural Diversity and Ethnic Minority Psychology is the official outlet for Division 45 of the American Psychological Association. This correlation reflects the fact that these journals are not core counseling psychology journals as they are published by other divisions and organizations. The fact that several counseling psychology programs are highly ranked on factor 2 and that 18 of the top 40 institutions have counseling psychology programs may indicate that multicultural/diversity is beginning to be recognized as an important research and training area. However, multicultural/ diversity research is not supplanting traditional research areas at these leading institutions. Given that multicultural/diversity issues are only one component of doctoral training in counseling psychology, it makes sense that multicultural/diversity research should comprise a smaller portion of the research at leading counseling psychology research institutions. The results also may indicate that some doctoral programs in counseling psychology may be developing the area of multicultural/diversity issues as a specialty. In terms of actual training on multicultural/diversity issues, the data suggest that students in counseling psychology programs may be getting more exposure to multicultural/diversity issues through research but do not lend any insight to formalized training in multicultural/diversity issues. In terms of research productivity, the top institutions in this study represent a select group that emerged out of large number of schools that were represented in the seven journals. The use of research productivity rankings is

726 THE COUNSELING PSYCHOLOGIST / September 2005 just one of many ways of gauging a program s productivity and quality and may not reflect actual training of students. Research productivity does not inherently gauge the overall quality of a program. Research productivity does not lend any evidence to the training model used by programs (scientistpractitioner) as many students can be well trained in the research process without faculty or students having published articles. Research productivity or lack of research productivity by a faculty member does not provide any specific evidence about the training provided to students by each faculty member. Traditionally, counseling psychology has relied on productivity or reputation as a measure of program quality (Delgado & Howard, 1994; Howard et al., 1987; Roose & Anderson, 1970). This approach may be misleading as it ignores various aspects that would indicate the quality of training. Training of counseling psychologists involves multiple skills and knowledge that must be assessed to determine the true quality of the training. Although not one method would gauge a program s quality, the use of multiple methods and outcomes would better indicate overall program quality. For instance, some of the outcomes that could be employed are graduates accomplishments, job and internship placements, assessment of clinical skills after graduation, and scores on the national licensing examination. Additionally, examination of specific skills germane to the field, such as multicultural competencies, could be formally assessed during the program or after completion. The use of multiple methods and outcomes is consistent with American Psychological Association accreditation criteria that examine outcomes of training. Despite this consistency, American Psychological Association accreditation alone does not allow for comparison between programs due to the individual nature of accreditation and the fact that American Psychological Association accreditation requires that each program devise its own training goals and outcomes which are evaluated by the American Psychological Association to determine accreditation. At the current time no national standards with which counseling psychology programs can be compared exist. If the field is truly interested in the overall quality of programs, then future research must consider myriad outcomes that represent program quality. No overall assessment for programs and graduate quality has been identified.it may behoove the field to develop a valid and reliable assessment for program and student quality. Several implications can be drawn from the data. First, the data provide the field with a gauge of the most productive universities that may be considered the leading institutions in the development of the field. Second, they provide students with information about which universities are producing research and publications, which can indicate potential research involvement. By examining the focus and mission of particular journals, students

Buboltz et al. / RESEARCH PRODUCTIVITY 727 can learn about the research focus of universities and their faculty and decide about their fit with the program. In conjunction, this study provides information on not only the traditional areas of counseling psychology but also on a second factor that is more related to diversity and multiculturalism, which was not present in the previous studies. The changes in rankings can be used by program members and administrators to examine the effectiveness of programmatic and institutional initiatives to increase the productivity and effectiveness of counseling psychology research programs. Finally, although controversial, studies of this nature can be used by programs and faculty to argue for resources and maybe even their existence in the current market. Studies of this nature must address the question of quantity versus quality of scholarship depicted in the results. Many would argue that studies that examine productivity in refereed journals with high rejection rates also give credence to the quality of the scholarship due to the rigorous review that these article have undergone. However, the actual impact of the published articles and their ideas on the field is not quantified in a study of this nature. Authors and their institutions that contributed few articles to these journals, but articles that were of high impact, would be ranked lower in the overall productivity rankings. Unfortunately, the examination of impact was beyond the scope of this study. In terms of the future, two avenues can be explored. First, for productivity, future research may include more journals that are believed to represent counseling psychology. Through the use of factor analysis, a greater understanding of the diversity of publications in the field will lend insight into the changing nature of the field. Factor scores could also be used to develop a more sensitive gauge of productivity across different domains. The second avenue would be examination of program quality. To date no large-scale systematic investigation of this question has been undertaken. Although articles of this nature attempt to address quality, many other aspects of program quality previously mentioned are unexplored. An undertaking of this nature would require considerable openness and intellectual honesty as the individuals would have to reveal specific information about their program and student performance. However, these traits would be necessary to meaningfully compare programs in terms of overall quality. REFERENCES Beutler, L., & Fisher, D. (1994). Combined specialty training in counseling, clinical, and school psychology: An idea whose time has returned. Professional Psychology, 25, 62-69. Buboltz, W., Miller, M., & Williams, D. (1999). Content analysis research in the Journal of Counseling Psychology. Journal of Counseling Psychology, 46, 496-503.

728 THE COUNSELING PSYCHOLOGIST / September 2005 Cox, W. M., & Catt, V. (1977). Productivity ratings of graduate programs in psychology based on publications in the journals of the American Psychological Association. American Psychologist, 32, 793-813. Delgado, E. A., & Howard, G. S. (1994). Changes in research productivity in counseling psychology: Revisiting Howard (1983) a decade later. Journal of Counseling Psychology, 41, 69-73. Diegelman, N. M., Uffelman, R. A., Wagner, K. S., & Diegelman, S. A. (2005). Current institutional trends in research productivity in counseling psychology journals. The Counseling Psychologist, 33, 327-339. Division 17 of the American Psychological Association. (2003). Retreived September 18, 2003, from http://www.div17.org Fouad, N. A., McPherson, R. H., Gerstein, L., Blustein, D. L., Elman, N., Helledy, K. I., et al. (2004). Houston, 2001: Context and legacy. The Counseling Psychologist, 32, 15-77. Gelso, C., & Fretz, B. (2001). Counseling psychology (2nd ed.). Fort Worth, TX: Harcourt College. Howard, G. S. (1983). Research productivity in counseling psychology: An update and generalization study. Journal of Counseling Psychology, 30, 600-602. Howard, G. S., Cole, D. A., & Maxwell, S. E. (1987). Research productivity in psychology based on publication in the journals of the American Psychological Association. American Psychologist, 42, 975-986. Roose, K. D., & Anderson, C. J. (1970). A rating of graduate programs. Washington, DC: American Council on Education. Retrieved September 18, 2003, from http://www.div17.org/sections/sigs.htm Sue, D. W. (2003). From the President. Division 17 Newsletter, 25.