How Universities Evaluate Faculty Performance: A Survey of Department Heads

Transcription

1 How Universities Evlute Fculty Performnce: A Survey of Deprtment Heds John A. Centr GRE Bord Reserch Report GREB No. br July This report presents the findings of reserch project funded by nd crried out under the uspices of the Grdute Record Exmintions Bord.

2 How Universities Evlute Fculty Performnce: A Survey of Deprtment Heds John A. Centr GRE Bord Reserch Report GREB No. br July

3 Abstrct Deprtment heds from institutions (minly universities) indicted the weight they generlly give to vrious criteri for evluting individul fculty members. The questionnire they responded to included: () the criteri used for evluting overll fculty performnce; (b) the sources of informtion for evluting teching; nd (c) the kinds of informtion used for judging scholrship or reserch performnce. Although teching, reserch, nd service re generlly cknowledged s the mjor functions of most universities, responses by the deprtment heds indicted tht public or university service is usully given little importnce in evluting fculty members for decisions regrding tenure, slry, nd promotion. The Reserch Universities emphsize reserch nd scholrship, s might be expected. The soclled DoctorlGrnting Universities nd the Comprehensive Universities nd Colleges sid tht teching ws rnked first in importnce, followed closely by reserch. The results of the study lso suggest n increse in the use of some of the more systemtic nd tngible kinds of evidence for evluting teching performnce (e.g., student rtings), though the use of such evidence flls fr short of wht deprtment heds think is needed. n generl, the evlution of reserch nd scholrship depends very much on the level of the institution nd the type of deprtment. For exmple, peer judgments of reserch nd the number of rticles in qulity journls re much more importnt in socil science deprtments in the Reserch Universities thn in the sme deprtments in the Comprehensive Universities nd Colleges. Additionl from: copies of this report, t $. per copy, my be ordered Grdute Record Exmintions Progrm Eductionl Testing Service Princeton, NJ 8 by Eductionl Testing Service. All rights reserved.

4 How Universities Evlute Fculty Performnce: A Survey of Deprtment Heds GREB No. br Decisions concerning tenure nd promotion re becoming incresingly more difficult to mke in colleges nd universities. During the expnsion yers of the sixties, most deprtments were concerned lrgely with recruiting nd keeping competent fculty members; but enrollments hve tpered off nd deprtments re no longer dding stff members. This stedystte condition hs put mny institutions t or ner prescribed limits in the percentge of fculty on tenure; in fct, the Americn Assocition of University Professors Annul Report on the Economic Sttus of the Profession found tht 6 percent of ll fculty members hd tenure in 6. n contrst to the lst decde, institutions re now often forced to mke fine distinctions between generlly competent stff members. Wht informtion do deprtments use in mking these decisions? How do the criticl decision mkers, such s deprtment heds, think n individul fculty member s performnce ought to be evluted? These questions were investigted in this study by surveying prctices t lrge number of universities. Previous Reserch Most studies of how fculty performnce is evluted hve been bsed on reports from cdemic dens or vice presidents. One of the erliest studies ws by Gustd (6) nd included ntionl smple of colleges nd universities. Astin nd Lee (6) repeted the survey bout five yers lter with similrly extensive smple of ll types of postsecondry institutions. Both studies sked dministrtors to indicte the importnce of vrious sources of informtion in evluting, first, fculty performnce nd, second, teching performnce. The results were nlyzed by type of institution. n both surveys the university dens of rts nd sciences rnked deprtment heds s the most importnt source of informtion. Although the dens rnked informl student opinion nd systemtic student rtings high in 6, five yers lter they indicted tht students were less fctor in promotions thn they hd been. Over percent of the rts nd science dens in the universities in the Astin nd Lee survey indicted tht clssroom teching nd reserch were given eqully high weight in ssessing fculty performnce. Seldin () repeted the Gustd nd the Astin nd Lee surveys in with cdemic dens in liberl rts colleges in order to exmine chnges tht might hve tken plce in the eightyer period. The dens reported more emphsis on systemtic student rtings in evluting teching (n increse from to percent of the colleges) nd slightly less emphsis on reserch in evluting overll fculty performnce. The incresed use of student rtings in mking decisions on fculty dvncement ws lso reported in studies tht included doctorllevel universities (Bejr, ; Boyd & Schietinger, 6). A Southern Regionl Eduction Bord study found, in ddition, tht the mjor purpose of evlution in doctorllevel universities ws to mke decisions on fculty dvncement, rther thn to improve instruction, nd tht deprtment heds hd greter responsibility thn cdemic dens or vice presidents in mking these decisions (Boyd & Schietinger, 6). One of the few studies tht smpled fculty perceptions of evlution procedures found tht the most influentil fctors in promotions nd tenure decisions were publictions, the deprtment hed s evlution, nd student rtings (Thorne, Scott, & Beird, 6). This study, which ws conducted t institutions in the Oregon Stte system, lso reported tht grnt support ws considered to be influentil in the physicl science deprtments. Although the pst studies point to the importnce of deprtment heds in mking decisions on dvncement, none surveyed these individuls perceptions of current prctices. With the exception of the Oregon Stte study, the erlier studies hd lso investigted dministrtors perceptions of institutionwide procedures rther thn the criteri used t the deprtment level. Fculty performnce might be viewed quite differently in different fields of study. n prticulr, the evlution of reserch nd scholrship, which the prior studies hd not looked t in detil, might be expected to vry considerbly by deprtment. Purpose of the Present Study The present study reports the importnce tht deprtment heds or chirmen give to vrious criteri in evluting individul fculty members. l n ddition to overll fculty performnce, the study discusses the chirmen s views of the use nd importnce of specific criteri for ssessing both teching nd reserch performnce. To determine the ex Both titlesdeprtment hed nd chirmnpply to the groups surveyed nd re used in this report lthough it is recognized tht the terms re not lwys synonymous. Use of chirmn rther thn chirperson or chir follows current APA style guidelines, s published in Americn Psychologist, June, p..

5 tent to which current prctices coincide with preferred prctices, respondents lso were sked to indicte wht importnce they thought should be ttched to ech criterion. Comprisons re discussed for four deprtmentl subgroups nd for three different types of universities. Procedure The questionnire used in the study (see Appendix A) prtilly overlpped the forms first used in the Gustd (6) nd Astin nd Lee (6) studies nd lter dpted by Seldin (). t differed from the forms used in the previous studies in tht () question regrding the criteri used to evlute scholrship or reserch performnce ws dded, nd () the criteri used to evlute overll fculty or teching performnce were somewht ltered or extended. Four copies of the questionnire were sent to the grdute dens of 68 universities ( percent rndom smple of the members of the Council of Grdute Schools in the United Sttes). The dens were sked to distribute the questionnires to four heds of deprtments from vriety of disciplines, giving preference to deprtments tht offered both grdute nd undergrdute degree progrms. One or more usble questionnires were received from universities prticiption rte of bout 8 percent. Questionnire returns from deprtment heds, bsed on four copies per university, mounted to bout 6 percent from ll universities tht were contcted, or 8 percent of the universities known to hve distributed the questionnires. Eight grdute dens sid they could not prticipte in the study. A totl of usble questionnires were included in the dt nlyses. For purposes of nlysis, the universities were grouped ccording to clssifiction scheme developed by the Crnegie Commission on Higher Eduction (). Three groups were formed by combining Crnegie subctegories. The first level, Reserch Universities, included () the universities tht led in federl finncil support in 6 nd tht wrded t lest doctor s degrees in 6 or (b) the institutions tht led in terms of totl number of doctor s degrees wrded during the yers 6 to. n ll cses, Ph.D. s, Ed.D. s, M.D. s on the sme cmpus, nd other doctor s degrees were counted. There were 8 deprtments from these Reserch Universities in the study. The second level, DoctorlGrnting Universities, combined two ctegories. The first ctegory consisted of institutions tht wrded or more doctorl degrees in 6 or received t lest $ million in totl federl finncil support in either 6 or. The second ctegory included institutions tht wrded t lest doctorl degrees in 6 nd few new doctorlgrnting institutions tht my be expected to increse the number of degrees wrded within few yers. There were deprtments from these DoctorlGrnting Universities in the study, 8 percent of which were in the first ctegory. lcked doctorl progrm or hd n extremely limited doctorl progrm. About 8 percent of the deprtments in this group were t institutions tht fit this description. The remining percent were t institutions tht offered liberl rts progrm nd t lest one professionl/occuptionl progrm or were specilized institutions (for exmple, schools of engineering or technology). Of the deprtments in the study, bout twothirds were t public institutions. Sixtyfour percent of these institutions offered Ph.D. or other doctorl degree, nd 6 percent offered mster s s their highest degree. Only of the deprtments offered nothing higher thn bchelor s degree. The questionnire respondents represented bout 8 different cdemic disciplines. To evlute possible differences in the responses of deprtment heds from different disciplines, the deprtments were grouped into four ctegories: humnities, socil sciences, nturl sciences, nd professionl fields. Humnities included ll lnguges nd literture, philosophy, religion, speech, fine rts, music, nd drmtic rts. Socil sciences included geogrphy, nthropology, economics, history, government, psychology, nd sociology. Nturl sciences included ll of the biologicl sciences, chemistry, physics, geology, nd mthemtics. Professionl fields included minly deprtments of eduction, engineering, nd business. n ddition, totl of bout dozen responses cme from deprtment heds in fields such s rchitecture, griculture, forestry, home economics, veterinry medicine, nd lw. Tble includes the number of deprtments in ech group nd t ech university level. There were deprt Tble. Number of Deprtments in the Study within Ech Deprtment Group nd t Ech University Level University Levels Reserch Universities DoctorlGrnting Universities Deprtment Groups Professionl Socil Nturl Fields Sciences Humnities Sciences Totl Comprehensive Universities nd Colleges 6 Totl 6 ments in the socil sciences, in the humnities, in the nturl sciences, nd 6 in the professionl fields. For ech of the three levels of universities determined by the Crnegie clssifiction scheme, there were between 6 nd deprtments in ech of the four discipline groups, thus llowing deprtment group by university level interctions to be investigted. The third level, Comprehensive Universities nd Col Ech criterion in the questionnire ws responded to on leges, consisted minly of institutions tht offered liberl rts progrm nd t lest two professionl or occuptionl progrms. Mny of them offered mster s degree, but fivepoint scle s follows: not vilble, not fctor, minor fctor, mjor fctor, nd extremely criticl fctor. The verge response for ech criterion ws computed for ech de

6 prtment group nd university level. Anlysis of vrince nd cnonicl discriminnt function nlysis were used in the study to investigte response differences. Results The results re presented in three sections: the criteri used for evluting overll fculty performnce, the sources of informtion for evluting teching, nd the kinds of informtion used for judging scholrship or reserch performnce. n ech section the dt were nlyzed by level of university nd by deprtment subgroup. Although differences for deprtment groups within ech of the university levels (deprtment by level interction) were lso investi gted, these interctions were significnt only in the evlution of scholrship or reserch performnce; cnonicl discriminnt function nlysis ws used to illustrte this interction. Criteri in Assessing Totl Fculty Performnce The questionnire included generl criteri tht might be used in evluting fculty members. These re listed in Figures nd nd in Tbles,, nd, long with the summries of the responses of deprtment chirmen. Clssroom teching, qulity of publictions, nd personl qulifictions (cdemic degrees nd professionl experience) were most frequently reported s mjor or extremely criticl Figure. Averge weight given to vrious criteri for evluting totl fculty performnce, by university level Criteri Clssroom teching** Not Minor Mjor Extremely Criticl Number of publictions** Qulity of publictions** Reserch nd/or cretive ctivity (independent of publiction) Supervision of student reserch, including serving on mster s nd doctorl committees** Student dvising* Cmpus committee work, service to college** Activity in professionl societies (hold office, edit journl, etc.) Public or community service** Competing job offers** Consulttion (government, business, etc.)* Personlity fctors Personl qulifiction (cdemic degrees, professionl experience, etc.)** *p <. **p <.Ol Key: Reserch Universities mm DoctorlGrnting Universities ~ Comprehensive Universities nd Colleges

7 fctors in judging fculty members. As indicted in Figure judged more importnt in the Reserch Universities thn t, however, deprtment chirmen t the three levels of in either of the other two university levels. Chirmen in Comstitutions gve different emphses to ech of the three prehensive Universities nd Colleges not only plced less criteri. Less weight ws plced on clssroom teching or emphsis on qulity of publictions, but lso gve it essenpersonl qulifictions in the Reserch Universities thn in tilly the sme weight s the number of publictions nd s DoctorlGrnting Universities or Comprehensive Univer college service (for exmple, cmpus committee work). sities nd Colleges. On the other hnd, the reverse ws true Other criteri in which universities differed ccording to for qulity of publictions, which, not surprisingly, ws Crnegie clssifiction were the following: supervision of Figure. Averge weight given to vrious criteri for evluting totl fculty performnce, by deprtmentl subgroup Criteri Not Minor Mjor Extremely Criticl Clssroom teching Number of publictions** Qulity of publictions** Reserch nd/or cretive ctivity (independent of publiction) Supervision of student reserch, including serving on mster s nd doctorl committees Student dvising** Cmpus committee work, service to college* Activity in professionl societies (hold office, edit journl, etc.)* Public or community service** Competing job offers** Consulttion (government, business, etc.)** Personlity fctors Personl qulifictions (cdemic degrees, professionl experience, etc.)* *p <. **p <.Ol Key: Professionl Fields = Socil Sciences ~...u Humnities = = l n Nturl Sciences 6

8 Tble. Current nd Preferred Criteri for Evluting Totl Fculty Performnce Professionl Fields Percentqe lndictin Extreme!lY Current Use nd mportnce in Personnel Ret CO mmendtion: T Criticl or Not mportnce Ech Should Hve in ersonnel Ret mmendtions Socil Sciences N=6 N= Humnities N= Nturl Sciences Professionl Fields N= N=6 Socil Sciences N= Humnities N= Nturl Sciences N= Clssroom teching Number of publictions Qulity of publictions Reserch nd/or cretive ctivity (independent of publiction) Supervision of student reserch, including serving on mster s nd doctorl committees Student dvising Cmpus committee work, service to college Activity in professionl societies (hold office, edit journl, etc.) Public or community service Competing job offers Consulttion (government, business, etc.) Personlity fctors Personl qulifictions (cdemic degrees, professionl experience, etc.) Not Not Not Ext. Ext. Ext. Crit. Fit. Crit. Fct. Crit. F:t Ext. Crit O Not F:t. Ext. Crit. Not Fit. 6 8 Ext. Crit. 8 Not F$. 6 8 Ext. Crit. 6 6 Not F:t Ext. Crit. 6 Not Fit. 6 8 student reserch (gretest emphsis in Reserch Universities), student dvising (gretest emphsis in Comprehensive Universities nd Colleges), nd competing job offers (most importnt in Reserch Universities). Figure illustrtes the differences in emphsis on ech criterion ccording to deprtmentl subgroup. Publictions (qulity nd number) were more importnt in the nturl nd socil sciences thn in the professionl schools nd the humnities. Student dvising, public service, nd consulttion were given more weight in the professionl deprtments thn in the other three deprtment groups. Current nd preferred criteri for evluting fculty members for decisions on promotions re given in Tbles nd (by deprtments) nd in Tble (by university level). The bsolute differences between current nd preferred emphses were lso investigted nd hve been tken into ccount in the following discussion. The percentges of de prtment heds reporting ech criterion s n extremely criticl fctor or not fctor re reported in Tble. As this tble indictes, there re not only vritions between the deprtment subgroups but, to some extent, within the subgroups s well. For exmple, percent of the socil science deprtment heds reported qulity of publictions to be n extremely criticl fctor nd percent reported it not to be fctor. n generl, Tble indictes tht qulity of publictions ws little more frequently chosen s criticl fctor thn teching ws in the socil sciences nd nturl sciences, while the reverse ws true in the professionl fields nd the humnities. Criteri tht chirmen preferred to emphsize more were qulity of publictions, supervision of student reserch, dvising students, nd, to some extent, unpublished reserch or cretive ctivity (Tbles nd ). The number of publictions should, ccording to the chirmen, be less criticl fctor. Severl criteri were not

9 Tble. Rnking of Current nd Preferred Criteri for Tble. Rnking of Current nd Preferred Criteri for Evluting Fculty for Promotion, Slry ncrese, or Tenure, Evluting Fculty for Promotion, Slry ncrese, or Tenure, by Deprtmentl Subgroup by University Level Clssroom teching Number of publictions Qulity of publictions Reserch nd/or cretive ctivity (independent of publiction) Supervision of student reserch, including serving on mster s nd doctorl committees Student dvising Cmpus committee work, service to college Activity in professionl societies (hold office, edit journl, etc.) Public or community service Competing job offers Consulttion (government, business, etc.) Personlity fctors Personl qulifictions (cdemic degrees, professionl experience, etc.) T Current Use nd mportnce P S i P = Professionl Fields; N = 6 (rho =.) S = Socil Sciences; N = (rho =.) H = Hummties; N = (rho =.) N = Nturl Sciences; N = (rho =.) mportnce Ech Should Hve 6 O i 6 6,.. Rnk difference correltions between current importnce nd preferred importnce re shown in prentheses. O Clssroom teching Number of publictions Qulity of publictions Reserch nd/or cretive ctivity (independent of publictron) Supervision of student reserch, including serving on mster s nd doctorl committees Student dvising Cmpus committee work, service to college Activity in professionl societies (hold office, edit journl, etc.) Publrc or community service Competing job offers Consulttion (government, business, etc.) Personlity fctors Personl qulifictions (cdemic degrees, professionl experience, etc.) T T Current Use mportnce Ech nd rr D( Mnce Should Hve ii i ii i it n r i i r O 6 Deprtments grouped by university clssifiction s follows: = Reserch Universities; N = 8 (rho =.) = DoctorlGrnting Universities; N = (rho =.) ll = Comprehensive Universities nd Colleges; N = (rho =.) Rnk difference correltions between current importnce nd preferred importnce re shown in prentheses. 6 fctors in evluting fculty performnce in mny deprtmentsin prticulr, competing job offers, consulttion, public or community service, nd personlity fctors. Moreover, none of these deserved more weight, ccording to the respondents. The respondents lso were generlly stisfied with the gret importnce given to personl qulifictions. Tble presents rnking of the current nd preferred criteri by level of university. Respondents in Reserch Universities thought tht qulity of publictions followed by clssroom teching should be the top two criteri, s indeed hs been the cse in current prctice. However, they would put less emphsis on the number of publictions nd more on reserch or cretive ctivity independent of publiction nd on student dvising, including the supervision of student reserch. Respondents in the second nd third level universities, where clssroom teching ws rnked first s current nd preferred criterion, would lso like to see student dvising nd supervision of student reserch emphsized. Evluting Teching Performnce A summry of the responses to possible sources of informtion used to evlute the teching performnce of individul fculty members ppers in Figures nd nd in Tbles, 6, nd. Chirmen evlutions, systemtic student rtings, nd collegues opinions were generlly the most influentil indictors of teching performnce. Lest used were videotpes of clssroom teching, the longterm followup of students, lumni rtings, collegue rtings bsed on clssroom visits, nd student exmintion performnce. The weight given to severl of the indictors differed by level of university (Figure ). Systemtic student rtings, for exmple, were less importnt t Comprehensive Universities nd Colleges thn t Reserch or DoctorlGrnting Universities. Dens evlutions nd selfevlutions were lest importnt t Reserch Universities. Teching improvement ctivities, such s prticiption in inservice progrms or fculty development prctices, tended to crry 8

10

11

12 Systemtic student rtings nforml student opinions Collegue rtings bsed on clssroom visits Collegue opinions Student exmintton performnce Content of course syllbi nd exmintion Chirmn evlution Den evlution Committee evlution Selfevlution or report Longterm followup of students performnce Alumni opinions or rtings Populrity of elective courses (e.g., enrollment) Videotpe of clssroom teching Teching improvement ctivities (prticiption in workshops, inservice progrms, etc.) Tble. Current nd Preferred Sources of nformtion for Evluting Teching Performnce F!,?ofessionl Fields N=6 Percentge ndicting Extremely Criticl nd Not Current Use nd mportnce T mportnce Ech Should Hve Socil Sciences N= Humnities N= Nturl Professionl Socil Sciences Fields Sciences N= N=6 N= Humnities N= Nturl Sciences N= Not Not Not Not Not Not Not Not Ext. Ext. Ext. Ext. Ext. Ext. Ext. Ext Grit. Fct. Crit. Fct. Crit. Fct. Crit. Fct. Crit. F:t. Crit. F;t. Crit. Fct Mt. F:t Tble 6. Rnking of Current nd Preferred Sources of nformtion for Evluting Teching Performnce, by Deprtmentl Subgroup Systemtrc student rtings nforml student oplnlons Collegue rtmgs bsed on clssroom visits Collegue opinions Student exmmtlon performnce Content of course syllbi nd exmlntlon Chirmn evlution Den evlution Committee evlution Selfevlution or report Longterm followup of students performnce Alumni oplnlons or rtmgs Populrity of elective courses (e g enrollment) Videotpe of clssroom teching Current Use mportnce Ech nd mportnce Should Hve S H N t j /, 6 Tble. Rnking of Current nd Preferred Sources of nformtion for Evluting Teching Performnce, by University Level Current Use nd mp ortnce All Systemtic student rtings nforml student oplnlons Collegue rtings bsed on clssroom vlstts Collegue opmlons Student exmlntlon performnce Content of course syllbi nd exmmtlons Chirmn evlution Den evlution 6 Committee evlution Selfevlution or report Longterm followup of students performnce Alumni opinions or rtings Populrity of elective Teching mprovement ctlvttles (prticiption in workshops inservice progrms etc ) P Professionl Ftelds N = 6 (rho = 86) Deprtments grouped by unlverslty clsslflctlon s follows S Socil Sciences N (rho z 88) Reserch Untversitles N 8 (rho ) H Humnities N (rho z 8) DoctorlGrnttng Unlversltles N (rho ) N Nturl Sciences N (rho = 8) ll = Comprehenslve Unlverslties nd Colleges N (rho Rnk difference correltions between current portnce re shown in prentheses mportnce Ech Should Hve it r 6 6 E O n E i ) Rnk difference correltions between current importnce nd preferred rn portnce re shown in prentheses

13 Evluting Scholrship nd Reserch Figures, 6,, nd 8 nd Tbles 8,, nd summrize the responses of deprtment heds to the kinds of informtion used in evluting scholrship or reserch performnce. n generl, the number of rticles published in qulity journls, the number of books of which the fculty member is the sole or senior uthor, nd the qulity of one s reserch nd publictions s judged by peers t the institution re the three most importnt types of informtion used. But there re importnt vritions between deprtmentl groups nd university levels. The Reserch Universities tend to put more emphsis on the judgment of peersboth from within nd outside the institutionthn do either DoctorlGrnting Universities or Comprehensive Universities nd Colleges (Figure ). Comprehensive Universities nd Colleges put the lest emphsis on most of the criteri listed, with the DoctorlGrnting Universities flling between them nd the Reserch Universities. The deprtmentl groups lso differed in how they evluted reserch nd scholrship (Figure 6). The professionl deprtments, compred with those in the rts nd sciences, plced less emphsis on peer judgments nd on the number of qulity publictions. Essentilly, however, the wy in which deprtments evluted reserch intercted with the level of the universitytht is, the deprtments differed ccording to the level of the university. This interction cn be illustrted by use of cnonicl discriminnt function nlysis of the groups (the four deprtments within ech of the three levels of universities) nd of the 6 kinds of informtion used to ssess reserch performnce. The nlysis provides n optiml discrimin Figure. Averge weight given to vrious kinds of informtion used to evlute scholrship or reserch performnce, by university level Kinds of nformtion Number of: Publictions Not in ll professionl journls Minor Mjor Extremely Criticl Articles in qulify journls** Unpublished ppers or reports Ppers t professionl meetings Cittions to published mterils** Books s sole or senior uthor** Books s junior uthor or editor* Monogrphs or chpters in books** Qulity of scholrly reserch nd publictions s judged by: Peers t the institution** Peers t other institutions** Deprtment chirmn** Selfevlutions Grnts or funding received Referee or editor of professionl journl* Honors or wrds from profession** *p <. **p <.Ol Key: Reserch Universities DoctorlGrnting Universities Comprehensive Universities nd Colleges

14 tion between the groups on the bsis of the 6 kinds of informtion, using the men responses of deprtment heds s the unit of nlysis. The results of the cnonicl discriminnt function nlysis, which re summrized in Appendix B nd in Figures nd 8, indicte two mjor dimensions of vrition mong the groups on the 6 kinds of informtion. Ech of these two dimensions ccounted for bout third of the vrince mong the groups for totl of 6 percent (Appendix B). The group centroids re plotted on the discriminnt xes in Figure. For exmple, in the figure, The discriminnt function weights given in Appendix B re difficult to interpret. Therefore, in order to better understnd the results, the betweengroups correltions between the originl responses nd the discriminnt xes were studied. These correltions hve the dvntge of being treted much like fctor lodings of observed vribles on orthogonl fctor xesthe cnonicl discriminnt functions (Cliff nd Krus, 6). identifies the position of professionl deprtments in Reserch Universities, S represents socil science deprtments in DoctorlGrnting Universities, nd so forth. n ddition to these group centroids, Figure includes 6 lines of vectors originting from the center of the plot. These 6 lines represent the wy the 6 kinds of informtion used to evlute reserch nd scholrship project into the discriminnt spce. The importnt fetures of these vectors re their directions nd their reltive lengths. By visulizing the direct perpendiculr projection of the group centroids onto these vectors, one cn get feeling for the reltive ordering mong the deprtment/university level groups s to how they evlute reserch performnce. A simplified illustrtion of the dt ppers in Figure 8. The two xes were rotted by visully estimting where they best fit the cnonicl vrites. The descriptions t the Figure 6. Averge weight given to vrious kinds of informtion used to evlute scholrship or reserch performnce, by deprtmentl subgroup Kinds of nformtion Not Minor Mjor Extremely Criticl Number of: Publictions in ll professionl journls Articles in qulity journls** Unpublished ppers or reports Ppers t professionl meetings* Cittions to published mterils Books s sole or senior uthor** Books s junior uthor or editor** Monogrphs or chpters in books** Qulity of scholrly reserch nd publictions s judged by: Peers t the institution** Peers t other institutions** Deprtment chirmn Den** Selfevlutions Grnts or funding received** Referee or editor of professionl journl Honors or wrds from profession *p <. **p <.Ol Key: Professionl Fields Socil Sciences Humnities Nturl Sciences

15 Figure. Plot of university level nd deprtmentl subgroup centroids nd the 6 kinds of informtion used to evlute scholrship nd reserch in the spce defined by the two lrgest cnonicl discriminnt xes. P Axis Selfevlution Grnts received. N. l H y Cittions Professionl wrds Ppers t professionl meetings Unpublished Monogrphs. H Books, Senior evlutions l H Key: = Reserch Universities = DoctorlGrnting Universities = Comprehensive Universities nd Colleges P = Professionl Fields S = Socil Sciences H = Humnities N = Nturl Sciences

16 Figure 8. Rotted cnonicl discriminnt xes, kinds of informtion used to evlute scholrship nd reserch performnce P Dimension Evlutions by den nd self Dimension Grnts received. N P. H. lls Peer evlutions Articles in qulity journls Professionl wrds l H l S l H Key: = Reserch Universities = DoctorlGrnting Universities = Comprehensive Universities nd Colleges P = Professionl Fields S = Socil Sciences H = Humnities N = Nturl Sciences

17 Tble 8. nformtion Used to Evlute Scholrship or Reserch Performnce r Percentge ndicting Extremely Criticl nd Not t Current Use nd mportnce r mportnce Ech Should Hve +ofessionl Socil Nturl Professionl Socil Fields Sciences Humnities Sciences Fields Sciences Humnities N=6 N= N= N= N=6 N= N= Nturl Sciences N= Number of: Publictions in ll professionl journls Articles in qulity journls Unpublished ppers or reports Ppers t professionl meetings Cittions to published mterils Books s sole or senior uthor Books s junior uthor or editor Monogrphs or chpters in books Ext. Crit. Not A Fct. Not Ext. Crit. Ftt. Ext. Crit. Not F:t. Ext. Crit. Not Fct. Ext. Crit Not Fit. Not Not Ext. Ext. Crit. F:t Crit. Fct. Ext. Crit. Not F:t Qulity of scholrly reserch nd publictions s judged by: Peers t the institution Peers t other institutions Deprtment chirmn Den Selfevlutions Grnts or funding received Referee or editor of professionl journl Honors or wrds from profession end of ech xis or dimension summrize which of the 6 vribles seem to best define the pole. The first dimension contrsts grnts received with the number of books published nd ppers presented t professionl meetings. As indicted by their position furthest out from the center, nturl science deprtments in the first two levels of universities (Reserch nd DoctorlGrnting Universities), followed t some distnce by thirdlevel nturl science deprtments (Comprehensive Universities nd Colleges) nd firstlevel professionl deprtments, were t the grnts received end of the continuum. As Figures nd 8 further illustrte, the humnities deprtments in the top two levels of institutions were t the booksppers pole. The second dimension contrsts evlutions by dens nd by the individul fculty member (selfevlution) with peer nd professionl evlutions. The ltter include judgments of reserch or publictions by peers from inside nd outside the institution, rticles published in qulity journls, wrds from the profession, nd, to lesser extent, the number of cittions to the individul s published mterils nd journl editorship. Evlutions by dens nd selfevlutions were emphsized most in professionl deprtments t second nd thirdlevel institutions s well s in humnities deprtments in thirdlevel universities. Peer nd professionl evlutions (for exmple, rticles in qulity journls) were emphsized most in socil science deprtments in firstlevel universities. To some extent, humnities nd nturl science deprtments in firstlevel universities lso emphsized peer nd professionl evlutions. n sum, the ssessment of reserch nd scholrship, s Figures nd 8 help to mke cler, vries considerbly depending on the type of deprtment nd level of institution. The current nd preferred importnce of ech criterion for ssessing reserch nd scholrship is summrized in Tbles 8,, nd. Tble 8 lso includes the percentges of deprtments tht use the criterion s n extremely criticl fctor or not t ll. The qulity of publictions s judged by peers, deprtment heds, or dens ws extremely importnt in mny deprtments, yet ws not used s criterion in mny others. For exmple, percent of the respondents in the socil science deprtments reported peers t other institutions to be criticl in judging scholrship, while percent did not cll on these peers s judges t ll. Deprtment heds in the Reserch Universities were generlly stisfied with the reltive nd bsolute weights they use in evluting reserch or scholrship. Those in the 6

18 Tble. Rnking of Current nd Preferred Criteri for Tble. Rnking of Current nd Preferred Criteri for Evluting Reserch or Scholrship Performnce, Evluting Reserch or Scholrship Performnce, by Deprtmentl Subgroup by University Level Number of: T Publictions in ll professionl journls Articles in qulity journls Unpublished ppers or reports Ppers t professionl meetings O Cittions to published mterils Books s sole or senior uthor Books s junior uthor or editor 8 6 Monogrphs or chpters in books Current Use nd m PO rtnce. 6 O 8 mportnce Ech Should Hve P S H N E 6 8 Number of: Publrctions n ll professronl journls Articles n qulity journls Unpublished ppers or reports Ppers t professionl meetings Ctttrons to published mterils Books s sole or senior uthor Books s juntor uthor or editor Monogrphs or chpters in books T Current Use nd mportnce T All iii..:. 6.: 6 mportnce Ech Should Hve.t.E Qulity of scholrly reserch nd publictions s judged by: Peers t the institution. Peers t other institutions 8 Deprtment chirmn Den Selfevlutions 6 6 Grnts or funding received Referee or editor of professionl journl Honors or wrds from profession.! P = Professionl Fields; N = 6 (rho =.8) S = Socil Sciences; N = (rho =.) H = Humnities; N = (rho =.8) N = Nturl Sciences; N = (rho =.) 8..: Rnk difference correltions between current importnce nd preferred importnce re shown in prentheses. Qulity of scholrly reserch nd publrctrons s judged by. Peers t the mstrtutron Peers t other rnstrtutions Deprtment chirmn Den Selfevlutions Grnts or funding received Referee or editor of professionl journl Honors or wrds from professron O O 6 O Deprtments grouped by umverstty clssrfrctron s follows: = Reserch Unrversrtres, N = 8 (rho ~ 8) = DoctorlGrnting Universrties; N = (rho =.) = Comprehensive Unrversrtres nd Colleges: N = (rho r 8) Rnk difference correltrons between current mportnce nd preferred rmportnce re shown n prentheses. Comprehensive Universities nd Colleges thought tht more weight should be given to judgments mde by peers t other institutions nd in the number of publictions in qulity journls. Most respondents, regrdless of level of university or deprtment, preferred to put reltively less emphsis on the sheer number of publictions in ll types of journls. The nturl science nd professionl deprtments lso preferred to put slightly less weight on grnts received thn they currently do. Discussion Teching, reserch, nd service re usully enumerted s the three functions of most universities nd, therefore, presumbly the three mjor concerns of fculty members. But the responses given by deprtment heds from universities indicte tht public or university service is generlly given little importnce in evluting fculty members for decisions regrding tenure, slry, nd promotion. The Reserch Universities, with lrge Ph.D. progrms nd hevy finncil support for reserch, emphsize reserch, s might be expected. Teching, though, ccording to the respondents, is close second in importnce. At the DoctorlGrnting Universities nd the Comprehensive Universities nd Colleges, teching ws rnked first in importnce, followed closely by reserch. Personl qulifictions, such s hving n dvnced degree nd the pproprite experience, were the only other criteri considered mjor fctors in dvncement decisions. Therefore, in ddition to community nd public service, student dvising nd service to the institution were lso seen s reltively minor fctors. n short, this study suggests tht once the pproprite credentils re in hnd, most universities nd deprtments evlute most fculty members s reserchersscholrs nd clssroom techers, with vrying emphsis depending on the level of the university or the type of the deprtment. Of course, it is importnt to note tht the chirmen were responding to the criteri s they pplied to their entire deprtment; severl pointed out tht individul

19 fculty members in the deprtment frequently hve vrying responsibilities nd tht the importnce of ny criterion depends on wht those responsibilities re. Some chirmen lso dded tht the significnce of the vrious criteri depends on the level of the dvncement in question nd tht ech cndidte ws expected to be outstnding in t lest one re (usully reserch or teching). Whether teching does, in fct, receive s much weight in promotion nd tenure decisions s chirmen report or whether they re merely pying lip service to its importnce is difficult to know. There is reson to doubt: The fculty in the previously reported Oregon Stte study (Thorne, Scott, nd Beird, 6) viewed publictions s the most influentil piece of evidence in decisions on dvncement. t my be tht reserch ctivity would hve rnked hed of clssroom teching for ll levels of universities (not just the Reserch Universities) if the questionnire hd listed single criterion (s with clssroom teching) insted of three seprte kinds of informtion (number of publictions, qulity of publictions, nd reserch n&or cretive ctivity independent of publiction). This my hve diffused the emphsis tht reserch nd scholrship ctivity ctully receives. The results of this study lso suggest n increse in the use of some of the more systemtic nd tngible kinds of evidence for evluting teching performnce. n comprison with the Astin nd Lee survey decde erlier, systemtic student rtings nd, to some extent, the content of course syllbi nd exmintions re used more frequently. Other recent studies, s discussed erlier, hve noted similr increse in the use of student rtings for tenure nd promotion decisions. Surprisingly, ccording to the responses of deprtment heds in this study, student rtings were even more importnt in the Reserch Universities nd the DoctorlGrnting Universities thn in the Comprehensive Universities nd Colleges. Possibly the lrger size of the Reserch nd DoctorlGrnting Universities fostered the use of forml student rtings s evidence of teching effectiveness. Although there ppers to be n increse in the use of t lest some substntive evidence for evluting teching, pprently its use flls fr short of wht deprtment heds think is needed. They would like to see even more emphsis on systemtic student rtings nd n evlution of the content of course syllbi nd exmintions. They lso believe tht forml collegue rtings bsed on clssroom visits nd longterm followup of how students perform should get more weight thn they currently do. But both forml collegue rtings nd the ssessment of longterm student outcomes re difficult mesures to obtin in ny systemtic nd usble wy. Collegue rtings bsed primr ily on clssroom observtion would in most instnces not be relible enough to use in mking decisions on tenure nd promotiont lest not without fculty members investing considerble time in visittions or trining sessions (Centr, ). And the ssessment of student lerning yers fter the students hve completed course, while possibly n idel indictor of the longterm effects of course or techer, is unwieldy to dminister; moreover, the effects re difficult to ttribute to specific techer. An ssessment of endofcourse lerning, lthough more mngeble, is lso subject to misinterprettion when used in tenure nd promotion decisions (Centr, ). No doubt, however, student lerning is n importnt criterion in ssessing teching, nd it my be tht collegues (or deprtment committee) could help review nd judge prepost chievement gins, student projects, or other evidence of course outcomes. n comprison with the evlution of teching, the evlution of reserch mkes possible the use of much more tngible evidence, lthough judgments still ply n importnt role in the evlution. Articles, books, wrds from the profession, nd, in the nturl sciences, grnts received re the most importnt kinds of informtion used. Qulity of reserch, ccording to the chirmen, is nd should be the primry considertion. Therefore, in most cdemic fields the number of rticles in qulity journls is of prime importnce, long with the judgments of reserch or scholrship mde by peers t the institution. n fct, chirmen feel tht more use should be mde of peers t other institutions s dditionl reviewers of the qulity of fculty member s reserch or scholrship. But in generl, the evlution of reserch nd scholrship depends very much on the type of deprtment nd the level of institution. The number of books nd ppers produced, for exmple, is especilly importnt in humnities deprtments t the top two levels of universities, but not t the third level. Peer judgments of reserch nd the number of rticles in qulity journls re importnt in socil science deprtments in the Reserch Universities but not in socil science deprtments in the Comprehensive Universities nd Colleges. These vritions point out tht the criteri used to evlute reserch nd scholrship should be set t the deprtment level rther thn t the institutionl levelor, for tht mtter, by the discipline s whole. ndeed, the sme might be sid for evluting totl fculty performnce: the individul deprtments probbly need some flexibility in setting their own criteri nd stndrds of performnce. Deprtment procedures should be considered within the generl frmework provided by university policies nd, s deprtment heds in this study indicted, should be bsed on s much hrd evidence s possible. 8

20 Bejr,.. A survey of selected dministrtive prctices supporting student evlution of instruction progrms. Reserch n Higher Eduction,,, 86. Boyd, J. E., & Schietinger, E. F. FcuZty mlution procedures in southern colleges nd universities. Atlnt, G.: Southern Regionl Eduction Bord, 6. References Astin, A. W., & Lee, C. 8. T. Current prctices in the evlu Centr, J. A. The how nd why of evluting teching. n tion nd trining of college techers. n C. B. T. Lee J. A. Centr (Ed.), Renewing nd evluting teching. Sn (Ed.), mproving college techirzg. Wshington, D.C.: Frncisco: JosseyBss,. Americn Council on Eduction, 6, 6. Cliff, N., & Krus, D.J. nterprettion of cnonicl nlysis: Rotted vs. unrotted solution. Psychometrik, 6,, s. Crnegie Commission on Higher Eduction. A clssifiction of institutions of higher eduction. Berkeley, Clif.,. Centr, J. A. Collegues s rters of clssroom instruction. Journl of Higher Eduction,, 6,. Gustd, J. W. Policies nd prctices in fculty evlution. Eductionl Record, 6,,. Seldin,. Hozo colleges evlute professors. Crotonon Hudson, N.Y.: BlythePennington, Ltd.,. Thorne, G. L., Scott, C. S., & Beird, J. H. Assessing fculty performnce: Finl project report. Monmouth, Oregon: Oregon Stte System of Higher Eduction, Teching Reserch Division, 6.

21 Appendix A Grdute Record Exmintions Bord PRNCETON, NEW JERSEY. AREA CODE 6 N AFFLATON WTH The Assocltlon of Grdute Schools The Council of Grdute Schools My 6 Richrd Ohlo Stte Armitge Unlverslty Der Collegue: My Brodbeck Unlverslty of ow Beverly Cssr Federl Ctty College Ovld Fl. Oeener Tulne Unlverstty. Wesley Elliott Fisk University Wytze Gorter University of Hwll t Mno Arthur F. Jckson North Crolin Agrmlturl nd Techncl Stte University Lyle V. Jones Umverslty of North Crolin t Chpel H Sterling M. McMurrin Uncverslty of Uth J. Boyd Pge Council of Grdute Schools (ex officio) George J. Rsnikoff Cliforni Stte University t Hywrd Lorene L. Rogers Unlverslty of Texs t Austin Ben Rothbltt Unlverslty of Chlcgo Hrry H. Slsler Unlversty of Florid Donld J. White Boston College W. Dexter Whtehed University of Virgint t Chrlottesville. Mrynn A. Ler Secretry to the Bord We re undertking survey of fculty evlution procedures nd need your ssistnce. The purpose of the survey is to investigte wht deprtments re doing nd wht they think ought to be done when evluting the performnce of fculty members. We hope you will be ble to complete the enclosed questionnire nd return it to your grdute den s soon s possible. He or she will forwrd your responses to reserchers t Eductionl Testing Service who will summrize the results. Questionnires re being distributed to smple of deprtment chirmen t bout universities. The questionnire sks bout the methods being used in your deprtment to evlute fculty for promotion, slry increse, or tenure. Plese indicte the importnce given to ech criterion when evluting performnce, including both teching nd scholrship or reserch performnce. We would lso like to know how you think vrious sources of informtion ought to be used. We recognize tht finl decisions re frequently mde beyond the deprtment level; however, recommendtions by deprtments re criticl. Therefore, this questionnire focuses on procedures tht re used to rrive t deprtmentl recommendtions. Becuse the response formt my not llow you to dequtely describe wht is done t your deprtment in fculty evlution (or wht you think should be done), we hve lso provided spce for your ddi tionl comments. The survey report will not identify individuls or institutions by nme. Thnk you for your help with the survey. A repor t of the results will be sent to you t your request nd we trust it will be of interest to you. Sincerely, cc: Mrynn A. Ler 'zs<b%. Chirmn

22 SURVEY OF DEPARTMENTAL PRACTCES N EVALUATNG FACULTY PERFORMANCE. Listed below re criteri tht might be tken into ccount in evluting fculty members for promotion, slry increse, or tenure. We would like you to give your best judgment bout: A. how importnt ech criterion is in personnel judgments currently being mde bout fculty members within the deprtment; (Column A) B. how importnt you think ech criterion should be. Tht is, given the gols of your deprtment nd institution, how much w$ht ought to be plced on ech fctor? (Column B) Respond s follows: = Not fctor = Minor fctor = Mjor fctor = Extremely criticl fctor = Not vilble Clssroom teching Number of publictions Qulity of publictions Reserch nd/or cretive ctivity (independent of publiction) Supervision of student reserch, including serving on.msters nd doctorl committees Student dvising Cmpus committee work, service to college Activity in professionl societies (hold office, edit journl, etc.) Public or community service Competing job offers Consulttion (government, business, etc.) Personlity fctors Personl qulifictions (cdemic degrees, professionl experience, etc.) Other (specify) Comments bout bove criteri: A Current use nd importnce in personnel recommendtions (Circle one response in ech row) B mportnce ech should _ hve in personnel recommendtions (Circle one response in ech row)

23 . Listed below re sources of informtion tht might be tken into ccount when evluting the teching performnce of individul fculty members in connection with promotion, slry increses, or tenure recommendtions. Wht is the current importnce of ech in your deprtment nd wht importnce do,you think ech should hve? = Not fctor = Minor fctor = Mjor fctor = Extremely criticl fctor o= Not vilble ndictors teching of performnce Systemtic student rtings nforml student opinions Collegue rtings bsed on clssroom visits Collegues' opinions Student e?mintion performnce Content of course syllbi nd exmintions Chirmn evlution Den evlution Committee Selfevlution evlution Long term follow students perform or report up of how Alumni opinions or rtings Populrity of elective courses ( e.., enrollment) Videotpe of clssroom teching Teching improvement ctivities (prticiption in workshops, inservice progrms, etc.) Others (specify) Current use nd importnce in evluting teching performnce (Circle one response in ech row) mportnce ech should hve in evluting teching performnce (Circle one response in ech row) Comments bout bove indictors:

24 . Listed below re kinds of informtion tht might be used to evlute the scholrship or reserch performnce of individul fculty members. Which ones re now used in your deprtment when mking personnel decisions? Which ones do you think should be used? = Not fctor = Minor fctor = Mjor fctor = Extremely criticl fctor = Not vilble Kinds of Number of: informtion Publictions in ll professionl journls Articles in qulity journls Unpublished ppers or reports Ppers t professionl meetings Cittions to published mterils Books s sole or senior uthor Books s junior uthor or editor Monogrphs or chpters in books Current use nd mportnce.. ech importnce in should hve in evluting scholrship evluting scholrship or reserch performnce or reserch performnce (Circle one response (Circle one response in ech row) in ech row) Qulity of scholrly reserch nd publictions s judged by: Peers t the institution Peers t other institutions Deprtment chirmn Den Selfevlutions Grnts or funding received Referee or editor of professionl journl Honors or wrds from profession Others (specify) Comments bout bove informtion: