Accounting for exogenous influences in a benevolent performance evaluation of teachers

Transcription

1 Aountng for exogenous nfluenes n a benevolent perforane evaluaton of teahers by Krstof DE WITTE Nky ROGGE Publ Eonos Center for Eono Studes Dsussons Paper Seres (DPS) July 2009

2 Aountng for exogenous nfluenes n a benevolent perforane evaluaton of teahers Krstof De Wtte and Nky Rogge 1 * ( ): Katholeke Unverstet Leuven (KULeuven) Faulty of Busness and Eonos Naasestraat 69, 3000 Leuven (Belgu); ( ): Maastrht Unversty TIER, Faulty of Eonos and Busness Kapoenstraat 23, ML 6200 Maastrht (the Netherlands) Krstof.dewtte@eon.kuleuven.be and (*): Hogeshool-Unverstet Brussel (HUBrussel) Centre for Eonos & Manageent (CEM) Storstraat 2, 1000 Brussels (Belgu) July 2009 Abstrat Students evaluatons of teaher perforane (SETs) are nreasngly used by unverstes and olleges for teahng proveent and deson akng (e.g., prooton or tenure). However, SETs are hghly ontroversal anly due to two ssues: (1) teahers value varous aspets of exellent teahng dfferently, and, to be far, (2) SETs should be deterned solely by the teaher s atual perforane n the lassroo, not by other nfluenes (related to the teaher, the students or the ourse) whh are not under hs or her ontrol. To aount for these two ssues, ths paper onstruts SETs usng a speally talored verson of the popular non-paraetr Data Envelopent Analyss (DEA) approah. In partular, n a so-alled Beneft of the doubt odel we aount for dfferent values and nterpretatons that teahers attah to good teahng. Wthn ths odel, we redue the pat of easureent errors and a-typal observatons, and aount expltly for heterogeneous bakground haratersts arsng fro teaher, student and ourse haratersts. To show the potentalty of the ethod, we exane teaher perforane for the Hogeshool Unverstet Brussel (loated n Belgu). Our fndngs suggest that heterogeneous bakground haratersts play an portant role n teaher perforane. Keywords: Teaher perforane, Data envelopent analyss, Condtonal effeny, Eduaton. JEL-lassfaton: C14, C25, I21 1 Correspondng author. Tel.: ; fax: E-al address: Nky.Rogge@hubrussel.be 1

3 1. Introduton Students evaluatons of teahng (SETs hereafter) are nreasngly used n hgher eduaton to evaluate teahng perforane. Yet, for all ther use, SETs ontnue to be a ontroversal top wth teahers, prattoners, and researhers sharng the onern that SET sores tend to be unfar as they fal to properly aount for the pat of fators outsde the teaher s ontrol. The reason for ths onern s twofold. On the one hand, there are the nuerous fndngs n the aade lterature whh suggest that one or ore bakground ondtons (e.g., lass sze, teaher gender, teaher experene, ourse grades, tng of the ourse) ay have a sgnfant nfluene on SET sores (see, for nstane, Brnbau, 1977; Cashn, 1995; Centra and Gaubatz, 2000; d Appollona and Abra, 1997; Feldan, 1997; Marsh, 1980, 1983, 1984, 1987, 2007; Marsh and Rohe, 1997, 2000; Sth and Knney, 1992). On the other hand, there s the pratal experene fro teahers theselves whh ndates that soe teahng envronents are ore onstrutve to hgh-qualty teahng (and, hene, hgh SET sores) whle other envronents ake suh a level of teahng less evdent. Ths potental unfarness n nd, several researhers have argued for a autous nterpretaton of SET sores. Baldwn and Blattner (2003) and Abra and d Apollona (1999), for nstane, reoended to base an analyss of teaher perforane not solely on SET sores (or rankngs). In ther opnon, SETs should be opleented wth the fndngs fro other evaluaton nstruents (suh as peer evaluatons, lass-roo vstatons). Soewhat surprsngly, only few researhers have argued n favour of atually adjustng SET sores for bakground varables (e.g., Wrght et al., 1984; Greenwald and Glore, 1997; Eery et al., 2003; Daves et al., 2007; Law and Goh, 2003). Eery et al. (2003, p. 44), for nstane, note that Any syste of faulty evaluaton needs to be onerned about farness, whh often translates nto a onern about oparablty. Usng the sae evaluaton syste [wthout properly aountng for the dfferenes n teahng ondtons] for everyone alost guarantees that t wll be unfar to everyone. Stated dfferently, unadjusted SET sores are potentally flawed and, therefore, unrelable as a easure of teaher perforane. Typally, proposed orreton proedures onsst out of three stages. In a frst step, SET sores are oputed wthout ontrollng for the nfluene of bakground varables. There are several ways to derve suh unontrolled SET sores fro questonnare data. One possblty s to opute an arthet ean of the ratngs on the questonnare tes. A soewhat slar approah onssts of sung the ratngs and expressng the as a perentage to the axal attanable overall ratng (e.g., Law and Goh, 2003). A thrd way s askng students to rate the overall perforane of the teaher on one sngle sale (e.g., Ells et al and Daves et al. 2007). In a seond phase, the pat (both n ters of sze and dreton) of one or ore bakground haratersts on the SET sores s deterned. Agan, several approahes are possble: a orrelaton analyss, a (ultvarate) analyss of varane, a (ultple) regresson analyss, or a ultlevel odelng approah. The approah ost frequently used s the ultple regresson analyss where the SET sore are regressed on several bakground haratersts (e.g., Law and Goh, 2003; Ells et al., 2003). 2

4 In a thrd and fnal step, the SET sores are adjusted for these nfluenes. Generally, ths nvolves developng a sple statstal proedure to orret ntal sores for the unfarness assoated wth bakground varables. For nstane, n an evaluaton of 165 behavoral and soal senes ourses letured at Mnot State Unversty between 1997 and 1998, Ells et al. (2003) found a sgnfant and postve orrelaton between SET sores and ean student grades. To adjust the sores for ths nfluene, the researhers developed the followng forula: Adjusted Ratng = y + ( y y ) wth y the average ratng gven to all ourses n the saple, y the orgnal unadjusted ratng, and y the average ratng for teahers wth the sae average ourse grade. A soewhat slar proedure was followed by Law and Goh (2003) and Daves et al. (2007). It s portant to note that a orreton of SET sores for the nfluenes of bakground haratersts s rather an exepton than the rule. Most studes only exane the pat of bakground varables (.e., step 1 and 2). As these papers ay be useful to poston our results, we outlne a suary of ther results below n Table 1. In lne wth the lterature, we lassfy bakground varables under three headngs: nstrutor haratersts (e.g., teaher age, experene, gender, dotoral degree, pedagogal tranng), student haratersts (e.g., (ean) student grades, the heterogenety of the students, questonnare response rate), and ourse haratersts (e.g., lass sze, the tng of the ourse). In short, results are rather xed. The sze and dreton of the assoatons see to be dependent on the rustanes, the ontent, the speftes of the onsdered teahng evaluaton nstruent, and the ethodology used to exane the relatonshps (e.g., ultlevel odelng versus regresson analyss). We beleve that there are two ssues why three-step proedures should be approahed wth auton. A frst ssue arses fro the oputaton of the SET sores n the frst step. In partular, t s oon prate to alulate sores as an arthet ean or as a su of the ratngs on questonnare tes (eventually expressed as a perentage to the axal attanable overall ratng). Essentally, ths ples that all teahng aspets are assued to be of equal portane. Whether suh equal weghts (and, n general, any set of fxed weghts) are approprate s questonable. Indeed, there are soe ndatons suggestng that equalty of weghts aross teahng aspets and/or over teahers s undesrably restrtve (e.g., Prthard et al., 1998, p.32), 3

5 Instrutor gender Teaher age and experene Pedagogal tranng Teaher Rank (guest/part-te vs. full-te) Table 1: Correlatons between bakground haratersts and SET sores Teaher-related haratersts Sgnfant orrelaton Insgnfant orrelaton Hgher SETs for feales: Kashak (1981); Hgher SETs for ales: Feldan (1992); Gender nteraton: Basow et al. (1987), and Basow (2000) (2003) Postve: MPherson (2006), Sth et al. (1992), d Appollona et al. (1997), Wagenaar (1995); Negatve: Baek et al. (2008), and Cohran et al. (2003); Nonlnear relatonshp: Langben (1994) Postve: Wagenaar (1995), Nasser et al. (2006), Full-te teahers wth lower SETs: Agner et al. (1986) Dotoral degree Negatve: Cohran et al. (2003), Nasser et al. (2006) Basow et al. (1985), MKeahe (1979), Cashn (1995), Fernandez et al. (1997), Hanok et al. (1992), Marsh et al. (1997), Ells et al. (2003), and Law et al. Feldan (1983), Law et al. (2003), Ells et al. (2003), and Koh et al. (1997) Cranton et al. (1986), Delaney (1976), Chang (2000), Stener et al. (2006), and Wllet (1980) Chang (2000) Student-related haratersts Sgnfant orrelaton Insgnfant orrelaton Student grades Postve: Greenwald et al. (1997), Langben (1994), Baek et al. Deano (1986), Abra et al. (1980) (2008), MPherson (2006), Isely et al. (2005), Marsh et al. (1997, 2000), Grffn (2001, 2004), Feldan (1997), Marsh (1980, 1983, 1984, 1987), et. Student Negatve: Dreeben et al. (1988), heterogenety Questonnare response rates Tng (2000), and Perry (1997) Postve: Koh et al. (1997) Negatve: MPherson (2006) Isely et al. (2005) 4

6 Table 1 (ontnued) Class sze Te of day Course-related haratersts Sgnfant orrelaton Insgnfant orrelaton Negatve: Law et al. (2003), Koh Feldan (1984), and Marsh et al. et al. (1997), Baek et al. (2008), (1997) Langben (1994), d Apollona et al. (1996), Deano (1986); Nonlnear: Chau (1997), and Marsh et al. (1992) Lower SETs n afternoon or evenng: DeBerg et al. (1990), Badr et al. (2006), Hanna et al. (1983); Hgher SETs n afternoon or evenng: Isely et al. (2005), Cranton et al. (1986) Stener et al. (2006), Koh et al. (1997), Law et al. (1997), and Husbands et al. (1993) As an llustraton of the latter, teahers value teahng aspets dfferently n the defnton (and, thus, the evaluaton) of exellent teahng. 2 These dfferenes ould be expeted gven the dfferent personaltes and abltes of teahers. Hene, usng fxed weghts n the buld-up of SET sores ay be soewhat ounterntutve. Moreover, n the absene of a onsensus on how teahng aspets exatly nterrelate, any hoe of fxed weghts wll be subjetve to soe extent. The use of fxed weghts an also ntrodue unfarness n teaher evaluatons. Indeed, fxed weghts ay favour teahers who perfor well on aspets that reeve hgh weghts, whle dsfavourng teahers who exel on aspets wth low assgned weghts. Unsurprsngly, dsllusoned teahers wll nvoke ths unfarness and the subjetvty n weght hoe to underne the redblty of the SET sores. Last but not least, teahers only get lted nforaton out of suh an arthet average, the essental reason beng that t s not at all lear what sores presely ply. Only when onstruted and nterpreted n a relatve perspetve to the perforanes of olleagues are SET sores eanngful. A seond ssue whh questons the auray of a three-step proedure s related to the plt separablty assupton. In partular, t s pltly assued that there s no dret lnk between the set of attanable SET sores and the teahng envronent (as easured by bakground varables related to the teaher, the students and the ourse). Spefally, the onstruton of SET sores and the study of the pat of bakground haratersts our n two separate analyses. Ths separablty ondton s probleat as both researh evdene (Cashn, 1995; d Appollona and Abra, 1997; Feldan, 1997; Marsh, 1984, 1987, 2007; Marsh and Rohe, 2000; et.) and pratal experene suggest a sgnfant dret nfluene of the pedagogal ondtons on teahng. It s therefore rual to the aurateness and redblty of SET sores to onsder the teahng envronent straghtforwardly n the oputaton of SET sores. 2 Illustratve are the strong nter-ndvdual dsagreeents often observed n the opnon of teahers on the approprate weghts. Only rarely do teahers assgn slar (fxed or equal) weghts. 5

7 The urrent paper ontrbutes to the lterature n that t learly devates fro the urrent ethodologes to (1) onstrut, (2) adjust and (3) analyze SET sores. Frstly, onsder the onstruton of SET sores. In ontrast to the tradtonal three-step approahes, we propose a speally talored verson of the Data Envelopent Analyss ethodology (DEA). The DEA odel has been developed by Charnes et al. (1978) as a non-paraetr (.e., t does not assue any a pror assupton on the produton fronter) tehnque to estate effeny of observatons. In the urrent paper, we do not apply the orgnal DEA odel, but rather an alternatve approah whh orgnates fro DEA. Ths so-alled beneft of the doubt (BoD) odel explots the haraterst of DEA that t, thanks to ts lnear prograng forulaton, allows for an endogenous weghtng of ultple outputs/aheveents (Melyn and Moesen, 1991). We desgn the BoD odel suh that t allows for easureent errors whh arrve fro the survey data. In partular, we apply nsghts fro the robust order- effeny sores of Cazals et al. (2002) to our spef BoD settng. As suh, the BoD odel has three ajor advantages. Frstly, for eah teaher perforane under evaluaton, the weghts on the questonnare tes are hosen n a relatve perspetve suh that the hghest possble SET sore s realzed. Therefore, teahers wth one or ore low SET sores an no longer blae these poor evaluatons to unfar weghts. Seondly, the BoD odel s flexble to norporate stakeholder opnon (e.g., teahers, students, experts) n the onstruton of the SET sores. Aong others, Prthard et al. (1998) strongly argued n favour of developng an evaluaton syste wth suh sgnfant and eanngful stakeholder (partularly the teahers) partpaton. In ther opnon, suh nvolveent s a neessary ondton for the redblty and aeptane of the evaluaton results. Thrdly, the robust spefaton of the BoD odel allows us to aount for outlyng and wrongly easured questonnare values. As a seond ontrbuton, we allow for envronent adjusted SET sores wthout assung a separablty between the teaher s perforane and the exogenous nfluenes. To do so, we further extent the robust (.e., the adapton of Cazals et al. (2002) to allow for easureent errors) BoD odel of Melyn and Moesen (1991) to the ondtonal effeny estates of Darao and Sar (2005, 2007a, 2007b). The latter non-paraetr tehnque allows us to nlude teaher, student and ourse related nfluenes edately n the effeny sores. Ths avods the ltatons of the prevously desrbed three-step proedure. A fnal ontrbuton s stuated at the analyss level of the effeny sores. By applyng the bootstrap based p-values of De Wtte and Kortelanen (2008), we an exane non-paraetrally the dreton of the nfluene of exogenous varables on the SETs. Ths s partularly onvenent beause t allows us to nterpret the fators whh reate low or hgh SET sores. To llustrate the pratal usefulness of the approah, we apply the odel on a dataset olleted at the Hogeshool Unverstet Brussel (Belgu) n the aade year Ths rh set oprses data on 16 questonnare tes (easurng several aspets of teaher perforane) and 11 bakground varables (.e., teaher age, teaher experene, teaher gender, tenure status, pedagogal tranng, dotoral degree, ean lass grade, student nequalty, questonnare 6

8 response rate, lass sze, and tng of the ourse). The results reveal the portane of norporatng exogenous haratersts. The reander of the paper s organzed as follows. Seton 2 desrbes the data. In the thrd seton we present bas DEA odel as well as ts robust and ondtonal extenson. We outlne how to enfore a seleton of approprate aggregaton weghts for teahng aspets, to enhane the robustness of SET sores, and to aount for bakground haratersts. Seton 4 reports the results. In the fnal seton, we offer soe onludng rearks and soe avenues for further researh. 2. The data We estate teaher perforane as easured by the perforane of a teaher on a spef ourse. In partular, we explore a detaled saple on 112 ollege ourses (=1,,112) taught by 69 dfferent teahers. Teahers who leture several ourses wll therefore have for several teaher perforane sores (SET-sores),.e. one for eah evaluated ourse. 3 These ourses were taught n the Coeral Senes and Coeral Engneerng progras at the Unversty College Brussels (HUB; a ollege n Belgu) n the frst and seond seester of the aade year Durng the last two weeks of these seesters 5,513 students were questoned. The questonnare oprsed 16 stateents to evaluate the ultple aspets of teaher perforane. Students were asked to rate the leturers on all tes on a fve-pont Lkert sale that orresponds to a odng rule rangng fro 1 (I opletely dsagree) to 5 (I opletely agree). To faltate the students understandng of the questons, stateents foussng on slar aspets of the teahng atvty were grouped nto key densons: Learnng & Value, Exanatons & Assgnents, Leture Organzaton, and Indvdual Leturer Report (For a detaled desrpton of the HUB-questonnare, see Rogge, 2009). The developent of the questonnare as well as the ategorzaton of the tes nto these key densons was largely based on a study of the ontent of effetve teahng, the spef ntentons of the evaluaton nstruent, and revews of prevous researh and feedbak. 5 For eah ourse (=1,,112) we alulate an average student ratng y, for eah questonnare te ( = 1,, 16): y = y,,, s s ourse j ( 1) 3 Beause the unt of observaton s the ourse, haratersts spef to the ndvdual student (e.g., gender, years n ollege) annot be nluded n the analyss. 4 At HUB, SETs are olleted to provde feedbak to teahers for provng teahng perforane and a easure of teahng qualty for personnel desons. 5 Based on a lterature revew, Marsh and Dunkn (1992, p. 146) onlude that ths approah s ore oonly used rather than statstal tehnques suh as fator analyss or ulttrat-ultethod analyss. 7

9 where y,, s denotes the appreaton on queston of student s for the teaher who s leturng ourse. All S students regstered for ourse (.e., s ourse ) and present at the oent of the questonnare are onsdered n the oputaton of the lass ean ratng. 6 To exane the effets (both n ters of dreton and sgnfane) of bakground haratersts on SET sores, the questonnare data are suppleented wth adnstratve data on several haratersts related to the teaher, the group of students and the ourse. Exept for the age of the teaher, all other teaher-related haratersts (the teaher gender, whether or not the teaher has less than 2 years of experene, whether or not he/she s a guest leturer, whether or not the teaher reeved pedagogal tranng n the past, and whether or not he/she has a dotoral degree) are duy varables. A duy varable of 1 stands for, respetvely, a feale teaher, a new teaher wth less than two years of experene, a guest leturer, reeved already pedagogal tranng, and has a dotoral degree. 7 Further, we nlude three bakground haratersts related to the students: the atual ean grade of the students n the lass, the nequalty of the dstrbuton of the student grades (as easured by the Gn oeffent whh an vary between 0 and 1, wth a Gn oeffent of 0 ndatng a perfetly equal dstrbuton and a Gn of 1 desgnatng the exat opposte), and the response rate to the questonnare. The latter aptures the rato of the nuber of people who answered the teaher evaluaton questonnare (.e., S ) to the (offal) lass sze. Fnally, two haratersts related to the ourse are nluded n the analyss: the lass sze and a duy ndatng whether the ourse s letured n the evenng. Suary statsts for the data on bakground haratersts are presented n Table 2. 6 Note that the nuber of students partpatng n the teaher evaluaton, S, an be lower than the offal lass sze as students an be absent durng the adnstraton of the questonnares. 7 Aountng for teaher haratersts s eanngful as students ay have strutural preferenes on gender or guest leturers. Moreover, n ths partular applaton, aountng for a dotoral degree s neessary as ths s only a reent requreent for HUB teahers (although also before teahers wth PhD where hred). 8

10 Teaher haratersts Table 2: Desrptve statsts on teaher, student, and ourse haratersts Mean Stdev Mn Max - Gender (Duy: 1: Feale, 0: Male) 0 (86) 1 (26) - Age Experene < 2 years (Duy: 1: Yes, 0: No) 0 (89) 1 (23) - Guest leturer (Duy: 1:Yes, 0: No) 0 (84) 1 (28) - Pedagogal Tranng (Duy: 1:Yes, 0: No) 0 (81) 1 (31) - Dotoral degree (Duy: 1:Yes, 0: No) 0 (60) 1 (52) Student haratersts - Mean lass grade (sore fro 0 to 20) Inequalty n grade dstrbuton (Gn oeffent) Response rate (%) 61.82% 21.29% 15.63% % Course haratersts - Class sze Evenng ourse (Duy: 1:Yes, 0: No) 0 (90) 1 (22) As Table 2 ndates, 86 on a total of 112 ourses were letured by ales; the age of the teahers vared between 27 years and 62 years; roughly 1 out of 5 ourses were letured by teahers havng less than 2 years of teahng experene; 28 of the 112 evaluated ourses were taught by guest leturers; respetvely 31 and 52 ourses were nstruted by teahers who reeved pedagogal tranng n the past and by teahers who have a dotoral degree. Note that there s a relatvely large proporton of ourses letured by teahers wthout a dotoral degree as ths s only a reent requreent to teah at HUB (although, also before PhD were teahng ourses). As for the student haratersts, the ean lass grade was about The average nequalty n the dstrbuton of student grades as easured by a Gn oeffent was wth standard devaton of Ths ndates that, on average, students grades see to be dstrbuted rather equal. Nevertheless, as ndated by the axu observed Gn oeffent of 0.306, there were notable exeptons to ths general pattern. The average response rate was roughly 62%, wth 80 out of 112 letures havng a response rate of ore than 50%. As we do not observe a systeat pattern n students who dd not respond, we onlude that our saple s unbased. As for the 9

11 ourse-related haratersts, lass sze ranged fro 2 to 222 students wth a ean of approxately 49 students ourses were letured durng the evenng. 3. Methodology 3.1 The Beneft of the Doubt odel To estate SET, we use a non-paraetr odel whh s rooted n Data Envelopent Analyss (DEA), an effeny easureent tehnque orgnally developed by Farrell (1957) and put nto prate by Charnes et al. (1978). In essene, DEA s a lnear prograng tool for evaluatng the relatve effeny of a set of slar enttes (e.g., frs, ndvduals) gven observatons on (possbly ultple) nputs and outputs and, often, no relable nforaton on pres. DEA does not requre any a pror knowledge on the funtonal for of the produton or ost funton. Before ntrodung the odel nto dept, note that the oneptual proble of DEA s slar to the SET proble. Slar as n DEA, we have to onstrut SET sores based on a large array of sngle-densonal perforane ndators (wth = 1,!, q ). Slarly, we have a pror no prese understandng on the exat portane of eah of these ndators. In fat, n oparson to DEA, the only dfferene s that the onstruton of SET sores only requres a look at the aheveents (thus, onsderng the outputs wthout expltly takng nto aount the nput denson). Forally, n the DEA settng, all evaluated enttes are assued to have a duy nput equal to one. 9 Ths onept was frst developed by Melyn and Moesen (1991). They labelled the resultng odel Beneft of the Doubt (BoD), a label that orgnates fro one of the rearkable features of DEA: the use of an endogenous weght seleton proedure n the aggregaton (Cherhye et al., 2007). The an oneptual startng pont of BoD estators (and, thus, fro DEA where they are rooted n), s that nforaton on the approprate weghts an be retreved fro the observed data theselves (.e., lettng the data speak for theselves). In partular, the bas dea s to put, for eah questonnare te, the perforane of a teaher on hs/her ourse y, n a relatve perspetve to the other teaher/ourse perforanes y j, (where y j, denotes the perforane on the questonnare te n all ourses j ( j 1,,,, n) =!! n the referene set ϒ ). A good relatve perforane of the evaluated teaher on a spef questonnare te ndates that ths teaher onsders ths aspet as relatvely portant. Aordngly, ths aspet should weght ore heavly 8 One ould argue for gnorng ourses wth a lass sze lower than 10 or 15 students (.e., Feldan, 1977 and Hobson and Talbot, 2001). However, our oputatons revealed that the pat of suh ourses on the results s only argnal. 9 The ntutve nterpretaton (see, aongst others, Lovell et al., 1995 and Cook, 2004) for ths fous ay be obtaned by sply lookng upon ths spef verson of the DEA-odel as a tool for suarzng perforanes on the several oponents of the evaluated phenoenon, wthout explt referene to the nputs that are used for ahevng suh perforanes. 10

12 n the teaher s perforane evaluaton. As a result, a hgh weght s assgned. The opposte reasonng holds for the teahng aspets on whh a teaher perfors weakly opared to the other olleagues n the oparson set. In other words, for eah teaher separately, BoD (and thus also DEA) looks for the weghts that axze the pat of the teaher s relatve strengths and nze the nfluene of the relatve weaknesses. As a result, BoD-weghts w, are optal n the sense that they are hosen n suh a way as to axze the teaher s SET sore SET ( ) 2007): 12 y. 10,11 Ths an be forally translated n the lnear prograng set-up (Cherhye et al.,,, w, = 1 q, j,! = 1,, ( ) SET ( y) = ax w y 2 s. t. q ( ) ( ) w y 1 j = 1,...,,, n n ϒ 2a ( ) ( ) w 0 = 1,..., q 2b w W = 1,..., q and e E. 2 e Thus, n the absene of any detaled nforaton on the true weghts, BoD assues that representatve weghts an be nferred fro lookng at the relatve strengths and weaknesses. Ths ndeed eans that the eah teaher s granted the beneft-of-the-doubt when t oes to assgnng weghts n the buld-up of hs/her SET ( ) y s (.e., one for eah evaluated ourse). Note that n ths BoD odel, teahers are granted onsderable leeway n the defnton of ther ost favourable weghts w,. In fat, optal weghts only need to satsfy two nor onstrants: the noralzaton onstrant ( 2a ) and the non-negatvty onstrant( 2b ). The frst restrton poses that no other teaher perforane present n the saple ϒ an have a SET sore hgher than unty when applyng the optal weghts w, of the teaher perforane under evaluaton. The seond onstrant states that weghts should be non-negatve. Hene, SET ( y) s a nondereasng funton of the perforanes on the several stateents (wth = 1,!, q ). Apart fro these restrtons, the foral odel ( 2) ( 2b) allows weghts to be freely estated n order to 10 For opleteness, we enton that BoD alternatvely allows for a worst-ase perspetve n whh enttes reeve ther worst set of weghts, hene, hgh (low) weghts on perforane ndators on whh they perfor relatve weak (strong) (Zhou et al., 2007). 11 Ths BoD odel s frst appled on the level of the four key densons before aggregatng the four resultng denson sores nto an overall SET sore. 12 Ths adjusted odel s forally tantaount to the orgnal nput-orented CCR-DEA odel of Charnes et al. (1978), wth all questonnare tes onsdered as outputs and a duy nput equal to one for all observatons. 11

13 axze SET ( ) y. Ths large freedo n weght hoe an be seen as an advantage as t enables teahers to put theselves n the best possble lght relatve to ther olleagues. Dsllusoned teahers an no longer blae a low SET sore to a harful or unfar weghtng shee. Any other weghtng shee than the one spefed by the BoD odel would worsen the SET sore. However, ths flexblty also arres soe potental dsadvantages as t ay allow a teaher to appear as a brllant perforer n a anner that s hard to justfy. For nstane, there s nothng that keeps BoD fro assgnng zero or quas-zero weghts to oponents of teahng (.e., questonnare tes ) on whh the teaher perfors poorly opared to the olleagues, thereby negletng those aspets n hs or her assessent. For exaple, n an extree senaro, all the relatve weght ould be assgned to a few questonnare tes, whh would then opletely deterne the SET sore. Further, there s the potental proble that the BoD odel ay selet weghts that ontradt pror stakeholder vews (e.g., students, teahers, pedagog experts, faulty board). To avod suh probleat weght senaros (zero or unrealst weghts), frequently, addtonal weght restrtons are ntrodued n the bass odel to enfore the nstallaton of proper weghts. Forally, the onstrant ( 2 ) s added wth W denotng the set of perssble weght values defned based upon the opnon of seleted stakeholders e E. In our applaton, we used a Budget Alloaton Method to ollet both student and teaher opnons on the approprate weghts. 13,14 Based on ther spefed weghts, we defned weght restrtons applyng to both the questonnare tes as well as the key densons. 15 Fro restrton ( 2 a), we an dedue that, for all evaluated teaher perforanes SET (=1,,n), SET ( ) y wll le between 0 and 1 wth hgher values ndatng a better relatve teahng perforane. In fat, ths onstrant hghlghts the relatve perspetve (.e., benharkng dea) of BoD: the ost favourable weghts for the evaluated teaher perforane w, are always appled to all n perforanes n the oparson set ϒ. One s n that way effetvely lookng whh of the teaher perforanes n ths saple are worse, slar or better. If SET ( ) 1 y <, ths ndates that the teaher ould perfor better on ourse. Indeed, there are other teahers n the saple ϒ who realze hgher SET sores even when applyng the evaluated teaher s ost favourable weghts w, (.e., weghts whh are less favourable than ther own optal BoD weghts). In ths stuaton, a strong ase an be ade for the noton that ths teaher 13 In prate, both a group of students and teahers were ontated and requested to share ther pereptons on the portane of the dfferent densons and tes nluded n the questonnare. 14 The ndvdual stakeholder opnons, as olleted by a Budget Alloaton Method, as well as a detaled desrpton of the weght restrtons are avalable fro the authors upon request. The Budget Alloaton Method s a partpatory ethod n whh stakeholders have to dstrbute 100 ponts over the tes alloatng ore to what they regard to be the ore portant tes. 15 See Rogge (2009) for a oprehensve dsusson of the stakeholder opnons and the weght restrtons. 12

14 perforane on ourse s of lower qualty. Only f ( y) = 1 SET, the teaher letures the ourse, relatve to the other evaluated ourses, n the best way (.e., he/she ats as hs/her own benhark). That s, he/she s not outperfored by other observatons j ( j 1,,,, n) applyng hs/her best possble weghts w,. =!! when 3.2 The robust BoD odel The orgnal BoD odel of Melyn and Moesen (1991) s deternst n the sense that t does not allow for outlyng observatons (e.g., arsng fro easureent errors). The latter observatons ould heavly dsturb the evaluaton sores. By adaptng the BoD odel to the robust evaluaton sores (also known as order-) of Cazals et al. (2002) we allow for easureent errors. Basally, the order- approah redues the pat of easureent errors by drawng repeatedly (.e., B tes) and wth replaeent observatons fro the orgnal saple of n (=112 n the urrent applaton) observatons. As outlned n Cazals et al. (2002), we draw only fro those observatons whh are obtanng hgher perforane sores Y than the evaluated observaton y, (.e., observatons for whh yeld that y, Y ). We label ths saller referene set as b, ϒ (wth b=1,,b). For eah of the B draws, the BoD-based SET sores are oputed relatve to ths subsaple of sze : = 1,, q b, b, =,,, ϒ w, = 1 SET ( y) ax w y y 3 s. t. q, j, b, ( ) ( ) w y 1 j = 1,..., ϒ 3a ( ) ( ) ( ) w 0 = 1,..., q 3b w W = 1,..., q and e E. 3 e Havng obtaned the B SET-sores, we opute the outler-robust BoD estate of SET as the arthet average of the B SET b, ( y) draws: 1 SET y SET y B b, ( ) = ( ) (4) B b = 1 In ontrast to the tradtonal BoD SET ( y ) sores, the robust SET ( y ) sores an be larger than unty. Indeed, thanks to drawng a subsaple of observatons wth replaeent fro the full saple for whh yeld that y, Y referene saple, the evaluated observaton wll not always be part of the b, ϒ. As suh, super-effent (.e., observatons wth a SET ( ) y sore hgher than 1) ould arse. The super-effent SET ( y ) sore s nterpreted as a teaher who s dong better than the average other teahers n ts referene saple. 13

15 Followng Darao and Sar (2005, 2007a, 2007b), we estate the value of as the level for whh the perentage of super-effent observatons dereases only argnally. Indeed, f s sall the probablty of drawng the evaluated observaton s rather low, and onsequently, we wll observe ore super-effent observatons. If, the robust sore onverges to the tradtonal BoD sore (.e., SET ( y) SET ( y) ). In our applaton, we seleted =50. Jeong et al. (2008) show that the order- estates have attratve propertes n that they are onsstent and have a fast rate of onvergene. Although these attratve propertes were derved for the orgnal DEA odel, the extenson to the BoD approah s rather straghtforward. 3.3 The robust and ondtonal BoD odel As already ndated by Cazals et al. (2002), and as developed by Darao and Sar (2005, 2007a, 2007b) for ontnuous varables and by De Wtte and Kortelanen (2008) for xed (.e., both dsrete and ontnuous) varables, the order- sores an be easly adapted to norporate the exogenous envronent (represented by R bakground haratersts z 1,...zR ). Whereas the robust order- BoD estates SET ( y ) are obtaned by drawng at rando and wth replaeent observatons (fro those observatons for whh yeld y, Y ), the ondtonal order- BoD estates are obtaned by drawng wth replaeent but wth a partular probablty observatons (fro those observatons for whh yeld y, Y and z, r Z ). In, z partular, we draw the referene group ϒ fro those observatons whh have the hghest probablty of beng slar to the evaluated observaton (slar n ters of the teahng envronent n whh the evaluated ourse was letured). The latter ondton orresponds to ondtonng on the exogenous haratersts z, r (.e., the teaher-related, student-related and ourse-related bakground haratersts as dsussed n Table 2). To do so, we sooth the exogenous haraterst Z by estatng a kernel funton around estate the BoD odel wth respet to the adapted referene set z, r. 16 Slar as before, we ϒ, z. The obtaned estates, labeled as SET ( y z ), are robust to outlyng observatons (e.g., arsng fro easureent errors) and nlude n one step the heterogenety Z arsng fro teaher, student and ourse haratersts. 3.4 Statstal nferene As a ajor advantage, the ondtonal order- BoD estates SET ( y z) allow us to exane the dreton of the effet on SET of the exogenous haratersts. In partular, the rato of the 16 Reark that one should use the approprate kernel for, respetvely, dsrete and ontnuous varables (De Wtte and Kortelanen, 2008). 14

16 ondtonal [.e., aounted for heterogenety; SET ( y z ) ] to the unondtonal [.e., wthout aountng for the envronent; SET ( y ) ] order- estates an be regressed on the ondtonng fator Z (Darao and Sar, 2005, 2007a, 2007b). Besdes a vsualsaton (whh we do not present here), a non-paraetr bootstrap proedure an be appled to obtan statstal nferene on the dreton of the effet. Inspred on the Darao and Sar (2005) fraework, we use a non-paraetr bootstrap to exane the effet of Z on the rato SET ( y z ) / SET ( y ) (see L and Rane (2007) for the bootstrap proedure). De Wtte and Kortelanen (2008) showed by sulaton that ths approah enables one to estate standard errors and p-values of the sgnfane of the nfluene of Z. Thanks to ths statstal nferene, we an explore whh teaher, student and ourse related varables have a sgnfant pat on the BoD estates. 4. Results Before estatng the robust and ondtonal BoD odel, we exane the tradtonal unondtonal BoD odel SET ( y ) (ths orresponds to the odel n Subseton 3.1). The results, presented n Table 3, reveal that the average BoD sore s rather hgh. The average unondtonal SET-sore of 0.83 ndates that, f all teahers would perfor on the four underlyng densons as well as the best perforng teaher, they ould, on average, nrease ther SET sores by 17%. Wthout aountng for exogenous haratersts, there s only one ourse evaluated as outstandng n all four key densons. As suh, the overall teaher perforane on ths ourse s evaluated exellent (hene, reevng the axal SET ( y) sore equal to 1). Table 3: BoD estates for three odel spefatons Denson 1 Denson 2 Denson 3 Denson 4 Aggregate BoD Learnng and value Exanatons and Assgnents Leture organzaton Indvdual Leturer report Unondtonal BoD odel Average St. Dev Mn Max Condtonal BoD odel 1 Average St. Dev Mn Max Condtonal BoD odel 2 Average St. Dev Mn

17 Max Condtonal BoD odel 3 Average St. Dev Mn Max On the level of the key densons, perforanes are, on the average, hgher on the densons Leture Organzaton and Indvdual Leturer Charatersts. Generally speakng, students pereve the requreents and agreeents onernng the exa evaluaton as nsuffent lear (.e., denson Exanatons & Assgnents obtans the lowest average perforanes). Both patterns are also observed n the other ondtonal BoD odels. More nterestng than the tradtonal SET ( y) -estates s the ondtonal odel (as dsussed n Subseton 3.3) n whh we aount for the R exogenous fators Z arsng fro teaher, student and ourse haratersts. As presented n Table 3 and 4, we estate three alternatve odel spefatons. Whereas Table 3 reports soe desrptve statsts on the effeny sores, Table 4 desrbes the nfluenes (favorable or unfavorable to the robust SET ( y) -sores and the orrespondng p-values) of the exogenous varables Z. If we aount for exogenous haratersts, the average teaher evaluaton sore nrease. The average teaher ould, f he/she would teah n a slar way as hs/her best prate teaher, nrease hs/her overall SET ( y z ) by 14%. Table 4: Statstal nferene of the BoD estates Model 1 Model 2 Model 3 Influene p-value Influene p-value Influene p-value Teaher haratersts Pedagogal tranng Favorable *** Favorable ** Favorable *** Havng a PhD Favorable *** Favorable Favorable Guest leture Unfavorable ** Unfavorable ** Age Unfavorable Favorable Student haratersts Mean Grade Unfavorable *** Unfavorable *** Unfavorable ** Gn of sores Favorable Course haratersts Class sze Favorable *** Evenng ourse Unfavorable *** Unfavorable *** R² where ***, ** and * denote, respetvely, sgnfane at 1, 5 and 10% level. 16

18 As a frst lass of varables, onsder the pat of the teaher haratersts. In the three odel spefatons, we observe a favorable and sgnfant pat of pedagogal tranng on the SET ( y z ) sores. In other words, teahers who followed a pedagogal tranng reeve hgher SET sores. Wagenaar (1995) and Nasser et al. (2006) report slar results. Seondly, aordng to the frst odel spefaton, havng obtaned a PhD has a favorable nfluene on SET ( y ). The latter observaton ontrast to prevous paraetr fndngs of Cohran et al. (2003) and Nasser et al. (2006). However, the two alternatve BoD odels fnd, n lne wth Chang (2000), an nsgnfant nfluene of a PhD degree. Thrdly, guest teahers see to be less appreated. Ths negatve assoaton ontrasts to prevous paraetr fndngs of Agner et al. (1986) (partte teahers are rated ore favourably) and Cranton et al. (1986), Delany (1976), Chang (2000), Stener et al. (2006), and Wllet (1980) (who found an nsgnfant effet). Fnally, age, gender and experene (ore or less than two years experene) do not sgnfantly hange the BoD sores (although the nsgnfant alternatve odels are not reported here). Ths s n lne wth the fndngs of soe prevous paraetr studes (e.g., Law et al., 2003; Ells et al., 2003; Feldan, 1993). However, as presented n Table 1, soe of these studes also obtaned opposte results (.e., postve or negatve sgnfant orrelatons). As a seond lass of exogenous varables, onsder the nfluene of student haratersts. Frstly, we observe a sgnfant negatve relatonshp between the ean grade of the lass and the SET sores. Ths ndates that teahers who are gradng ore generously do not obtan better students evaluatons. Although ths ontrasts to general beleves (see Table 1), t an be ntutvely explaned. Indeed, underperforng teahers ay ark ore generously to proptate ther students (for a teahng perforane of lower qualty). Seondly, teahers leturng for a ore heterogenous group of students do not obtan dfferent SET sores (.e., student heterogenety has an nsgnfant effet on SET sores). Whether ths result ontradts the fndngs of Dreeben et al. (1988), Tng (2000), and Perry (1997) s unknown as n none of these studes student heterogenety was easured by the Gn oeffent of the dstrbuton of the grades. Thrdly, the questonnare response rate does not have a sgnfant effet on SET ( y z ). Ths result onfrs the fndng of Isely et al. (2005), but ontradts the results of Koh et al. (1997) and MPherson (2006) who found, respetvely, that the questonnare response rate s postvely and negatvely related to the SET sores. As a thrd and fnal lass of exogenous varables, we onsder two ourse haratersts: lass sze and tng of the ourse (.e., durng dayte or n the evenng). Teahers who are teahng n larger lasses are evaluated by the students as sgnfantly better. Although ths postve assoaton ontradts prevous fndngs n the lterature (e.g., Law et al., 2003; Koh et al., 1997; Baek et al., 2008; Langben, 1994; d Apollona et al., 1996; and Crttenden et al., 1975; et.), t s probably an endogenous fndng as the shool anageent assgns the largest groups to the (n ther opnon) best teahers. Ths onfrs prevous fndngs of teahers of relatvely larger lasses beng evaluated ore postvely (e.g., Chau, 1997; Marsh and Rohe, 1997; Marsh and Dunkn, 1992; and Wood et al., 1974). Slar to the fndngs of DeBerg et al. (1990), Badr et al. (2006) and Hanna et al. (1983), we fnd that ourses taught n the evenng are less appreated by the students. Ths ontradts general beleves. As Table 1 shows, prevous studes reported 17

19 postve assoatons (Isely et al., 2005 and Cranton et al., 1986) or non-sgnfant orrelatons (e.g., Husbands et al., 1993, Law et al., 1997, et.). It s portant to note that our study s, due to data onstrants, lted for the reason that t does not opute SET sores that are orreted for all bakground haratersts whh, n the lterature, have been found to nfluene teaher perforane. As prevous researh (see, aong others, Greenwald and Glore, 1997; Marsh and Rohe, 2000; Grffn, 2001, 2004; et.) has suggested, other varables (e.g., student gender, pror nterest n the ourse, ourse workload, et.) ght also affet SET sores. 5. Conluson To be far, students evaluatons of teaher perforane (SETs) should be deterned solely by the teaher s atual perforane n the lassroo, not by other bakground nfluenes (related to the teaher, the students or the ourse) whh are not under hs or her ontrol. Unfortunately, any epral studes ndated that SET sores apture also the effets of suh bakground fators. Ths paper has proposed a speally talored verson of the Beneft of the Doubt (BoD) odel (whh s rooted n the popular non-paraetr Data Envelopent Analyss (DEA) approah) to (1) onstrut SET sores, (2) adjust the for the pat of bakground varables, and (3) analyze the pat of these varables on the SET sores. In oparson to the oon prate of buldng SET sores as an arthet average of the ratngs on the questonnare tes and analyzng the pats of bakground varables on these sores (only rarely SET sores are atually adjusted for these nfluenes) n separate steps, ths approah has several advantages. Frstly, for eah teaher under evaluaton, the weghts on the questonnare tes are hosen n a relatve perspetve suh that the hghest possble SET sore s realzed. Therefore, teahers wth one or ore low SET sores an no longer blae these poor evaluatons to unfar weghts. Seondly, the BoD odel s flexble to norporate stakeholder opnon (e.g., teahers, students, experts) n the onstruton of the SET sores. Clearly, ths nvolveent s benefal for the redblty and aeptane of the evaluaton results. Thrdly, the BoD odel s extended to onstrut robust SET-sores. Ths advantage s partularly useful as questonnares ay ontan soe easureent errors or atypal observatons. Fourthly, BoD an be further developed to aount for several bakground varables (dsrete and ontnuous) wthout assung a separablty between the teaher s perforane and these exogenous nfluenes. As a fnal result, ths yelds envronent adjusted robust and optal SET sores n lne wth stakeholder opnon. To analyze non-paraetrally the exat pat (both n ters of dreton and sze) of the bakground varables on SET sores, we appled the bootstrap based p-values of De Wtte and Kortelanen (2008). Ths s partularly onvenent beause t allows us to nterpret the bakground fators whh reate low or hgh SET sores. The results ndate that, on average, slghtly hgher ratngs are gven to teahers who (a) follow a pedagogal tranng, (b) have a dotoral degree, () are only atve at the unversty, (d) are less generously n arkng, (e) leture for larger lasses, and (f) leture durng dayte. Alternatve exaned bakground 18

20 haratersts (.e., teaher age, teaher experene, teaher gender, student nequalty, and questonnare response rate) dd not sgnfantly nfluene the teaher perforanes. Both the exstene and strength of the relatonshps between bakground varables and SET sores vares wthout doubt wth the partular (exogenous) rustanes and ondtons. Therefore, t would be nterestng for future researh to apply the proposed ethodology n several evaluaton settngs to hek for reurrng patterns n the results. In the sae ven, t would be nterestng to apply our non-paraetr ethod to the data of prevous studes to opare the results. If dfferent results would be obtaned, at frst sght, the results of our ethod ould be preferred as no a pror assuptons are requred. Another suggeston would be to expand our study wth other bakground varables that have been found to orrelate wth SET sores n the lterature (e.g., student gender, pror nterest n the ourse, ourse workload, et.). Further, although not beng a onsderaton of ths paper, we stress the portane of studyng the exat ehanss by whh aforeentoned bakground varables nfluene SET sores n ore detal. However, as the lterature reports on xed fndngs, t s very lkely that spefyng suh ehanss wll turn out to be partularly oplex. Or, n the words of Feldan (1998, p. 43): In prnple, and learly n prate, the searh for the ondtons and ontexts that deterne the exstene, strength, dreton, and pattern of assoatons between varables of nterest s an on-gong searh and probably a never-endng one. Lterature Abra, P. C., & d Apollona, S. (1999). Current onerns are past onerns. Aeran Psyhologst, 54(7), Agner, D.J., & Thu, F.D. (1986). On student evaluatons of unversty teahng. The Journal of Eono Eduaton, 17(4), Badr, M.A., Abdulla, M., Kaal, & M.A., Dodeen, H. (2006). Identfyng potental basng varables n n student evaluaton of teahng n a newly aredted busness progra n the UAE. Internatonal Journal of Eduatonal Manageent, 20(1), Baek, S.-G., & Shn, H.-J. (2008). Multlevel analyss of the effets of student and ourse haratersts on satsfaton n undergraduate lberal arts ourses. Asan Paf Eduaton Revew, 9(4), Baldwn, T., & Blattner, N. (2003). Guardng aganst potental bas n student evaluatons: What every faulty eber needs to know. College Teahng, 51(1), Basow, S.A., & Dstenfeld, M.S. (1985). Teaher Expressveness: More Iportant for Male Teahers Than Feale Teahers? Journal of Eduatonal Psyhology, 77, Basow, S.A., & Howe, K.G. (1987). Evaluatons of College Professors: Effets of Professors' Sex-Type, and Sex, and Students Sex. Psyhologal Reports, 60, Basow, S. A. (2000). Best and worst professors: Gender patterns n students'hoes. Sex Roles: A Journal of Researh, 43(5/6),

21 Brnbau, R. (1977): Fators Related to Unversty Grade Inflaton, Journal of Hgher Eduaton, 48(5), pp Cashn, W. E. (1995). Student ratngs of teahng: the researh revsted. IDEA Paper Nr.32. Cazals, C., Florens, J.P., & L. Sar (2002). Nonparaetr Fronter Estaton: A Robust Approah. Journal of Eonoetrs, 106 (1), Centra, J.A., & Gaubatz, N.B. (2000): Is there gender bas n student evaluatons of teahng, Journal of Hgher Eduaton, 71(1), pp Chang, T.-S. (2000). Student Ratngs: What Are Teaher College Students Tellng Us about The? Paper presented at the Annual Meetng of the Aeran Eduatonal Researh Assoaton (New Orleans, LA, Aprl 24-28, Charnes, A. Cooper, W.W., & Rhodes, E. (1978). Measurng the effeny of deson akng unts. European Journal of Operatonal Researh, 2, Chau, C.-T. (1997). A bootstrap experent on the statstal propertes of students ratngs of teahng effetveness. Researh n Hgher Eduaton, 38(4), Cherhye, L., Moesen, W., Rogge, N., & Van Puyenbroek, T. (2007). An ntroduton to beneft of the doubt oposte ndators. Soal Indators Researh, 82, Cohran, H.H. Jr., Hodgn, G.L., & Zetz, J. (2003). Student Evaluatons of Teahng: Does Pedagogy Matter? Journal for Eono Eduators, 4(1), Cook, W.D. (2004). Qualtve Data n DEA. In W.W. Cooper, L. Seford, and J. Zhu (Eds.), Handbook on Data Envelopent Analyss, Kluwer Aade Publshers, Dordreht, Cranton, P., & Sth, R. (1986). A New Look at the Effet of Course Charatersts on Student Ratngs of nstruton. Aeran Eduatonal Researh Journal, 23(1), Crttenden, K.S., Norr, J.L., & LeBally, R.K. (1975): Sze of Unversty Classes and Student Evaluaton of Teahng, Journal of Hgher Eduaton, Vol. 46, No. 4, pp D Apollona, S., & Abra, P.C. (1996). Varables oderatng the valdty of student ratngs of nstruton: A eta-analyss. Paper presented at the 77th Annual Meetng of the Aeran Eduatonal Researh Assoaton. D Appollona, S., & Abra, P.C. (1997). Navgatng student ratngs of nstruton. Aeran Psyhologst, 52(11), Darao, C., & Sar, L. (2005). Introdung Envronental Varables n Nonparaetr Fronter Models: A Probablst Approah. Journal of Produtvty Analyss, 24 (1), Darao, C., & Sar, L. (2007a). Advaned robust and nonparaetr ethods n effeny analyss: Methodology and applatons. Seres: Studes n Produtvty and Effeny, Sprnger. 20