Online Posting of Teaching Evaluations and Grade Inflation. Talia Bar University of Connecticut. Vrinda Kadiyali Carnell University

Transcription

1 Online Posting of Teaching Evaluations and Grade Inflation Talia Bar University of Connecticut Vrinda Kadiyali Carnell University Asaf Zussman Hebrew University of Jerusalem Working Paper September Fairfield Way, Unit 1063 Storrs, CT Phone: (860) Fax: (860) This working paper is indexed on RePEc,

2 Online Posting of Teaching Evaluations and Grade Inflation * Talia Bar, Vrinda Kadiyali, and Asaf Zussman Abstract: In 2008 the faculty senate of Cornell University s College of Agriculture and Life Sciences (CALS) decided to publish mean student evaluations of teaching online. The stated goal of the policy was to provide useful information to students as they design their program of study. Using data from CALS we study the effects of this policy change on teaching evaluations, grading outcomes and students course choices. Identification relies on the differential response of initially-low and initially-high rated instructors. While student evaluations of teaching increased, the policy change contributed to grade inflation and had little effect on course enrollment and composition. * We are grateful to administrators at Cornell s Office of the University Registrar and the College of Agriculture and Life Sciences for providing us with the data, to Haim Bar and the staff at Cornell s Statistical Consulting Unit for their assistance in preparing the data for analysis, and to Ken Couch, Delia Furtado, Steve Ross, David Simon and colleagues at Cornell University and the University of Connecticut for valuable comments. Assistant Professor, Department of Economics, University of Connecticut. 365 Fairfield Way Unit 1063 Storrs, CT talia.bar@uconn.edu. Nicholas H. Noyes Professor of Management & Professor of Marketing and Economics, Johnson School of Management, Cornell University. kadiyali@cornell.edu. Senior Lecturer, Department of Economics, Hebrew University of Jerusalem. asaf.zussman@mail.huji.ac.il. 1

3 1. Introduction In many institutions of higher education, student evaluations of teaching (SET) survey results are a primary measure of teaching effectiveness and are often relied upon in tenure and promotion decisions. Some colleges and universities have started making available online summary information from SET surveys 1, and others are considering this move. Proponents of making SET information public suggest that this will help students select courses that better serve their educational needs, reduce reliance on less dependable sources of information on teaching quality such as ratemyprofessor.com, and serve to recognize excellence in teaching. In February 2008 the faculty senate of the College of Agriculture and Life Sciences (CALS) at Cornell University approved legislation making the students numerical responses to end-of-semester student appraisal of courses and teachers surveys publicly available to members of the Cornell community. As stated in the college web site, the goal for enacting these new rules is to provide useful information to students as they design their program of study at Cornell. 2 The online posting of student evaluations began in the fall semester of We use data from CALS to study the effects of this policy change on teaching evaluations, grading outcomes and students course choices. We are particularly interested in understanding the relationship between student evaluations of teaching and grading outcomes, as grade inflation and high grade 1 For example, Carnegie Mellon University (see Michigan State University (see and the University of Arizona (see using tces course selection). 2 See staff/course evaluations/. 2

4 levels have been subjects of concern and public debate in recent decades. 3 It has long been hypothesized that student evaluations of teaching contribute to grade inflation (see, e.g. Rosovsky and Hartley, 2002). Studies, both observational and experimental, have established that students tend to award professors with higher evaluations when they expect to receive higher grades. Surprisingly, despite the fact that this creates an incentive for instructors to inflate grades, the issue has received little empirical scrutiny. The policy change at Cornell allows us to help fill the gap. We conjecture that the online posting of SET survey results put pressure on instructors to improve their ratings and that this pressure was especially strong for instructors who initially (pre-policy change) had relatively low scores. Our analysis shows that following the policy change, teaching evaluations increased much more for initially-low rated instructors than for their initially-high rated peers. This improvement was likely at least in part due to instructors enhanced teaching efforts (for example, the SET survey results suggest that instructors increased their availability for consultations with students). At the same time, the analysis also suggests that the online posting of evaluations affected grading outcomes, with grades increasing the most in courses taught by initiallylow rated instructors. There is no evidence that the policy change affected course enrollment or composition (average SAT score, share of female students, and share of students taking the course as a requirement). Thus, the policy change does not seem to have affected selection of students into courses. Additional analysis reveals that taking a course with a highly evaluated instructor does not improve students performance in subsequent courses (a commonly used measure of learning). Taken 3 For discussion, see Achen and Courant (2009), Bar, Kadiyali and Zussman (2009 and 2012), and Butcher, McEwan, and Weerapana (2014). 3

5 together, the evidence is therefore consistent with the hypothesis that professors responded to the policy change by inflating students grades. The analysis in this paper builds on and contributes to several lines of research. Much of the empirical literature on SET surveys focuses on documenting and explaining the observed positive correlation between evaluations and (actual or expected) student grades. 4 Many of the studies are observational and thus do not provide credible identification of causal effects. A small set of experimental studies offer more convincing evidence on how grades affect student evaluations. Johnson (2003) provides both a thorough review of this literature and arguably the most influential and compelling experimental results documenting a causal positive effect of student grade expectations on student evaluations of teaching. The results are based on a large-scale field experiment conducted at Duke University in the academic year. More recently, Butcher, McEwan and Weerapana (2014) have shown that the adoption of an anti-grade-inflation policy at Wellesley College led, among other things, to a decline in student evaluations of teaching. In this paper we build on these results in arguing that to the extent that instructors are aware of the evaluations-grades relationship (which seems likely), they face an incentive to inflate grades. Another important set of related studies aims at determining to what extent SET survey scores reflect teaching effectiveness and student learning. A leading approach within this literature has been to investigate the effect of instructor SET ratings in a given course on student grades in future courses. For example, Carrell and West (2010) use data from the U.S. Air Force Academy, where students are randomly assigned to professors in core courses. They show that student 4 Love and Kotchen (2010) offer a theoretical model linking students evaluations of teachers and grades. 4

6 evaluations of teachers in these courses do not have a statistically significant effect on student performance in follow-on required courses. Research by Braga, Paccagnella and Pellizzari (2011), using data from Bocconi University, yields a similar conclusion. Consistent with these findings, in this paper we find no evidence that higher evaluations are associated with improved student learning as measured by performance in future courses. 5 This supports our interpretation that the policy change at CALS contributed to grade inflation. Our paper studies the effects of online posting of results from SET surveys. This is a policy question that today concerns many institutions of higher education. While such a policy has already been implemented in several schools, to the best of our knowledge its effects have not been studied formally. We hope that the results presented in this paper will find their way into the policy debate. The rest of the paper is organized as follows. In the next section we provide details on the data and methodology used in the analysis. Results are presented in Sections 3 and 4. Section 5 concludes. 2. Data and methodology We use data on courses taught at CALS between 2003 and 2011 and on students taking them. The online posting of student evaluations of teaching began in the fall semester of 2008, so our data spans multiple semesters before and after the policy change. The college provided us with responses to student evaluations of teaching 5 An alternative approach to measure student learning has been to be use outcomes from a centrally graded test. See, for example, Beleche, Fairris and Marks (2012). 5

7 surveys for each course. We merged this information with data provided by the university registrar on the courses and on the students taking them. Course level data include the department the course was offered in, year, semester, course title, course number, enrollment, and instructor name/s (which we replaced with an instructor identification number to protect anonymity). Student level information includes a student identification number (appropriately disguised to preserve anonymity), gender and course grade. We also obtained students SAT scores, which enable us to better assess students' inherent abilities. 6 Responses to SET surveys are anonymous. This implies that we cannot match the registrar student level data with student responses to the surveys. Thus, our main analysis is conducted at the course level. We aggregated survey responses and registrar student level data by course to generate mean evaluation, mean grade and other course characteristics. Later, we use student level data in examining the relation between SET ratings and teaching effectiveness. For our analysis we kept only courses with a single professor whose name is known. The unit of observation is a combination of course, instructor and semester, e.g. course J, taught by professor K, in semester t. Keeping only courses that were taught by the same instructor before and after the policy change, we have 735 course-instructor-semester observations. Table 1 provides summary statistics for these observations. Professors at CALS seem to be well liked by students, with an average of 4.33 (out of 5) in overall course rating and 4.15 (out of 5) in overall course rating. The college is highly selective: the average (combined math and verbal) SAT score in our sample is 1,350. On average, there are about 42 students in each course, of which 58% are female. 6 SAT scores are available for only a subset of the students. 6

8 [Table 1] Our study focuses on student responses to two key questions that were asked in the same format in all the surveys administered during the period under investigation. The first question is This teacher deserves an overall rating of, with possible responses ranging from 1 Very poor instructor to 5 Excellent instructor. The second question is This course deserves an overall rating of, with possible responses ranging from 1 Very poor course to 5 Excellent course. Additional analysis examines responses to questions that pertain to more specific aspects of teacher and course quality. Student responses to the surveys were collected either on paper ( bubble sheets ) or online. In 2003 all surveys were administered by paper; by 2011 more than 70 percent of the surveys in our data were administered online. 7 To examine the effects of the policy change on teaching evaluations, grades, and course enrollment and composition, we take a difference-in-differences approach, comparing outcomes before and after the implementation of the policy for instructors who differ in their initial evaluations. We first compute the average mean (across students) evaluation for each instructor-course combination across all its pre-policy change observations (this is done separately for the teacher rating and for the course rating). We then estimate the following model: _ (1) є where is the outcome for course-instructor i in semester t, is an indicator taking the value of 1 in all the post-policy change semesters (fall 2008 and later), _ is the initial evaluation measure discussed above, is a course- 7 Our results are robust to controlling for the type of SET form. 7

9 instructor fixed effect, is a linear time trend, is a vector of time-varying control variables at the course-instructor level (detailed below), and є is a well-behaved error term. 8 Our main interest is in the coefficient, which captures the differential effect of the policy on instructors with different initial evaluations. Note that the coefficient captures the effect of the policy change on instructors with a (hypothetical) initial evaluation rating of zero. In the first set of models, our dependent variable is the mean overall rating of the teacher (or the course) in the SET surveys. In these models the vector includes the mean SAT score of students in the course, the (natural) logarithm of enrollment, the share of female students in the course, and the share of students who reported that their reason for taking the course is that it is required. We include these variables in the model as they may affect teaching evaluations. For example, it is possible that, all else being equal, higher enrollment is associated with lower evaluations. Our hypothesis is that the policy change provided an increased incentive for instructors to obtain high ratings on the SET surveys. Instructors can improve their ratings with increased teaching effort (coming more prepared to class, offering more office hours, etc.) but also by inflating grades. We hypothesize that the policy change exerted the strongest positive effect on the teaching evaluations and on the grading outcomes of the initially low-rated instructors, i.e. we expect β 0 and β 0. Using similar models we then analyze the effects of the policy change on course enrollment and composition. This serves two purposes to examine whether the stated goal of the policy to inform and influence student choices was achieved and 8 Note that since we have a fixed effect for each course instructor combination, the model cannot include the un interacted initial evaluation variable. 8

10 to isolate the mechanisms for the policy-induced grade changes. In Section 4 we continue the effort of isolating the mechanisms behind the change in grading outcomes by exploring the ability of SET ratings to predict student grades in subsequent courses. 3. Effects of the policy change We start with a preliminary look at the data to motivate the econometric investigation. We then present results from regression analyses testing the effects of the policy change on teaching evaluations, mean grades, course enrollment and course composition A preliminary look at the data Figures 1-3 illustrate our main findings. To generate the figures we use the value of the _ variable described above (for the overall teaching rating question) to divide instructor-course combinations into two groups: those with above median value are labeled initially low-rated teachers, while the others are labeled initially high-rated teachers. In Figure 1, Panel a illustrates the evolution over time of mean evaluations for initially-low versus initially-high evaluation professors while Panel b compares pre- and post-policy change mean evaluations for the two groups. Similarly, Figure 2 illustrates the effects of the policy on mean grades, and Figure 3 illustrates the effects of the policy on course enrollment. 9 [Figure 1-3] 9 The patterns observed in Figures 1 3 are robust to dividing instructor course observations using the overall ratings of courses rather than of teachers. 9

11 Figure 1 clearly shows that following the implementation of the policy, the evaluations of initially low-rated teachers increased, while those of the initially high-rated teachers displayed a minor decline. Figure 2 demonstrates that the policy change was accompanied by an increase in grades assigned by initially low-rated teachers; in contrast, there was no change in the grades assigned by initially highrated teachers. Finally, Figure 3 suggests that there was no change in course enrollment following the implementation of the policy for either group of teachers. These patterns are consistent with our hypotheses regarding the effects of posting SET survey results online. We now turn to an econometric investigation of these hypotheses Teaching evaluations We use Equation (1) to examine how the introduction of the policy affected the overall teacher and course ratings. Results for the overall teacher ratings are shown in Table 2. The first column presents the results from estimating a baseline model. As hypothesized, the coefficient of the indicator (β ) is positive while the coefficient of the interaction term _ (β ) is negative. Both coefficient estimates are highly significant. The values of the coefficients can be used to compute the estimated effects of the policy change for instructors in different parts of the initial evaluation distribution. For example, an instructor at the 25 th percentile of the distribution ( _ =4.094), experienced an estimated rating increase of ( ) while an instructor at the median of the distribution ( _ =4.350) experienced an estimated rating increase of In contrast, an instructor at the 75 th percentile of the distribution ( _ =4.633) 10

12 experienced a rating decrease of The decrease in rating for the top prepolicy change instructors could be due to relative evaluations: as initially low-rated instructors improve, their initially high-rated colleagues no longer seem as exceptional. 11 [Table 2] In column (2) we add a linear time trend to the baseline model. The results indicate the existence of a positive trend; at the same time, the coefficients of interest ( and ) maintain their size and statistical significance. In columns (3)- (6) we gradually add the time-varying controls at the course level (i.e. the elements of the vector ): students mean SAT score, log enrollment, the share of female students and the share of students taking the course as a requirement. 12 This has only a minor effect on the magnitude and statistical significance of the coefficients of interest; none of the added controls is significantly associated with teaching evaluations. These results are robust to clustering of standard errors at the instructor-course level (see Table A1 in the Appendix). Table 3 presents results from the analogous analysis for overall course evaluations. The results are very similar to those presented in Table 2, which is not surprising given that mean teacher and course evaluations are highly correlated (0.88). Unlike the case for teacher evaluations, for course evaluations we find a significant association with the time-varying course characteristics: evaluations are 10 Formal tests show that the first two results are highly significant (p values of 0.000) while the third one is only marginally significant (p value of 0.085). 11 The results are robust to the inclusion of lagged instructor evaluation in the regression. The coefficients of interest, β and β, maintain their size and statistical significance, reassuring us that the results are not driven by mean reversion. 12 Note that moving from column (2) to column (3), the number of observations drops. This is due to the unavailability of SAT scores for a subset of students. 11

13 lower when enrollment is higher and when the share of students taking the course as a requirement is higher. [Table 3] Four versions of the SET surveys were administered at CALS during the period studied here. Our measures of overall teacher and course quality reflect responses to the two key questions that appeared in all surveys. The different SET surveys contained a large number of additional questions that concerned specific aspects of teacher and course quality. We next use again Equation (1) to examine how the online posting of survey results affected responses to 18 such additional questions. 13 The list of questions and key regression results are reported in Table 4. [Table 4] The results presented in Table 4 are similar to those shown in Tables 2 and 3. In all cases the coefficient of the indicator ( ) is positive while the coefficient of the interaction term _ ( ) is negative; in the vast majority of cases the coefficients are highly statistically significant. This pattern is not surprising since the additional questions examined here are strongly correlated with overall teacher and course ratings (the correlations range from 0.54 to 0.94) Student grades To examine the effects of the policy change on mean course grades we again estimate Equation (1). Baseline results are presented in column 1 of Table 5. Similar 13 Over time, some survey questions were changed, added or removed. To maintain consistency, we matched the surveys as best as we could. For example, in one version of the survey, students had to respond to the statement The instructor seemed to dislike students and in another they had to respond to the statement The instructor seemed to like students. Responses ranged from 1 completely disagree to 5 completely agree. In such instances we reversed the scale for the negatively worded statement. 12

14 to results for teaching evaluations, we find that the coefficient of the indicator is positive and the coefficient of the interaction term _ is negative. In courses taught by an instructor at the 25 th percentile of the distribution of initial evaluations, grades increased by ( ) following the policy change. This is about 15% of a standard deviation of mean grades in the sample. Similarly, grades increased by in courses taught by an instructor at the median of the distribution ( _ =4.349) and by 0.01 in courses taught by an instructor at the 75 th percentile of the distribution ( _ =4.629). 14 [Table 5] In columns (2)-(6) we add to the baseline specification a linear time-trend and the time-varying course characteristics. The results suggest the existence of a positive time trend and that mean course grades are positively correlated with mean SAT scores and negatively correlated with enrollment. The magnitudes of the main coefficients of interest, and, change somewhat, but maintain their signs and statistical significance. Results are almost identical when initial ratings pertain to the course instead of the teacher (see Table A2 in the Appendix). We also examine the effect of the policy change on the share of grades that are either very high (A+) or very low (C+ or below). On average, in our sample 9.4% of the grades were A+ and 7% of the grades were C+ or below. In Table 6 we show the results of estimating Equation (1) using these shares as the dependent variables. Consistent with the results presented in Table 5, we find that the share of very high 14 Formal tests show that the first two results are statistically significant (p values of and 0.044, respectively) while the third one is insignificant (p value of 0.558). 13

15 grades increased by more and the share of very low grades decreased by more for initially low-evaluation instructors. [Table 6] The results presented so far suggest that by putting differential pressure on initially low-rated instructors, the policy change implemented by CALS led to improved evaluations but also to higher student grades for this group of instructors relative to others. How should one interpret this pattern? One possibility is that the pressure led the initially low-rated instructors to inflate grades (or lower standards), for which they were rewarded by higher student evaluations. A second possibility is that the pressure led the initially low-rated instructors to exert more effort: evaluations rose because these instructors became better teachers; grades rose because student learned more. A third possibility is that the implementation of the policy induced changes in course enrollment and composition which led to an increase in both evaluations and grades in courses taught by initially low-rated instructors. Of course, a combination of these reasons could have led to the observed effects. In the next sub-section we examine the course enrollment and composition explanation. In Section 4 we examine the improved learning explanation Course selection Recall that the stated objective of the policy change was to provide useful information to students as they design their program of study at Cornell. If students respond to the online posting of SET survey results when making their course selection, we might expect enrollment to change. Highly-rated professors (or those with improving ratings) might attract more students. If a student s benefit from taking a class with a professor whose evaluations are high or improving depends 14

16 on the student s ability, gender, or reason for taking the course, then we might also observe changes in course composition in terms of the mean SAT score, share of female students or share of students taking the course as required. To study the effects of the policy change on course enrollment and composition we estimate a version of Equation (1) which excludes the vector. Course enrollment and each of the various measures of course composition (the elements of ) are now the dependent variables. The results are in Table 7, where in columns (1)-(4) initial rating is measured using the overall evaluation of the teacher while in columns (5)-(8) it is measured using the overall evaluation of the course. We find no evidence that the policy change affected enrollment, mean SAT score, share of female students or the share of students for whom the course is required. These results have two implications. First, they suggest that the stated goal of the policy, which was to inform and influence student choices, does not seem to have been achieved. Second, they allow us to rule out the third explanation highlighted above for the simultaneous relative increase in evaluations and grades for the initially low-rated instructors, which focused on changes in course enrollment and composition. 15 [Table 7] 4. Evaluations and future grades In this section we look for evidence for the second explanation highlighted above for the simultaneous relative rise in evaluations and grades for the initially low-rated instructors that it due to increased teaching effectiveness. As noted in 15 This conclusion is further supported by the fact that neither enrollment nor course composition is associated with the overall evaluation of teachers (see Table 2). 15

17 the introduction, several existing studies use performance in future courses as a measure of teaching effectiveness in contexts where student assignment to instructors is randomized. These studies find that SETs are not reliable measures of teaching effectiveness. In our analysis below we use the same approach, although a major difference is that in the present context the assignment of students to courses is not random. We note, however, that as shown in the previous sub-section, there do not seem to have been any systematic changes in course composition with the implementation of the policy. Thus, selection might not be a major concern. For each observation in the student level dataset we construct a measure of the student s exposure to highly evaluated instructors (or courses). The measure is the average overall teaching (or course) evaluations score of instructors from whom the student previously took courses in the same department. We then estimate the following model: _ є (2) where is the grade of student i taking the course-instructor combination j in semester t, _ is the measure of the student s history of exposure to highly evaluated instructors discussed above, is a student fixed effect, is courseinstructor fixed effect, is a semester fixed effect, is a vector of time-varying control variables at the course level (enrollment, mean SAT and share of students taking the course as a requirement), and є is a well-behaved error term. Our interest is in the coefficient which captures the effect of past exposure to highly evaluated instructors on current student grades. Results are reported in Table 8. [Table 8] 16

18 In the baseline specification (column 1) the only right hand side variable is _. We find that past exposure to highly rated instructors is negatively but insignificantly associated with current grades. In column 2 we add student fixed effects. This vastly increases the explanatory power of the regression but has very little effect on the coefficient of interest. Similarly, adding course-instructor fixed effects (column 3), semester fixed effects (column 4), and course controls (column 5) has little effect on. Estimating the full version of Equation (2) separately for the pre- and post-policy change periods (columns 6-7) yields similar results. When we calculate _ using the overall course ratings instead of the overall teacher ratings, results are almost identical (see Table A3 in the Appendix). In sum, consistent with the existing literature, the results presented in this section do not provide evidence in support of the claim that SETs are reliable measures of teaching effectiveness. This suggests that the simultaneous relative increase in teaching evaluations and student grades observed for the initially lowrated instructors are likely not the result of improved teaching effectiveness. 5. CONCLUSION Several institutions of higher education either adopted or consider adopting a policy of posting summary results of SET surveys online so they can be viewed by students. The main rationale for adopting such policies is that this information can aid students in choosing courses. However, very little is known about the policies effects on various outcomes of interest. In this paper we attempt to fill the gap by analyzing a specific case: the 2008 adoption of such a policy by the College of Agriculture and Life Sciences at Cornell University. The results offer a cautionary tale which we hope can inform policy debates about student evaluations of teaching and grade inflation. 17

19 We conjecture that the online posting of SET survey results put pressure on instructors to improve their ratings, and that this pressure was especially strong for instructors who prior to the policy change had relatively low scores. Our differencein-differences analysis indicates that the policy change resulted in a relative increase in the teaching evaluations of instructors whose pre-policy teaching evaluations were low. This was likely at least in part due to increased teaching efforts. Indeed, we find for example that availability to students increased most strongly for the initially low-rated instructors. 16 At the same time we find that the policy change led to a relative rise in grades assigned by the initially low-rated instructors. Our analysis indicates that this increase in grades is likely not due to improved teaching effectiveness. The results are thus consistent with the claim that the policy change contributed to grade inflation. Our analysis also indicates that the main goal of the policy change seems to have not been achieved: there was no effect on course enrollment or composition. It is possible that students already knew who the highly evaluated professors are, so the policy change did not add relevant information and thus did not affect selection. It is also possible that students either (a) do not care much about the course and instructor characteristics that are measured in the SET surveys or (b) care about them but are unable to make course selections based on this information due to constraints in their curriculum or schedule. Finally, it is possible that students become more informed but also (rationally) expect higher grades from initially low- 16 Increased effort devoted to teaching may come at a cost. A recent paper by De Philippis (2013) explores the effects of a policy adopted by Bocconi University with the aim of improving research performance. She shows that the policy change led to deterioration in teaching quality. This raises the concern that strengthening incentives to improve teaching quality might negatively affect research efforts. 18

20 rated instructors and so their desire for higher grades might be moderated by the lack of appeal of lower-quality teaching. 19

21 REFERENCES Achen, Alexandra C., and Paul N. Courant What Are Grades Made Of? Journal of Economic Perspectives, 23(3): Bar, Talia, Vrinda Kadiyali, and Asaf Zussman Grade Information and Grade inflation: The Cornell Experiment. Journal of Economic Perspectives, 23(3): Bar, Talia, Vrinda Kadiyali, and Asaf Zussman Putting Grades in Context. Journal of Labor Economics, 30(2): Beleche, Trinidad, David Fairris, and Mindy Marks Do Course Evaluations Truly Reflect Student Learning? Evidence from an Objectively Graded Post-Test. Economics of Education Review 31(5): Braga, Michela, Marco Paccagnella, and Michele Pellizzari Evaluating Students Evaluations of Professors. Economics of Education Review 41: Butcher, Kristin F., Patrick J. McEwan, and Akila Weerapana The Effects of an Anti-Grade-Inflation Policy at Wellesley College. Journal of Economic Perspectives, 28(3): Carrell, Scott E., and James E. West Does Professor Quality Matter? Evidence from Random Assignment of Students to Professors. Journal of Political Economy, 118(3): De Philippis, Marta Research Incentives and Teaching Performance: Evidence from a Natural Experiment. London School of Economics and frdb. Johnson, Valen E Grade Inflation: A Crisis in College Education. New York: Springer. Love, David A. and Matthew J. Kotchen Grades, Course Evaluations, and Academic Incentives. Eastern Economic Journal, 36(2): Rosovsky, Henry, and Matthew Hartley Evaluation and the Academy: Are we Doing the Right Thing? Cambridge, MA: American Academy of Arts and Sciences. 20

22 TABLE A1 EFFECT OF POLICY CHANGE ON INSTRUCTOR EVALUATIONS (clustering by course-instructor combination) Dependent variable: Mean instructor evaluation (1) (2) (3) (4) (5) (6) Policy *** *** *** *** *** *** (0.265) (0.271) (0.276) (0.276) (0.276) (0.271) Initial_eval*Policy *** *** *** *** *** *** (0.060) (0.060) (0.061) (0.061) (0.061) (0.060) Time trend ** ** ** ** ** (0.005) (0.006) (0.005) (0.005) (0.006) Mean SAT score (0.433) (0.426) (0.427) (0.423) ln(enrollment) (0.049) (0.049) (0.047) Share female (0.106) (0.105) Share required (0.125) Course-instructor fixed effects Yes Yes Yes Yes Yes Yes Observations R Notes: Policy is an indicator taking the value of 1 in all the post-policy change semesters (fall 2008 and later). Initial_eval is the average mean (across students) instructor evaluation for each course-instructor combination across all its pre-policy change observations. SAT scores are available for only a subset of students. Share required is the share of students who reported that their reason for taking the course is that it is required. Estimated by OLS. Robust standard errors, clustered at the course-instructor combination, in parentheses. *, **, *** represent statistical significance at the 10, 5, and 1 percent levels. 21

23 TABLE A2 EFFECT OF POLICY CHANGE ON STUDENT GRADES (using initial course instead of instructor evaluation) Dependent variable: Mean student grades in course (1) (2) (3) (4) (5) (6) Policy *** ** *** *** *** *** (0.133) (0.132) (0.144) (0.142) (0.142) (0.142) Initial_eval*Policy *** *** *** *** *** *** (0.032) (0.032) (0.035) (0.034) (0.034) (0.034) Time trend ** * * * (0.003) (0.003) (0.003) (0.003) (0.003) Mean SAT score ** ** ** ** (0.237) (0.233) (0.231) (0.231) ln(enrollment) ** ** ** (0.023) (0.023) (0.023) Share female (0.080) (0.079) Share required (0.066) Course-instructor fixed effects Yes Yes Yes Yes Yes Yes Observations R Notes: Policy is an indicator taking the value of 1 in all the post-policy change semesters (fall 2008 and later). Initial_eval is the average mean (across students) course evaluation for each course-instructor combination across all its pre-policy change observations. SAT scores are available for only a subset of students. Share required is the share of students who reported that their reason for taking the course is that it is required. Estimated by OLS. Robust standard errors in parentheses. *, **, *** represent statistical significance at the 10, 5, and 1 percent levels. 22

24 TABLE A3 DOES PAST EXPOSURE TO HIGH EVALUATION COURSES INCREASE CURRENT STUDENT GRADES? Dependent variable: Current student grade (1) (2) (3) (4) (5) (6) (7) Course evaluation history (0.062) (0.083) (0.070) (0.070) (0.069) (0.129) (0.077) Student fixed effects No Yes Yes Yes Yes Yes Yes Course-instructor fixed effects No No Yes Yes Yes Yes Yes Semester fixed effects No No No Yes Yes Yes Yes Course controls No No No No Yes Yes Yes Observations 7,252 7,252 7,252 7,252 7,157 3,781 3,376 R Notes: Each column reports results from a separate regression where the dependent variable is the grade of a specific student, taking a specific course-instructor combination in a specific semester. Course evaluation history is the average overall instructor evaluation scores of instructors from whom the student previously took courses in the same department. Course controls include ln(enrollment), mean SAT (SAT scores are available for only a subset of students) and share of students taking the course as a requirement. In column 6 the analysis is restricted to grades assigned in the pre-policy change period (spring 2008 and earlier) while in column 7 the analysis is restricted to grades assigned in the post-policy change period. Estimated by OLS. Robust standard errors in parentheses. *, **, *** represent statistical significance at the 10, 5, and 1 percent levels. 23

25 TABLE 1 SUMMARY STATISTICS Mean Std. Dev. Min. Max. N Mean instructor evaluation Mean course evaluation Mean SAT score (/1000) Enrollment Share female Share required Notes: The unit of observation is course-instructor-semester combination. SAT scores are available for only a subset of students. Share required is the share of students who reported that their reason for taking the course is that it is required. 24

26 TABLE 2 EFFECT OF POLICY CHANGE ON INSTRUCTOR EVALUATIONS Dependent variable: Mean instructor evaluation (1) (2) (3) (4) (5) (6) Policy *** *** *** *** *** *** (0.255) (0.263) (0.276) (0.274) (0.274) (0.270) Initial_eval*Policy *** *** *** *** *** *** (0.057) (0.058) (0.061) (0.060) (0.060) (0.060) Time trend ** *** *** *** *** (0.005) (0.005) (0.005) (0.005) (0.005) Mean SAT score (0.351) (0.346) (0.347) (0.344) ln(enrollment) (0.040) (0.040) (0.039) Share female (0.113) (0.112) Share required (0.109) Course-instructor fixed effects Yes Yes Yes Yes Yes Yes Observations R Notes: Policy is an indicator taking the value of 1 in all the post-policy change semesters (fall 2008 and later). Initial_eval is the average mean (across students) instructor evaluation for each course-instructor combination across all its pre-policy change observations. SAT scores are available for only a subset of students. Share required is the share of students who reported that their reason for taking the course is that it is required. Estimated by OLS. Robust standard errors in parentheses. *, **, *** represent statistical significance at the 10, 5, and 1 percent levels. 25

27 TABLE 3 EFFECT OF POLICY CHANGE ON COURSE EVALUATIONS Dependent variable: Mean course evaluation (1) (2) (3) (4) (5) (6) Policy *** *** *** *** *** *** (0.269) (0.276) (0.270) (0.263) (0.264) (0.260) Initial_eval*Policy *** *** *** *** *** *** (0.064) (0.064) (0.064) (0.062) (0.062) (0.061) Time trend ** ** ** ** ** (0.005) (0.006) (0.005) (0.005) (0.006) Mean SAT score (0.417) (0.406) (0.406) (0.399) ln(enrollment) ** ** ** (0.042) (0.042) (0.041) Share female (0.123) (0.120) Share required * (0.108) Course-instructor fixed effects Yes Yes Yes Yes Yes Yes Observations R Notes: Policy is an indicator taking the value of 1 in all the post-policy change semesters (fall 2008 and later). Initial_eval is the average mean (across students) course evaluation for each course-instructor combination across all its pre-policy change observations. SAT scores are available for only a subset of students. Share required is the share of students who reported that their reason for taking the course is that it is required. Estimated by OLS. Robust standard errors in parentheses. *, **, *** represent statistical significance at the 10, 5, and 1 percent levels. 26

28 TABLE 4 EFFECT OF POLICY CHANGE ON VARIOUS EVALUATION QUESTIONS (1) (2) (3) (4) (5) (6) Initial Initial instructor course evaluation evaluation *policy R 2 Policy *policy R 2 Policy Instructor is enthusiastic about teaching ** ** * ** (0.287) (0.064) (0.256) (0.060) I learned to value new viewpoints *** *** *** *** (0.263) (0.059) (0.230) (0.055) Instructor stimulated interest in the subject matter *** *** *** *** (0.340) (0.075) (0.283) (0.067) I learned more in this course than I expected to learn *** *** *** *** (0.295) (0.067) (0.263) (0.064) Instructor was well-prepared in this class *** *** *** *** (0.281) (0.063) (0.268) (0.063) Instructor seemed to like students (0.297) (0.066) (0.266) (0.063) I learned a great deal in this course *** *** *** *** (0.316) (0.071) (0.278) (0.066) Instructor puts material across in an interesting way *** *** *** *** (0.310) (0.069) (0.280) (0.066) After taking this course I feel more comfortable tackling complex issues *** *** *** *** (0.278) (0.064) (0.262) (0.064) Instructor was responsive in taking questions (0.291) (0.064) (0.247) (0.058) I would recommend this course to a fellow student *** *** *** *** (0.345) (0.076) (0.310) (0.073) Instructor was readily available for consultations with students *** *** *** *** (0.319) (0.070) (0.246) (0.058) 27

29 TABLE 4 - CONTINUED EFFECT OF POLICY CHANGE ON VARIOUS EVALUATION QUESTIONS (1) (2) (3) (4) (5) (6) Initial Initial instructor course evaluation evaluation *policy R 2 Policy *policy R 2 Policy Instructor treated students fairly (0.262) (0.057) (0.239) (0.055) Course helped me develop ability to think through a problem/argument *** *** *** *** (0.276) (0.063) (0.265) (0.064) Instructor stimulated class discussion *** *** *** *** (0.353) (0.079) (0.325) (0.077) Course challenged me intellectually *** *** *** *** (0.300) (0.070) (0.291) (0.072) I would take another course from this instructor *** *** *** *** (0.386) (0.086) (0.341) (0.081) In comparison to all the other instructors I've had, he/she was one of the best *** *** *** *** (0.412) (0.092) (0.358) (0.085) Notes: Each row presents output from a pair of separate regressions where the dependent variable is the mean response to the evaluation question specified in the row heading. The explanatory variables in the regressions reported in columns 1-3 are identical to those of column 6 in Table 2; the explanatory variables in the regressions reported in columns 4-6 are identical to those of column 6 in Table 3. In specific, Policy is an indicator taking the value of 1 in all the post-policy change semesters (fall 2008 and later), Initial instructor evaluation is the average mean (across students) instructor evaluation for each course-instructor combination across all its pre-policy change observations, and Initial course evaluation is the average mean (across students) course evaluation for each course-instructor combination across all its pre-policy change observations. The number of observations in the regressions is either 662 or 663. Estimated by OLS. Robust standard errors in parentheses. *, **, *** represent statistical significance at the 10, 5, and 1 percent levels. 28

30 TABLE 5 EFFECT OF POLICY CHANGE ON STUDENT GRADES Dependent variable: Mean student grades in course (1) (2) (3) (4) (5) (6) Policy ** * *** *** *** *** (0.150) (0.149) (0.151) (0.151) (0.150) (0.150) Initial_eval*Policy ** ** *** *** *** *** (0.034) (0.034) (0.035) (0.035) (0.035) (0.035) Time trend ** * * * (0.003) (0.003) (0.003) (0.003) (0.003) Mean SAT score ** ** ** ** (0.239) (0.235) (0.234) (0.233) ln(enrollment) ** ** ** (0.023) (0.023) (0.023) Share female (0.081) (0.080) Share required (0.067) Course-instructor fixed effects Yes Yes Yes Yes Yes Yes Observations R Notes: Policy is an indicator taking the value of 1 in all the post-policy change semesters (fall 2008 and later). Initial_eval is the average mean (across students) instructor evaluation for each course-instructor combination across all its pre-policy change observations. SAT scores are available for only a subset of students. Share required is the share of students who reported that their reason for taking the course is that it is required. Estimated by OLS. Robust standard errors in parentheses. *, **, *** represent statistical significance at the 10, 5, and 1 percent levels. 29

31 TABLE 6 EFFECT OF POLICY CHANGE ON THE SHARES OF VERY HIGH AND VERY LOW GRADES (1) (2) Dependent variable: Share of A+ grades Share of C+ or below grades Policy ** *** (0.085) (0.066) Initial_eval*Policy ** *** (0.020) (0.015) Time trend (0.002) (0.001) Mean SAT score ** (0.135) (0.103) ln(enrollment) ** (0.011) (0.008) Share female (0.046) (0.031) Share required (0.031) (0.026) Course-instructor fixed effects Yes Yes Observations R Notes: Policy is an indicator taking the value of 1 in all the post-policy change semesters (fall 2008 and later). Initial_eval is the average mean (across students) instructor evaluation for each course-instructor combination across all its prepolicy change observations. SAT scores are available for only a subset of students. Share required is the share of students who reported that their reason for taking the course is that it is required. Estimated by OLS. Robust standard errors in parentheses. *, **, *** represent statistical significance at the 10, 5, and 1 percent levels. 30

32 TABLE 7 EFFECT OF POLICY CHANGE ON COURSE COMPOSITION (1) (2) (3) (4) (5) (6) (7) (8) Dependent variable ln enroll. Mean SAT Share female Share required ln enroll. Mean SAT Share female Share required Policy (0.333) (0.033) (0.110) (0.120) (0.312) (0.034) (0.106) (0.118) Initial_eval*Policy (0.076) (0.008) (0.025) (0.027) (0.074) (0.008) (0.026) (0.028) Time trend *** *** (0.007) (0.001) (0.002) (0.003) (0.007) (0.001) (0.002) (0.003) Course-instructor fixed effects Yes Yes Yes Yes Yes Yes Yes Yes Observations R Notes: Each column presents output from a separate regression where the dependent variable is specified in the column heading. Policy is an indicator taking the value of 1 in all the post-policy change semesters (fall 2008 and later). In column 1-4 Initial_eval is the average mean (across students) instructor evaluation for each course-instructor combination across all its pre-policy change observations. In column 5-8 Initial_eval is the average mean (across students) course evaluation for each course-instructor combination across all its pre-policy change observations. Estimated by OLS. Robust standard errors in parentheses. *, **, *** represent statistical significance at the 10, 5, and 1 percent levels. 31