CFA member survey about the student evaluation of teaching policy language in the new CBA. Fall 2012

Transcription

1 CFA member survey about the student evaluation of teaching policy language in the new CBA. Fall 2012 The Pomona chapter of the California Faculty Association surveyed its members concerning the provision in the newly ratified Collective Bargaining Agreement (CBA) that could change campus policy about the number of student evaluations required for each faculty member. Previously, such policy was left up to campuses, and at Cal Poly Pomona, a policy requiring a minimum of two evaluations per year had been established through shared governance processes. This policy is part of Appendix 10 ( Appendix 10 asserts that the department should be given the maximum possible latitude in collecting, assessing and reporting available information on teaching performance consistent with this policy (0.c). The new CBA would allow the President to uphold Appendix 10 and retain it as local policy. However, the default in the new CBA is a system-wide policy requiring all courses be evaluated: Written or electronic student questionnaire evaluations shall be required for all faculty unit employees who teach. All classes taught by each faculty unit employee shall have such student evaluations unless the President has approved a requirement to evaluate fewer classes after consideration of the recommendations of appropriate faculty committee(s). In cases where student evaluations are not required for all classes, classes chosen for evaluation shall be representative of the faculty unit employee's teaching assignment, and shall be jointly determined in consultation between the faculty unit employee being evaluated and his/her department chair.... The chapter leadership recognizes that under Appendix 10, departments have promulgated a variety of student evaluation policies, and that already many faculty are subjected to departmental evaluation policies requiring that all courses be evaluated. The diversity in student evaluation policies reflects the culture of the University. Evaluation policies, including RTP criteria, reside in individual departments, and departmental control over these matters is a shared value of the university. The potential change in student evaluation policy has been one of the more controversial elements in the new CBA. Thus, the CFA chapter decided to survey its members about the issues surrounding the possible change in campus policy. This report summarizes the findings from the survey. It is divided into four sections covering: 1) the survey itself and the claim that it represents the views of the membership; 2) basic quantitative findings; 3) cross-tabulations of views on student evaluations with views on where the locus of control over the number of evaluations should lie; and 4) qualitative findings summarizing the content of over 150 open-ended comments. 1. The survey and evidence that respondents are a representative sample An inviting CFA members to participate in the survey was sent from surveymonkey to 625 CFA members at 8 am on 7 November, with a complete by deadline of 16 November. A reminder was sent to non-responders at 8 am on 15 November. 299 responded; of these, 284 answered the CFA student evaluations survey 1 7 January 2013

2 substantive questions for a response rate of 45 percent. The respondent profile is very similar to the CFA membership profile. Table 1 compares rank and status for the membership and the survey respondents. Table 1 rank and status: membership v survey Assistant Professor Associate Professor Full Professor Lecturer Total CFA members (as of November 2012) % 13% 12% 43% survey % 15% 15% 41% The respondent file s distribution among colleges also compares to the membership, as shown in Table 2. Table 2 Colleges: membership v survey) ag cba collins ceis eng env class sci Did not state, or multiple colleges & all university Total CFA members (as of October 2012) % 1 2% 8% 14% 8% 26% 21% 5% survey % 1 3% 8% 14% 9% 29% 18% 4% 2. CFA member opinions about the number of classes that should be subject to student evaluation of teaching at Cal Poly Pomona In addition to college and status/rank, the survey asked additional questions as background to the main questions concerning faculty views about Cal Poly Pomona s policy on student evaluation of teaching. Cal Poly Pomona s policy has established a minimum number of classes two-- per faculty member that should be subject to student evaluation of teaching. Many departments have evaluation criteria requiring more. Faculty were asked about these policies as well as whether or not the possible imposition of a policy requiring all classes would represent change. A slim majority of respondents already are subjected to a policy of all classes, but there is great variety by college, as well as one in ten who are unsure about the current policy affecting them. Figure 1 illustrates this variety. CFA student evaluations survey 2 7 January 2013

3 Figure 1. Would the requirement that all courses have student evaluations represent a change for you, personally? AG CBA COL CEIS ENG ENV CLASS SCI OTHER* ALL I am not sure no yes *Did not state, or multiple colleges & all university Figure 2a compares responses to two key questions from the survey. Each asked for level of agreement with these statements: 1. The President should have discretion over the number of student evaluations that should be administered for each faculty member. 2. Individual departments should have discretion over the number of student evaluations that should be administered for each faculty member Figure 2a. Who should have discretion over the number of student evaluations that should be administered? Departments The President strongly agree strongly agree 82 percent of respondents agreed or strongly agreed that departments should have this discretion, compared to 25 percent who said that the President should. Figure 2b shows the breakdown by college of support for departments having discretion over the number of evaluations. CFA student evaluations survey 3 7 January 2013

4 Figure 2b. Individual departments should have discretion over the number of student evaluations that should be administered for each faculty 3 strongly agree strongly agree 2 1 AG CBA COL CEIS ENG ENV CLASS SCI OTHER* ALL *Did not state, or multiple colleges & all university Some respondents agreed or d with both statements. Whereas agreeing that Departments should have discretion and ing that the President should have discretion can be interpreted as a preference for Departments, instances in which a responded agreed or d with both statements suggests a lack of preference. Data was recoded, sorting the responses into three categories: preference for Departments (64%), preference for the President (7%), and no clear preference (29%). The recoded variable was used in cross-tabulations of preferences with questions about student evaluations. For example, the question reported in Figure 1 is reported again in Figure 3a. Here, those who responded yes a policy of all courses would mean a change are compared with those who said no, and the responses are cross-tabulated by the recoded preference variable. As Figure 3a shows, those for whom the Figure 3a. Who should decide about the number of student evaluations? no yes all President should control number of evals Preference unclear* Departments should control number of evals Would the requirement that all courses have student evaluations represent a change for you, personally? * respondent 's answers were similar on 2 questions about control, e.g. the responded "agreed" or "strongly agreed" that departments should have control and "agreed" or "strongly agreed" that the presendent should have control. policy would be a change were more likely to articulate a preference for departments having control over the number of evaluations. In bargaining, the administration promoted the new student evaluation policy as something that would be in the interest of lecturers. Lecturers who responded to CFA student evaluations survey 4 7 January 2013

5 Figure 3b. Who should decide about the number of student evaluations? Part time Full time Assistant Associate Full lecturer lecturer Professor Professor Professor * respondent 's answers were similar on 2 questions about control, e.g. the responded "agreed" or "strongly agreed" that departments should have control and "agreed" or "strongly agreed" that the president should have control. Pres Preference unclear* Dept the survey did not, however, support putting control over the number of evaluations into the hands of the President. Figure 3b shows that a majority of lecturers polled support keeping the locus of control in the hands of Departments. 3. CFA member opinions of student evaluations of teaching cross-tabulated with opinion about locus of control Respondents answered four questions about student evaluations. Figure 4 shows the results from these questions. Respondents were split on the utility of more data for assessing teaching performance. Respondents were even more split with 29% choosing neither agree nor on the question of whether student evaluation data is inherently suspect. The greatest level of support was for the third statement, an affirmation that student evaluations have utility when they, faculty, can apply them to their own development as instructors. Supporting this affirmation, respondents displayed the greatest level of rejection for the last statement which said that student evaluations are not very useful to me. Figure 4. Four questions about student evaluations Evaluating all courses provides valuable data for assessing teaching performance. As data, student evaluations of teaching are inherently suspect. The results of student evaluations of my own classes provide me with feedback that is useful to me as I prepare for future teaching. I may glance at the results of student evaluations of my own classes, but they are not very useful to me as I prepare for future teaching. 2 strongly agree agree neither agree nor strongly CFA student evaluations survey 5 7 January 2013

6 Results of three of the above questions are cross-tabulated with the recoded locus of control variable and illustrated in Figures 5a, Figure 5a. Who should decide about the number of student evaluations? 5b and 5c agree & strongly agree neither agree nor & strongly Pres Preference unclear* Dept Evaluating all courses provides valuable data for assessing teaching * respondent 's answers were similar on 2 questions about control, e.g. the responded "agreed" or "strongly agreed" that departments should have control and "agreed" or "strongly agreed" that the president should have control. As support for the statement Evaluating all courses provides valuable data for assessing teaching performance decreases, preference for the Department as the locus of control increases. Figure 5a illustrates this relationship. Figure 5b illustrates responses to an opposing statement. The results shown in Figure 5b are consistent with those shown in Figure 5a. As support for the statement As data, student evaluations of teaching are inherently suspect decreases, preference for the Department as the locus of control also decreases. Figure 5b. Who should decide about the number of student evaluations? agree & strongly agree neither agree nor & strongly As data, student evaluations of teaching are inherently suspect. Pres Preference unclear* Dept * respondent 's answers were similar on 2 questions about control, e.g. the responded "agreed" or "strongly agreed" that departments should have control and "agreed" or "strongly agreed" that the president should have control. Figures 5a and 5b contrast results for two questions representing acceptance and rejection of student evaluations of teaching as valid tools for evaluating faculty. Figure 5c illustrates results for a statement lying somewhere between acceptance and rejection: The results of student evaluations of my own CFA student evaluations survey 6 7 January 2013

7 classes provide me with feedback that is useful to me as I prepare for future teaching. Not surprisingly, the cross-tabulated results also lie in between the results for the more extreme statements Figure 5c. Who should decide about the number of student evaluations? agree & strongly agree neither agree nor & strongly Pres Preference unclear* Dept The results of student evaluations of my own classes provide me with feedback that is useful * respondent 's answers were similar on 2 questions about control, e.g. the responded "agreed" or "strongly agreed" that departments should have control and "agreed" or "strongly agreed" that the president should have control. 4. CFA member opinions about a variety of issues relating to the proposed change in student evaluation policy at Cal Poly Pomona. The survey included space for an open-ended comment at the end. 157 respondents made open-ended comments. Comments were categorized into 10 categories, with up to three categories per comment. The categories and the proportion of comments in each are shown in Table 3. Table 3. Comments categorized subtable Number of Proportion of all 4 category comments comments a policy 97 62% b interpretation 48 31% c questionnaires 25 16% d cost 16 1 e fairness 9 6% f online 8 5% g quality control 5 3% g exemptions 4 3% g innovation stifling 4 3% g CFA 3 2% Each comment category is briefly described and illustrated with sample comments. policy CFA student evaluations survey 7 7 January 2013

8 Almost two thirds of the comments elaborated on the survey theme of Cal Poly Pomona s policy regarding the number of student evaluations required of faculty. Comments ranged from support for /rejection of the all courses policy to elaborations of other possible policies. Table 4a provides a sampling of policy comments. Table 4a. Policy comments Student evaluations for every course is an excellent procedure; it gives input from the student to contribute an irreplaceable perspective on the teaching, and over time with sufficient evaluations insight as to what works and doesn't work so well in a course. I do not think we should have to evaluate supervisory courses (senior project, special projects, thesis etc). I like the idea of departments setting their own numbers, with a minimum like it used to be of at least 2/yr. Recommendation: Classes with mean value of the first exam being and more definitely must have student evaluation I think requriing [sic] every class to be evaluated is burdensome and will lead to problems. What if an instructor forgets to evaluate one class out of the nine they teach each year? What are the repercussions of that? What if the class is online and no students choose to complete an evaluation? It depends on how the evaluation measures are used and if instructors are provided expectations or it it is benchmarked. Teaching is a multiple attribute service and using only students' evaluation is unlikely to provide a good measure of it. Sure, there are peer-evaluations that are used in tenure/retention decisions but these are also problematic because inter-personal relationship in the department/unit might interfere with an unbiased judgment. The jury is out as to what is the "best" way to measure teaching effectiveness, but I do not believe there has been a sincere effort by any university to address the issue. Incidentally, my teaching evaluations, by both students and peers, are excellent, and I suppose that makes me a great teacher. interpretation Nearly one third of the comments highlighted problems relating to the interpretation of the results of student evaluations. Comments ranged from support for /rejection of the all courses policy to elaborations of other possible policies. Table 4a provides a sampling of policy comments. Table 4b. interpretation comments There is no objectively reliable way to interpret student evaluations, which are very often subject to manipulations. Faculty members, especially the younger ones, very often find themselves pressured to improve their student evaluations rather than to improve their teaching. Even though on a few occasions persistent negative trends and patterns in student evaluations do suggest problems on the part of an instructor, more often than not, they are swamped in statistical noise and do not provide meaningful information. Teaching evaluations should be used to study for outlier responses as compared to a representative group. A negative outlier does not necessarily indicate "poor" teaching, but does indicate a need for further investigation and careful consideration by the instructor. The outliers should be determined for each question, rather than averaged over all questions. I see this as a tool to indicate a need to intercede with the instructor to help improve instruction and student relations, rather than just be used to punish. When you teach difficult subject matter and grade appropriately, your evaluations are not going to be as CFA student evaluations survey 8 7 January 2013

9 good as easy professors who hand out grades. This is a serious infringement upon the academic freedom of professors and will dilute the quality of education as professors will teach to get the good evals. Multiple choice evaluations don't give valuable feedback to teachers. I have had more useful comments from my own (anonymous) evaluations using survey monkey in which I can include specific examples, classes, etc. and ask for student opinions evaluations of those. questionnaires A number of respondents used the comment section to address problems with the questionnaire used in the student evaluation of teaching. Departments control what goes on these questionnaires, and these comments were in most cases very specific. Table 4c. questionnaire comments The problem with the student evaluations is not the number but the quality of the instruments being used. In the past students could indicate comments, which were helpful. Now that option is no longer available. It seems the instrument is less useful than it was and the emphasis is on using a less useful instrument more frequently. This doesn't make any sense to me. I think they evaluations used in the [XXX] department should gather more detailed information and have an area for narrative comments by the students. The evaluations should elicit information that differentiates between feedback about student's "like" of the professor because of the difficult of their assignments, exams, etc... and feedback that determines the quality of the content of the class. Depending on how the questionnaires are designed, the student evaluations of teaching can be helpful for program assessment and evaluation of teaching. Departments should maintain the last say in designing the questionnaires to make them most meaningful. In my experience the actual evaluation questions that we use are not very useful. I'd like to see more help at the University level in designing forms that give less ambiguous feedback about specific instructor behaviors (in course design, in the classroom, in evaluating students, etc.) that could more easily translate into improvements. cost One in ten comments addressed the costs in dollars, staff time and faculty time of increasing the number of evaluations. No comments addressed costs to promote the allocation of additional resources for student evaluations. Table 4d. costs What is the rationale for this new policy? Also, what is the administrative cost of implementing it? CFA student evaluations survey 9 7 January 2013

10 These are also considerations in forming any opinions. In general, I think more data is better. However, I also think that this increases costs dramatically for something that is less important in a time when funds are not plentiful. I think it is a bad change to implement. I am not opposed to having student evaluations in all classes, but do not see the wisdom of it, especially given at a time of depleting resources, bigger classes, overwork for faculty, we are also burdening the entire system (including those who will review these evaluations) with more that must be done with less and less. What was wrong with two classess [sic] every year as per faculty contract? fairness A number of comments addressed fairness, with a recurring theme of unfairness about current policies in some departments that have different standards for different categories of faculty (lecturers, probationary and tenured). No comments addressed fairness to promote differential standards. Table 4e. fairness My department RTP document has a specific requirement for non-tenured faculty to have all their courses being evaluated by students. I agree with that, however, I think to be fair to everyone, other faculty members in the department regardless of their ranks or tenure status should be also requested to do so, therefore the comparison between the faculty's performance would be valid. I already evaluate every class that I teach and I don't know why anyone would argue with that policy. It sure is tempting to skip the classes that go poorly, but requiring it in all classes avoids the real possibility of faculty hand-picking their best classes for evaluation. Doing all classes is the fairest and most honest policy. I think that we ought to at least PRETEND to respect students and allow them to evaluate professors EVERY SINGLY [sic] CLASS. There certainly should not be a two-tier system where adjuncts have to do student evals for every class, but not Tenured/tenure track professors. Furthermore, we ought to make the evaluations count for something and do something about professors who are CONSISTENTLY given low marks by students. Students do not dislike professors who make them work hard. Students dislike professors who fail to do their jobs, mistreat students and in all ways fail in their primary mission (which SHOULD be teaching).... In my department, part-time lecturers, even those with 3-year contracts and teaching 9 courses per year have ALWAYS been required to conduct student evaluations in ALL CLASSES. Tenured and tenure-track faculty were ONLY required to conduct 4 evaluations over a 2-year period. I personally don't care how many classes need to be evaluated as long as it is the SAME for ALL FACULTY. The evaluations in my department are 8 very general questions, poorly worded and useless to me in any professional way. The requirement for lecturers to have all classes evaluated, I suspect, leads to grade inflation by some faculty members. online CFA student evaluations survey 10 7 January 2013

11 A number of comments promoted the idea of putting student evaluations online. Most were offered without rationale, but a few offered specific reasons for promoting online evaluations ranging from the integrity of the evaluation to environmental savings and efficiency. One comment cautioned against online evaluations. Table 4f. online It would be very helpful if the students evaluations were conducted online, after the completion of the course. That would reduce the amount of paperwork and classroom time spent in the process and would streamline the evaluation process. It is remarkable that this isn't online. It is un-environmental and slow to do it with the paper process we currently have. Many other CSUs have a much more online-modern approach to this. I is difficult to find someone else to give your evaluations, having evals online would greatly improve this. Faculty would have the result immediately and might be more willing to look at the result because it is close to the time that the course took place..... I will also point out that when I teach classes online it is almost impossible to collect enough data through the voluntary online student evaluation to be meaningful because few students bother to complete it... I'd like to have online evaluations be available for more or all classes. I had a course with a very smart student with a very strong personality and charismatic leadership skills who d with some of my requirements. He thought that he could skip an assignment and not have it counted against him because he knew the material so well. I didn't see it that way. I can't prove it, but the pattern of the evaluations suggested that he was lobbying against me after I handed out the evaluations and left the room. Online evaluations might prevent something like that. They would also be helpful in a lab class that I teach where the largely offline instructional format makes it difficult to conduct evaluations in class. quality control, exemptions, innovation stifling, CFA One in ten comments brought up one or more of the following issues: quality control of the evaluation process, exemptions needed for certain courses (most example were supervisory courses such as senior projects and internships), the dampening effect on pedagogical innovation of reliance on student evaluations, and criticism of CFA. Table 4g. quality control, exemptions, innovation stifling, CFA quality control exemptions innovation stifling The evaluation process is not secure; it leaves ample room for tampering. Student evaluations are not logistically feasible in some classes (capstone courses, for example). Student evals are like Yelp reviews. Some, but not all, are trustworthy. Unless they are taken into consideration ONLY in the aggregate, doing 9 per year is not any better than 2 per year. I do evaluations in all my classes, but would like the option of NOT doing so. New preps, piloting new material or trying out new teaching techniques present possibilities of less positive reviews. If we are CFA student evaluations survey 11 7 January 2013

12 CFA going to be penalized for trying new things, I am going to stick with old, tried and proven. Goodbye, innovation. The issue concerning the questionable reliability of student evaluation has been widely discussed and debated. I remain unclear about why the CSU adds this requirement to the new CBA and why the CFA accepts this new requirement at the end of its contract negotiation.... CFA student evaluations survey 12 7 January 2013