Student Interpretations of the Scale Used in the BYU Online Student Ratings Instrument Paul Fields, PhD Department of Statistics Trav Johnson, PhD Center for Teaching and Learning
Student Interpretations of the Scale Used in the BYU Online Student Ratings Instrument Summary There are potentially over 30,000 students in hundreds of classes filling out the Online Student Ratings at BYU each semester. Their responses are collected on an 8-point scale, yet the instrument does not provide a definition of the scale items or an anchor for students to use in making their ratings. To fully interpret their responses it is necessary to understand the way the students define the points on the scale. To this end, BYU students were surveyed over two academic years concerning their interpretation of the scale. A representative sample of BYU students was surveyed. The proportions of students in the sample matched the proportions of students enrolled at BYU by college, department, class standing, gender and nationality. In total, 101 students participated in the survey. The surveys were conducted in both fall and winter semesters as well as spring and summer terms. The survey was conducted as a think aloud interview. Students were given verbal and written instructions, a random selection of questions from BYU s student rating form, a tape recorder and a quiet room alone. Each student responded with his or her personal interpretation of the scale by answering the basic question, What characteristics would lead you to describe the course (or professor) as exceptional the highest rating possible on the scale? And similarly for other questions on the student ratings form, How would you decide to mark very strongly agree again the highest rating possible concerning? While students gave varied interpretations, two prominent messages emerged: Students consider a course to be exceptional if they perceive that the course is relevant to their lives now and in the future. Students rate a professor as exceptional if they perceive the professor cares about the students learning. According to the students responses, students measure a course s relevance by its applicability to their major discipline, future job and life. In addition, students measure a professor s interest in their learning by the professor s accessibility, availability, approachability and willingness to help students outside of class. Five additional questions not on the student ratings instrument were included in the survey. In response to these questions students indicated they are more likely to complete the student ratings if offered extra credit, they tend to fill out the student ratings for all courses if any, and they provide written comments if they have a strong opinion about the course or professor. 2
Family, Home, and Social Sciences Fine Arts and Communications School of Management Engineering and Technology Student Life Humanities Biology and Agriculture Physical and Mathematical Sciences Health and Human Performance Education Nursing International and Area Studies Law School Graduate Studies Continuing Education Religious Education Sample Design A stratified sample design with proportionate sampling was used to randomly select 101 BYU students to participate in the survey. The proportions of students in the sample by college, department, class standing, gender and nationality matched the proportions of students enrolled at BYU as closely as possible. For example, Figure 1 shows how closely the composition of the sample was to the composition of BYU students by college. Notice the proportions are almost identical. Figure 1: The proportion of students enrolled at BYU by college (purple) and the proportion of students from each college who participated in the survey (maroon). 18% 16% 14% 12% 10% 8% 6% 4% 2% 0% 3
Survey Results Two over-arching themes emerged from the survey. Students indicated that the relevance of the course and a professor who cared about their learning made a learning experience exceptional. Students said they valued courses they perceived as relevant to their lives now and in the future. Additionally, they perceived a professor as caring about them and their learning if he or she was accessible and available to help them. This was noted even more often by the female students than the male students. There are five questions on BYU s Online Student Ratings Instrument that are often emphasized in reviewing the student ratings for a course: 1. I learned a great deal in this course. 2. Course materials and activities are effective in helping students learn. 3. This course helped me develop intellectual skills (such as critical thinking, analytical reasoning, integration of knowledge). 4. The instructor showed genuine interest in students and their learning. 5. The instructor provided opportunities for students to become actively involved in the learning process. Focusing on these Five Learning Items, four themes emerged. They are shown in Table 1 listed in rank order with the most frequently mentioned at the top of the list. Also shown are sub-themes for each theme also in order of frequency. Notice that every theme and sub-theme is actionable something a professor could do to improve his or her student ratings. Table 1: Four themes and their sub-themes collected from the Five Learning Items on the Online Student Ratings instrument. The themes and sub-themes are listed in order of frequency from highest to lowest. Theme Be Available Use Active-Learning Make it Real Keep it Fresh Sub-Theme Professor was available to help or to talk Professor was concerned about students' learning Professor took the time during office hours to help students Used in-class and out-of-class projects, learning by doing Held open class discussions, professor asked for comments Learned by participating, professor involved students Used hands-on group and individual projects Course helped students understand application Can apply it in my life, career; it's relevant, valuable Interesting, applicable course materials, text Can incorporate learning into career, the real world Course helped develop new way of thinking The material was new and innovative Learned from everything - book and other materials As an example, suppose a professor wanted to do one thing next semester to improve his or her student ratings to have more strongly agree responses to the Five Learning Items. The 4
action that should have the greatest impact is to Be Available to students, since that is the theme most frequently mentioned. Adding active-learning, making it real and keeping it fresh should produce a winning combination. Another theme revealed in the survey dealt with feedback on students work. Repeatedly the students in the survey mentioned that feedback needed to be prompt and prescriptive with an opportunity to improve their work. Although the importance of prompt feedback is well-known, it is noteworthy that students value feedback that provides direction in how to improve their performance coupled with the opportunity to do so; otherwise the feedback has limited value. A complete report of the survey results is attached. The students responses to each question on the student ratings instrument are shown and the top responses are ranked by frequency. Illustrative quotes are listed for each question to clarify the students meaning in their responses. Items that can be most directly affected by the professor and may provide the greatest opportunities to affect the students responses to the Online Student Ratings are indicated with checkmarks. 5