An Examination of Assessment Methods Used in Online MBA Courses

An Examination of Assessment Methods Used in Online MBA Courses Shijuan Liu Richard Magjuka Xiaojing Liu SuengHee Lee Kelly Direct Programs, Indiana University Curtis Bonk Department of Instructional Systems Technology Indiana University The importance of assessment methods for teaching and learning is well documented in the literature of education (e.g., Brown, Bull, & Pendlebury, 1997). Instructors can indicate what they think is important for students to learn through the assessment methods they use. From the assessment, students can tell the expectations of their instructors as well as to what extent their learning outcomes meet the instructors expectations. Assessment is also found to be a key factor in motivating or demotivating students to learn (e.g., Harlen & Crick, 2003). Additionally, assessment methods are acknowledged to be crucial for ensuring the quality of individual courses as well as programs that offer the courses (Ford, 2002). While there is rich literature addressing assessment in general education, discussions concerning assessment in online environments, especially in the context of MBA online education, are still very limited. This study examined the assessment methods instructors used in online MBA courses offered by a top ranking business school in the United States. Research questions included: (a) What assessment methods did the instructors use? (b) Why did they choose these methods? and (c) How did students respond to these methods in the course evaluation? Findings of the study will have implications for online instructors and instructional designers, as well as administrators. Methodology An accredited online MBA program offered by a top ranking business school in a large Mid-western university was selected for this study. Initiated in 1999, the MBA program was designed for professionals who wish to continue their employment while earning their MBAs. Instructors were fulltime professors who also taught at the counterpart residential programs of this school. Forty-eight credit hours in total were required by this program. Among them, 14 courses (39 credit hours) were mandatory. Two of the mandatory courses were offered in the first week of each academic year in a face-to-face format. This study examined assessment methods used in the 12 mandatory courses that were offered entirely online. Syllabi of the 12 courses offered during the academic year 2005-2006 were first examined. Among the 12 courses, seven of them were taught by two instructors. Four of the seven courses used exactly the same syllabi and assessment methods. In the other three courses, instructors used different syllabi and assessment methods. Because the instructors used different assessment methods, the three courses were regarded as different sections. Therefore, 15 sections of the 12 courses in total were analyzed. In order to better understand the assessment methods instructors used, we also observed the archived courses. Anonymous student course evaluation data associated with the assessment methods used in the 15 sections were analyzed. Additionally, we approached instructors teaching the courses, and interviewed 12 of them who agreed to participate in the study. Interviews were conducted face-to-face and audio taped, then transcribed. 1

Results and Discussion Nine assessment methods were identified from the 15 examined sections (see the table below). Asynchronous discussion forums were used most frequently (in 11 sections), followed by quizzes and exams (in 7 sections). Quizzes and exams in this study referred to assessment tasks including at least one item in the format of multiple-choice questions. Case analysis and problem analysis methods were also used frequently in assessing students. There seems to be no hard distinction between case analysis and problem analysis. Problem analysis in this study referred to those associated with calculation. For instance, in a course on quantitative analysis, students were asked to analyze some real-world problems such as forecasting sales systems and air ticket reservations using Excel. The problem analysis method was also found in courses concerning economics and accounting. Projects were defined as assessment tasks that allowed students to choose topics or companies that interested to them. For instance, in one management course, the instructor asked students to choose a company interested to them and applied what they learned from the course (e.g., its core ideology, strengths, and weakness). Three sections used projects to assess students. Simulations were used as assessment tasks in two courses both related to management. Essay assignments were not used often. In one course on information technology, for their final exam, students were asked to write an essay to synthesize and apply some of the course learning. Reflection was used in one management related course. In that course, students were asked to write three reflection papers. Two of them were team reflections and the other one was individual reflection on their experience in the simulation. Teamwork was used in the majority of the examined sections. While several sections asked students to evaluate their peers performance in teamwork, only one section listed peer evaluation as a separate assessment task and gave students credit for completing it. Because of space limits, this paper only discussed in detail the use of asynchronous discussion forums and quizzes and exams. Assessment methods (task) Frequency (N=15) Asynchronous discussion forums 11 Quizzes and exams 7 Case analysis 6 Problem analysis 6 Projects 3 Simulations 2 Essays 1 Reflections 1 Peer evaluation on team contribution 1 Asynchronous Discussion Forums Online discussion and participation were required in the majority (73%) of the examined courses. The percentage that this assessment task counted in the final grades varied, ranging from 5% to 40%. Four courses (sections) did not require students to participate in discussions, but this did not mean that there was no discussion. For instance, in one course, the instructors created discussion forums for students to ask questions and post their comments or ideas. Both instructors teaching this course were interviewed. According to them, the major purpose for creating the discussion forums was to ensure fairness in that the answers they provided could be seen by the whole class instead of only one or a few students. In evaluation of the course, some students indicated that they hoped they would be pushed to participate in the discussions. As one of them wrote, The course was great, but I think the students need to be forced to do a bit more discussion. We all have lives outside school, and if we are not forced into it, we aren t going to do it. Some simple weekly discussions would have really helped my understanding, but since it wasn t requirement I was always found a personal excuse of why not to do it. Similarly, in evaluation of 2

two other courses that did not require students to participate in discussions, some students echoed the need for including discussions in the assessment tasks. Online discussions were organized in a variety of ways in the courses examined. In one course, the instructor required students to participate in discussions about four topics. The whole class discussed on the questions that the instructor provided. There was no strict rule for students to participate in discussion except the general guidelines and grading criteria, which emphasized meaningful and substantial contribution. When being asked why he did not limit the number of student postings or assigning roles to students in discussions as some literature suggested, the instructor explained it was because he believed that the essence for discussions was that students were able to speak what and when they wanted to say or share. In another course, its instructor put students in teams. Each team was assigned a role from three roles: defendant, plaintiff, and appellate court judge. After discussion, each team was responsible for submitting a report on their argument. Teams rotated the roles and were evaluated based on the role they played. Students commented highly of the court forum discussion in course evaluation. They mentioned that the court forums were fun and really brought home the concept of text, and excellent ways to learn the materials. Many benefits of including discussions in online courses were identified from interviews with instructors and student course evaluation data. The benefits could be grouped in two categories. One was that online discussions were necessary for an online course. Through discussion forums online students were able to interact with their peers instead of just learning from books and other course materials. It helped build learning community and network. Another was that discussions could be advantageous for online education. Some instructors and students mentioned that students were able to learn more from participating in online discussions compared to learning in a traditional residential environment. As one student wrote, When you go to class, you are expected to answer questions on the spot and does not really give you time to put some thoughts on your answers. In contrast, when you do it remotely, you have time to think. In addition to the advantage of having time to think deeper, compared to discussions in residential environments, one instructor pointed out that the communication in online environments could be multi-dimensional in that many students were able to participate at the same time. Furthermore, the discussions can be archived and tracked. Three concerns were identified from the student course evaluation data. The first concern was associated with the high percentage given to discussions for one s final grade. In one course, online discussions counted for 40% of students final grade. While some students commented favorably that discussion forums gave them more community feelings and net work opportunity, some students indicated their frustration. One student said, Reading a sheer number of forum postings was near impossible. Another student concurred and mentioned that s/he did not have the time nor patience to read 900 postings weekly. Some students seemed to stand in between. As one wrote, I learned plenty from the other students, but I would have participated just as much if the forum grade was 20% of my overall grade versus 40%. Two suggestions students provided for this course were to limit the number of postings and being provided with more guidance from the instructor in the discussions. The second concern was the fairness of grading. For instance, one student commented that giving high grade to students who did not participate was not fair for those who attempted to learn and contributed a lot. The third concern was associated with the depth of the discussions. According to one student, while s/he learned a lot from the discussions, s/he believed the discussions could have been deeper. As s/he wrote, The case discussions were excellent to read; however, I did feel that students often agreed too quickly in forums. One suggestion the student provided was to ask students to include at least one posting to be counterpoint. The advantage of doing this was that it would prepare them for the real world where consensus is only reached after a series of arguments. 3

Quizzes and Exams Seven of the 15 examined sections used quizzes and exams. In one course, the instructors allowed students to take the quiz (counting for 5% of their final grade) multiple times. Students would be awarded full credits however many times it took them, as long as they could get 75% of the items right. The purpose of using this quiz, according to the instructors stating in the syllabus, was to ensure everyone has a minimum foundation in the area that the course addressed. In the other courses, students were allowed to take the quizzes and exams only once. Quizzes and exams were typically timed. The duration for a quiz or exam in the examined course varied, from 70 minutes, 90 minutes, 2 hours, 6 hours, 3 days, to one week. According to a couple of interviewed instructors, timing quizzes and exams was intended to help decrease possibilities that students consulted others or books and notes. Quizzes and exams were specified to be taken individually in all the examined courses. One course asked students to sign an honor code to prevent collaboration among students. Three courses further stated clearly that quizzes and exams were closed book and closed note. In one course, the instructors used three open-book quizzes and one closed-book final exam. According to the interviewed instructor who taught and originally designed this course, the open-book quizzes and closed-book exam were intended to test the same concepts. The open-book format helped students familiarize themselves with taking tests online. The closed-book format was intended to motivate students to learn at a better level. From his perspective, if something was in one s brain, then he really learned it. Similarly, when a person was asked a question, he just went and looked it up in books, and then maybe that thing was not in his brain yet. One great concern that students had in taking closed-book quizzes and exams was associated with fairness. Several students mentioned that closed-book exams indirectly encouraged unethical behavior and could put honest students at disadvantage. This concern appeared especially strong in a course in which closed-book quizzes and exams counted for 80% of one s final grade. Students seemed to like better assessment related to the real world. As one student commented in the course evaluation: I feel I would have learned more if we did some homework assignments that connected to real world problems. Another suggestion was to avoid giving too many points to a single exam. For instance, in one course, the final exam counted for 40% of students final grade. In the course evaluation, some students suggested having two or three smaller tests during the semester instead of using only a huge one at the end. According to one student, the huge final exam was like a sudden death for an individual in the sense that there is a lack of feedback on an individual s participation. Quizzes and exams were usually perceived in the literature as assessment methods to test relevant low level of knowledge such as recalling facts. Analysis of the test items used in one course and student evaluation data indicated that when thoughtfully designed, multiple choice questions could help test higher order thinking skills as well as students ability in applying materials. As one student said in evaluation of the course, I was impressed with the quizzes. In most courses it does not seem like the quizzes actually testing your ability to apply the materials covered in the lessons, whereas the quizzes in this course did. Several other students concurred and commented that the quizzes were very good learning tools. Summary and Implications Based on the results of the study, the following suggestions are provided for instructors and instructional designers to consider in design and use of asynchronous discussion forums and online quizzes and exams in assessing students. For asynchronous discussions, instructors are suggested to include their objectives and rationales, provide clear guidance for students in participating in discussion, as well as to explain in 4

detail how their participation will be graded. In use of online quizzes and exams, following issues are suggested to be considered: Purposes of using online quizzes and exams in the course; whether students are allowed to consult to books and notes, or work in teams; whether the quizzes and exams should be timed, and how much time students should be given; whether students are allowed to try the quizzes and exams more than once; and what percentage the quizzes and exams should count for students final grade. And finally, wherever appropriate, it is advisable for instructors to covey to students their rationales for the decisions they make concerning use of quizzes and exams as well as other assessment tasks in general. References Brown, G., Bull, J., & Pendlebury, M. (1997). Assessing student learning in higher education. London: Routledge. Ford, M. L. (2002). Preparing students for assessment in the on-line class. In R. S. Anderson, J. F. Bauer & B. W. Speck (Eds.), Assessment strategies for the on-line class: From theory to practice (pp.77-82). New Directions for Teaching and Learning. no.91, San Francisco: Jossey-Bass. Harlen, W., & Crick, R. D. (2003). Testing and motivation for learning. Assessment in Education, 10(2), 169-207. Biographical Sketches Shijuan Liu is a doctoral candidate in Instructional Systems Technology and a research fellow at Kelly Direct Programs at Indiana University. She had experience working as an assistant professor at a top ten university in China as well as teaching undergraduate and graduate level courses at Indiana University. Her research interests include assessment and evaluation, online education, pedagogical use of technology, computer assisted language learning, Chinese language and culture. Her work has appeared as book chapters and in such referred journals as the Internet and Higher Education and Quarterly Review of Distance Education. She can be reached at shijliu@indiana.edu. Dr. Richard J. Magjuka is a professor in the Kelley School of Business, Indiana University. He has been the faculty chair of Kelley Direct since its inception. His primary research interests in online education include quality assurance, online pedagogy, and design and delivery of effective online courses. His scholarly work related to online education has appeared in peer-reviewed journals such as Journal of Asynchronous Learning Network, International Journal of E-Learning, and Journal of Interactive Learning Online. He can be reached at rmagjuka@iupui.edu. Dr. Xiaojing Liu is a research fellow at Kelley Direct Programs at Indiana University, Bloomington. Her research interest focuses on online learning, information systems, communities of practices, and knowledge management. Her scholarly works have appeared on Journal of Asynchronous Learning Network, International Journal of E-Learning, and Journal of Interactive Learning Online. She can be reached at xliu@indiana.edu. Dr. Seung-hee Lee is a senior research scientist at Kelley Direct Programs within Kelley School of Business at Indiana University. Dr. Lee earned her doctorate from Hanyang University in Seoul, Korea, in 2003. Her major research interests are online collaboration, reflective technologies, e-learning in higher education, and online moderating/mentoring. She can be contacted at seuselee@indiana.edu. Curt Bonk is a former accountant and CPA who received his master's and Ph.D. degrees in educational psychology from the University of Wisconsin. Dr. Bonk is now Professor of Instructional Systems Technology at Indiana University and adjunct in the School of Informatics. Curt is President of CourseShare and SurveyShare (see http://php.indiana.edu/~cjbonk/). He can be contacted at (cjbonk@indiana.edu). 5