WEBEDQUAL: DEVELOPING A SCALE TO MEASURE THE QUALITY OF ONLINE MBA COURSES Rose Sebastianelli, sebastianer1@scranton.edu, (570) 941-4287 Nabil Tamimi, tamimin1@scranton.edu, (570) 941-4288 Kingsley Gnanendran, skg355@scranton.edu, (570) 941-4190 Kania School of Management, University of Scranton, Scranton, PA 18510 ABSTRACT This paper presents preliminary work toward the development of a valid and reliable scale (WebEdQual) to measure and ultimately improve the quality of online MBA courses. Scale items reflect the various aspects of online education presumed to affect the quality of e-learning experiences. Based on data collected from MBA students who have completed online courses, we use factor analysis and derive empirically the following nine underlying e-learning quality dimensions: Professor-Student Interaction, Course Content-Structure, Content Rigor, Technology, Student-Student Interaction, Assessment, Flexibility-Convenience, Team-Based Learning, and Delivery Method. Our results indicate that several of these dimensions are correlated significantly with student outcome measures. Keywords: Online MBA, Quality, Satisfaction, Perceived Learning. INTRODUCTION The shift toward online business education is well underway as evidenced by the increasing number of business courses and business degree programs offered entirely online. In the first of a series of three papers exploring online education, Dykman and Davis [2] provide an excellent discussion of what is fueling this shift toward Web-based instruction addressing issues such as educational access, changing paradigms for teaching and learning, competition among universities, a more global perspective, improved technologies, and economic considerations. Moreover, Web-based education is rapidly gaining acceptance from both the student perspective (e.g., convenience, flexibility) as well as that of the educational institution (e.g., cost effectiveness, new markets). With some predicting that the Internet will be the primary channel for delivering MBA programs in the future, it is hardly surprising that business educators have directed attention toward improving the quality of e-learning experiences. Our study continues in this line of research as we work toward the development of a scale to measure the quality of online MBA courses. We include items that relate to various aspects of online instruction presumed to affect the quality of e-learning experiences, and based on data collected from online MBA students, we use factor analysis to derive a set of underlying e- learning quality dimensions. While the scale needs to be further refined, preliminary results do indicate that many of the extracted factors correlate significantly with outcome measures, such as perceived quality of, and student satisfaction with, online MBA courses. An empirically validated scale to measure the quality of online MBA courses (WebEdQual) would be a useful tool for both monitoring and improving the online delivery of MBA courses and programs.
LITERATURE REVIEW Best Practices A number of articles suggest approaches for assuring a quality online educational experience. Grandzol and Grandzol [5], in a review of the literature on best practices for online business education, provide guidance on course design and delivery, student services, and administration. Consistent themes include standardizing course structure, modularizing course content, giving prompt and constructive feedback, providing technical support for students, creating a learnercentered environment, providing training for faculty, and limiting class sizes. Some of these components have been shown to increase student satisfaction. For instance, students report higher levels of satisfaction and increased learning in courses with greater consistency in structure and format compared to those with less consistency [10]. Similarly, Dykman and Davis [3] reiterate many of these same best practices (e.g., a standardized approach to course design), but also stress the importance of clearly defined learning objectives and planning in an online environment where courses must be prepared almost entirely in advance. Unlike in a traditional classroom, learning objectives cannot evolve as an online course progresses. Consequently, they advocate articulating overall learning objectives for the course as well as specific learning objectives for each unit to drive course design and better define student expectations. In a study that included not only business but several other disciplines (education, liberal arts, etc.), Gaytan and McEwen [4] surveyed both faculty and students about their perceptions regarding online education. Specifically, they were interested in the types of instructional strategies and techniques for assessing student performance that are perceived to enhance the quality of online learning. The major recommendations from their findings are that (1) faculty should use a variety of techniques for evaluating student work such as exams, quizzes, projects, portfolios, discussion boards, and (2) students need meaningful and timely feedback to facilitate online learning. Given the obvious concerns faculty have about using online exams to assess student performance (e.g., the inability to verify the identity of the test taker in an unproctored online environment), Khare and Lam [6] provide a critical evaluation of online exams both in terms of pedagogical and technical issues. They suggest that online exams should focus on mastery of knowledge and competence of skills rather than on factual recall. Furthermore, based on data from an online MBA program, they conclude that a comprehensive online exam is one of several important assessment tools for measuring effectiveness and providing feedback. Online MBA Outcomes Several empirical studies have focused specifically on factors that affect outcome measures in online MBA courses and programs. In an early study, Bocchi, Eastman and Swift [1] profiled students in an online MBA program and identified strategies that were successful in addressing potential attrition issues. These students, who tended to be older with significant professional business experience, cited accessibility, convenience, fit with career and personal growth plans as their primary reasons for joining the online MBA program. They found the following retention strategies to be successful: a cohort and team-based approach to learning, extensive faculty feedback and interaction, relevant content and activities, and faculty who are both interested and competent in teaching online.
Exploring the possibility that more complex relationships exist among the variables affecting perceived quality, perceived learning and satisfaction in online courses, several studies employed structural equation modeling (SEM) to examine proposed interdependencies. Marks, Sibley and Arbaugh [7] used SEM to test their theoretical model that shows online instructional factors (instructor-student interaction, student-student interaction and student-content interaction), online educational advantages (e.g., convenience) and student characteristics (e.g., gender) affecting satisfaction indirectly through their impact on perceived learning. Using data collected from students taking online courses within a traditional MBA program, they found that instructor-student interaction is the most important factor affecting perceived learning/satisfaction in an online environment. Student-student interaction, few (but not all) student-content interaction variables, and online educational advantages were also found to be significant. Online educational advantages, although significant, were found to be the least important. Finally, student characteristics were not found to significantly impact perceived learning/satisfaction in online courses. Another study, carried out by Peltier, Schibrowsky and Drago [9], also used SEM but to test a much more complex model of interrelationships among various dimensions and the perceived quality of online learning. Their study included the following six dimensions of online teaching quality: (1) student-to-student interactions, (2) student-to-instructor interactions, (3) instructor support / mentoring, (4) lecture delivery quality, (5) course content and (6) course structure. Not only were interrelationships among these dimensions proposed in the model, but each dimension was hypothesized to have a direct positive relationship on the perceived quality of an online learning experience. Based on data collected from students enrolled in an online MBA program at a Midwestern university, they found, among several significant interrelationships, that three dimensions (instructor support / mentoring, course content, and course structure) have a significant positive relationship with the perceived quality of online learning. No student characteristics were considered in this study. Research Setting METHOD This study was carried out in conjunction with the MBA program at a private comprehensive university. Both traditional and online MBA programs are offered, with the online program having been initiated in the spring of 2008. Traditional MBA students may opt to take up to six credits online; however, students entering either program may be required to take up to twelve one-credit foundation modules which are offered solely online. Consequently, most traditional MBA students take at least one course online. The online MBA program is offered via partnership with a provider of online education that helps academic institutions with program and course development, marketing and publicity, student recruitment and retention, and technical support services. As such, all online courses in the program follow many best practices cited in the literature such as consistent course structure, modularized content, 24/7 help desk for students, explicit student learning objectives, extensive training for faculty on ANGEL (the Web platform used for course delivery), and limited class sizes. Multiple options are used to encourage a high level of professor-student and student-
student interaction. These include a discussion forum in which students are required to make a minimum of two posts per week in response to questions posed by the instructor, an ask the instructor forum for students to raise questions about problems they encounter with course material, a student lounge designed for more social, non course-related discourse, and e-mail. Questionnaire The questionnaire consists of two sections. The first section includes a few questions that gather identifying information (i.e., student ID number) and some background data (e.g., employment status). The second section consists of a series of statements for which students are asked to indicate their level of agreement on a seven point Likert-type scale (1 = strongly disagree, 2 = disagree, 3 = somewhat disagree, 4 = neither agree nor disagree, 5 = somewhat agree, 6 = agree, 7 = strongly agree). These scale items were developed to capture the major factors thought to impact students perceptions of, and satisfaction with, online learning that have been cited in the literature. We include items that represent all six dimensions (course content, course structure, lecture delivery quality, professor-student interaction, student-student interaction, and instructor support/mentoring) used in previous research [9]. We also include items to measure online educational advantages, such as convenience, that have also been cited [7]. Finally, we develop items to capture two new additional factors, assessment of student performance and technology issues. The number of items totaled 59. We also include three statements to capture student outcome measures, namely students perceptions of the quality of their online MBA courses, student satisfaction with their online MBA courses, and the extent to which students believe that they have met the learning objectives in their online MBA courses. We pre-tested the questionnaire using a small sample of traditional MBA students who had taken at least one online course. Based on the pre-test results, some items were eliminated or reworded to improve clarity. In accordance with standard practice, some questions were negatively worded and the order of the items was randomized (similar questions relating to the same aspect of online instruction were not presented together). This was done to reduce the potential of any halo effects. The complete list of items is shown in the Appendix. Data Collection Students in the online MBA program and those in the traditional MBA program who had taken at least one course online were contacted via e-mail to participate in the study. As this is part of an ongoing longitudinal study, the data for this paper were collected at two different points in time, in mid-august 2009 and then again in mid-may 2010. The e-mail providing a link to the online survey was sent to a total of 280 online MBA students and 100 traditional MBA students. An incentive lottery was used to increase participation. Respondent Profile RESULTS A total of 110 students completed our online survey resulting in a 29% response rate. Of those responding, 75 % are enrolled in the online MBA program, 85% are employed full-time, and
55% are male. They report an average of 11 years professional experience, ranging from none to 35 years. It should be noted that students applying to the online MBA program must have a minimum of 3 years of professional (or supervisory) experience; however there is no such admission requirement for students in the traditional on-campus MBA program. Factor Analysis In order to exploit the intercorrelations among the 59 items representing various aspects of online instruction, factor analysis was performed to identify the underlying e-learning quality dimensions. Prior to analysis, the mean item rating was used to replace any missing values for individual items and the responses to negatively worded questions were reversed. Based on an examination of the scree plot, nine factors were extracted to account for about 54% of the total variation in the observed ratings. Figure 1 shows the items that loaded strongly on each of the nine factors. In developing this factor solution, items with loadings less than 0.35 (after varimax rotation) were dropped. Under each factor the items are listed in descending order according to loading magnitude. The nine dimensions extracted are labeled based on the specific items that load most heavily on each factor. Therefore, the nine underlying dimensions are Professor-Student Interaction, Course Content-Structure, Content Rigor, Technology, Student-Student Interaction, Assessment, Flexibility-Convenience, Team-Based Learning, and Delivery Method. In order to assess the internal consistency of these derived factors, Cronbach s alpha was computed as a measure of reliability. These are also reported in Figure 1. FIGURE 1: Factor Solution: Items Loading on Each Factor Factor I Professor-Student Interaction (Cronbach s alpha =.891) My online professors are very responsive to students concerns. Most of my online professors respond to questions in a timely manner. Most of my online professors actively facilitate discussion in forums. Most of my online professors participate in the Student Lounge on a regular basis. My online professors often identify key points to facilitate learning. My online professors show interest in students progress. I find the professor s responses to questions posted to the Ask the Instructor Forum helpful. In general, there is little interaction between the professors and students in my online courses. My online professors adjust their methods of instruction based on student feedback. Course content is communicated effectively in my online courses. My online professors encourage the expression of different viewpoints. My online professors design course content to stress important concepts. Feedback from professors is meaningful in my online courses.
Factor II Course Content - Structure (Cronbach s alpha =.801) My online MBA courses are current and up to date. Tasks for the week helps me to meet course requirement deadlines. The content in my online courses is applicable and useful to professional work situations. My online courses are organized in a way that is easy to navigate. The consistent format for each course makes it easy for me to access materials I need for the week. The content in my online courses add value to my MBA educational experience Individual assignments helped me to understand content through application of concepts and skills covered in the course. In general, the exams in my online courses are fair. The weekly overview and objectives clearly identify learning goals to be achieved in the upcoming week. Factor III Content Rigor (Cronbach s alpha =.495) The content in my online courses is less rigorous than I expected. The content in my online courses is challenging. In general, my online courses cover fewer topics than I expected. I don t spend much time studying for online exams. In general, my online courses cover fewer topics than I expected. Factor IV Technology (Cronbach s alpha =.692) Technology problems interfere with my online learning. I am often frustrated with technology in my online courses. I sometimes encounter problems accessing materials in my online courses. I am sometimes overwhelmed by the volume of posts to the discussion forum. Technical support is available when I need it. More often than not I felt intimidated asking my online professor questions. I would generally classify myself as being tech-savvy. Factor V Student-Student Interaction (Cronbach s alpha =.671) Most students participate more than required in Discussion Forums. Other students posts to the Discussion Forum are helpful in understanding different viewpoints. I learn more from my fellow students in online courses than I have in traditional classroom settings. I don t feel comfortable asking my online professors for advice. My online courses offer me ample opportunities for problem solving experiences. Other students posts to the Discussion Forum are not useful in learning course content. Factor VI Assessment (Cronbach s alpha =.999) The grades I receive in my online courses accurately reflect my performance. I like having access to course materials during online exams.
Factor VII Flexibility-Convenience (Cronbach s alpha =.574) Taking courses online makes it easy for me to balance my education with other responsibilities. I consider not having to commute to class a major advantage of online courses. Enrolling in online courses allows me to complete my MBA in less time than expected. PowerPoint slides are an effective way to deliver course content online. I like that course content becomes available weekly rather than all at once at the beginning of my online courses. Factor VIII - Team-Based Learning (Cronbach s alpha =.146) Student evaluation is more likely based on individual rather than group work in my online courses. Group work is encouraged by professors in my online courses. In general, my online professors are supportive of my personal circumstances. I have enough opportunities for collaborative team work in my online courses. Factor IX Delivery Method (Cronbach s alpha =.280) I would like to see more video audio content available in my online courses. Video audio presentations facilitate learning the key topics in my online courses. I prefer online courses that use a variety of methods (e.g., assignments, discussion posts, exams) to evaluate student performance. The online environment is not favorable for learning certain topics typically taught in MBA programs. Correlation Analysis In order to determine how these dimensions relate to the outcome measures of perceived quality, satisfaction, and student achievement of learning objectives, correlations were computed between the average ratings of the items comprising each factor and responses to the three outcome items. These correlations appear in Table 1. TABLE 1: Correlations of Factors with Outcomes Quality Satisfaction Learning Objectives I. Professor-Student 0.449 0.678 0.168 II. Content-Structure 0.629 0.551 0.407 III. Content Rigor 0.322 0.249-0.007 IV. Technology 0.222 0.039 0.344 V. Student-Student 0.631 0.455 0.203 VI. Assessment 0.257 0.108 0.196 VII. Convenience 0.202 0.463 0.151 VIII. Team-Based 0.297 0.241 0.344 IX. Delivery Method 0.326 0.201 0.208
DISCUSSION AND IMPLICATIONS While a number of studies have considered how dimensions of online instruction relate to perceived quality, perceived learning and satisfaction in online courses, to the best of our knowledge these dimensions were determined a priori. In other words, researchers measured these dimensions using items that they grouped together prior to gathering student responses. Consequently, our study adds to the literature by attempting to derive these underlying dimensions empirically using factor analysis on student responses. That said, our study is exploratory in nature and more work is needed to refine our scale. Cronbach s alpha values for a few of our derived factors are too low. Although the generally acceptable minimum alpha is usually 0.70, Nunnally [8] suggests a somewhat lower threshold, such 0.50, for exploratory work involving the use of newly developed scales. The final two factors extracted in our analysis, Team-based Learning and Delivery Methods, fail to meet even this lower threshold. Nonetheless, we are encouraged that a number of our better defined and more reliable factors are congruent with those cited in the literature and used in previous research studies. The first two factors extracted in our analysis, Professor-Student Interaction and Course Content-Structure, have a sufficient number of strongly loading items as well as suitable values for Cronbach s alpha indicating a high level of internal consistency. Moreover, these two dimensions have been found to be significant in predicting student outcome measures [7] [9]. Our correlations are consistent with these previous findings (see Table 1). Student responses to the outcome item My online courses are of high quality are correlated fairly strongly (0.629) with the average ratings across the items comprising Factor II (Course Content-Structure). In addition, student responses to the item I am very satisfied with the online courses in my MBA program are correlated fairly strongly with average ratings across the items comprising Factor I (Professor- Student Interaction). Our preliminary results also suggest that perceived quality is correlated fairly strongly with Student-Student Interaction and that satisfaction is correlated moderately with Course Content- Structure and Flexibility-Convenience. With respect to students perceptions about meeting learning objectives in their online MBA courses, we find this to be the most correlated with Course Content-Structure, Technology, and Team-Based Learning; while significant at α = 0.05, these correlations are not very strong. An obvious limitation of our study is the sample size. In addition to enabling more definitive results regarding the extracted dimensions, more observations would have allowed us to test a complex, and perhaps more realistic, model of interrelationships among the factors and outcome measures using SEM. The online MBA program being studied is growing rapidly, and as noted above, we plan to continue surveying students as part of an ongoing longitudinal approach. Further work is needed to refine our scale items. Some items may have to be discarded (those that did not load on any factors), some reworded, and some new items added. The goal is to develop a reliable and valid scale that can be used to measure and ultimately improve the quality of online MBA education.
APPENDIX: Items on Questionnaire 1. The online environment is not favorable for learning certain topics typically taught in MBA programs. 2. The content in my online courses add value to my MBA educational experience 3. The content in my online courses is applicable and useful to professional work situations. 4. My online courses offer me ample opportunities for problem solving experiences. 5. The content in my online courses is challenging. 6. The content in my online courses is less rigorous than I expected. 7. In general, my online courses cover fewer topics than I expected. 8. Topics covered in my online courses are easy to learn. 9. My online MBA courses are current and up to date. 10. Individual assignments helped me to understand content through application of concepts and skills covered in the course. 11. The weekly overview and objectives clearly identify learning goals to be achieved in the upcoming week. 12. The consistent format for each course makes it easy for me to access materials I need for the week. 13. Tasks for the week helps me to meet course requirement deadlines. 14. My online courses are organized in a way that is easy to navigate. 15. I like that course content becomes available weekly rather than all at once at the beginning of my online courses. 16. PowerPoint slides are an effective way to deliver course content online. 17. I would like to see more video audio content available in my online courses. 18. Online materials contain information not covered in the textbook. 19. Course content is communicated effectively in my online courses. 20. Video audio presentations facilitate learning the key topics in my online courses. 21. My online professors design course content to stress important concepts. 22. I regularly post questions to the Ask the Instructor Forum. 23. I find the professor s responses to questions posted to the Ask the Instructor Forum helpful. 24. Most of my online professors actively facilitate discussion in forums. 25. In general, there is little interaction between the professors and students in my online courses. 26. Most of my online professors respond to questions in a timely manner. 27. Most of my online professors participate in the Student Lounge on a regular basis. 28. Feedback from professors is meaningful in my online courses. 29. More often than not I felt intimidated asking my online professor questions. 30. Most students participate more than required in Discussion Forums. 31. Other students posts to the Discussion Forum are helpful in understanding different viewpoints. 32. Other students posts to the Discussion Forum are not useful in learning course content. 33. I have enough opportunities for collaborative team work in my online courses. 34. I learn more from my fellow students in online courses than I have in traditional classroom settings. 35. Group work is encouraged by professors in my online courses.
36. Student evaluation is more likely based on individual rather than group work in my online courses. 37. I am sometimes overwhelmed by the volume of posts to the discussion forum. 38. In general, my online professors are supportive of my personal circumstances. 39. I don t feel comfortable asking my online professors for advice. 40. My online professors are very responsive to students concerns. 41. My online professors encourage the expression of different viewpoints. 42. My online professors show interest in students progress. 43. My online professors adjust their methods of instruction based on student feedback. 44. My online professors often identify key points to facilitate learning. 45. I consider not having to commute to class a major advantage of online courses. 46. Taking courses online makes it easy for me to balance my education with other responsibilities. 47. Enrolling in online courses allows me to complete my MBA in less time than expected. 48. I like the ability to access my online courses at any time. 49. Online exams should be included as one way of assessing student performance in every course. 50. In general, the exams in my online courses are fair. 51. The grades I receive in my online courses accurately reflect my performance. 52. I like having access to course materials during online exams. 53. I don t spend much time studying for online exams. 54. I prefer online courses that use a variety of methods (e.g., assignments, discussion posts, exams) to evaluate student performance. 55. I would generally classify myself as being tech-savvy. 56. Technical support is available when I need it. 57. Technology problems interfere with my online learning. 58. I am often frustrated with technology in my online courses. 59. I sometimes encounter problems accessing materials in my online courses. Outcome Measures My online courses are of high quality. I am very satisfied with the online courses in my MBA program. I have met the learning objectives in most of my online MBA courses. REFERENCES [1] Bocchi, J., Eastman, J.K., & Swift, C.O. Retaining The Online Learner: Profile of Students In An Online MBA Program and Implications for Teaching Them. Journal of Education for Business, 2004, 79 (4), 245-253. [2] Dykman, C.A. & Davis, C.K. Online Education Forum Part One The Shift Toward Online Education. Journal of Information Systems Education, 2008, 19 (1), 11-16.
[3] Dykman, C.A. & Davis, C.K. Online Education Forum Part Three A Quality Online Educational Experience. Journal of Information Systems Education, 2008, 19 (3), 281-289. [4] Gayton, J., & McEwen, B. C. Effective Online Instructional and Assessment Strategies. The American Journal of Distance Education, 2007, 2(3), 117-132. [5] Grandzol, J. R., & Grandzol, C. J. Best Practices for Online Business Education. International Review of Research in Open and Distance Learning, 2006, 7(1), 1-18. [6] Khare, A., & Lam, H. Assessing Student Achievement and Progress with Online Examinations: Some Pedagogical and Technical Issues. International Journal on E- Learning, 2008, 7(3), 383-402. [7] Marks, R.B., Sibley, S.D., & Arbaugh, J.B. A Structural Equation Model of Predictors for Effective Online Learning. Journal of Management Education, 2005, 29 (4), 531-563. [8] Nunnally, J.C. Psychometric Theory, 2 nd ed., New York: McGraw-Hill, 1978. [9] Peltier, J.W., Schibrowsky, J.A., & Drago, W. The Interdependence of the Factors Influencing the Perceived Quality of the Online Learning Experience: A Causal Model. Journal of Marketing Education, 2007, 29 (2), 140-153. [10] Shea, P.J., Fredericksen, E.E., Pickett, A.M., Pelz, W. & Swan, K. Measures of Learning Effectiveness in The SUNY Learning Network. In J. Bourne and J.C. Moore (Eds.) Online Education: Learning Effectiveness, Faculty Satisfaction and Cost Effectiveness, 2001, 31-54.Needham, MA: Sloan C.