Toward a Value Framework for Online Learning Systems

Transcription

1 /02 $17.00 (c) 2002 IEEE 1 Toward a Value Framework for Online Learning Systems Yair Levy and Kenneth E. Murphy Florida International University College of Business Administration levyy@fiu.edu Abstract Many universities and private corporations are investing significant capital in online learning initiatives. Willingness on the part of the student to take part in online learning is essential for success. This paper queries the student concerning the features of these systems that add value to their learning experience. A questionnaire is utilized for gathering student values associated with online learning systems and the organization supporting these technologies. Results from the survey and a literature review are organized in a proposed value framework for online learning course effectiveness. Keywords Distance Learning, Asynchronous Learning Networks (ALN), Online Learning Systems, Customer Values. 1. Introduction Educational institutions and private corporations have and will continue to invest significant capital in online learning technology, training and system administration. It is estimated that the portion of online learning market devoted to technical training alone will grow from $1.7 billion in 2000 to about $5.3 billion in 2003 [16]. Given the magnitude of investments into synchronous and asynchronous learning networks (ALNs), it is prudent to gain a better understanding of the factors that influence the learner s perception of course effectiveness in these environments. This paper proposes and begins to build a framework for understanding how the values of students affect their perceptions of course effectiveness within the virtual classroom. The fundamental research hypothesis underlying this initial research is that values related to online learning system configuration, including technology, IT support personal and IT management structure, play a significant role for students in their perception of online course effectiveness. This paper begins the process of validating the hypothesis above by elucidating learner values for online learning systems. Within a course, a learner value is defined as a factor that influences course effectiveness, that is, a factor that adds value to the course for the student. The literature contains significant research focusing on the effect of student opinions on course effectiveness; however, values related to online system configuration have not generally been included in the prior work. After gathering values related to system configuration, similar answers will be grouped to form the core set of values related to system configuration. These along with values found in prior research, (e.g., course organization, interaction, student motivation, etc.), shall be utilized to build a comprehensive set of learner values. In subsequent work, these values will be hypothesized as influencing the perceived course effectiveness in virtual learning. To accomplish this the reliability of these values must first be validated, and a model for predicting a student s perception of online course effectiveness based on these values will be built. A framework allowing decision makers to consider the relevant values in developing effective online learning programs will be the ultimate result. Understanding the levers affecting student perception of online learning effectiveness should be of great interest for administrators, practitioners, and researchers. In the quest to elucidate the set of values that influence the effectiveness of value of online learning systems this study follows the approach of Keeney [11]. Values are gathered through structured questionnaire with students who are enrolled in online learning courses. The students taking part in the study are graduate students in a college of business enrolled in online assisted (in person and online) and mixed (synchronous and asynchronous) online courses. Constructing the set of student values is an important endeavor for the long-term goal of this research stream, but it is also important on its own as the opinions of students with respect to technology and support have not been investigated in depth in online learning environments.

2 /02 $17.00 (c) 2002 IEEE 2 2. Online Learning Networks Asynchronous Learning Networks are defined on the ALN s website [1] as: people networks for anytime - anywhere learning. ALN combines self-study with substantial, rapid, asynchronous interactivity with others. In ALN learners use computer and communications technologies to work with remote learning resources, including coaches and other learners, but without the requirement to be online at the same time. The most common ALN communication tool is the World Wide Web. Implicit in the above definition is both the technical infrastructure as well as the students, instructors and many other facets of most learning environments. ALNs are, of course, not possible without the technology and systems that facilitate the delivery of the course. Among the biggest providers of applications that assisting the delivery of ALNs are WebCT ( Blackboard ( and Lotus LearningSpace ( For the purpose of this study the definition of ALN will be extended in two ways. First the definition is broadened to include synchronous as well as asynchronous learning over computer networks and any combination of the two approaches. Synchronous learning systems on the Internet include slide show, application demonstration, web demonstrations, breakout rooms, and two-way audio between the professor and students, whiteboard and instant surveys. These features are part of most virtual classroom environment systems available in the market today. Some systems utilize the telephone lines to deliver the audio and the Internet to deliver the data, while other tools use the Internet infrastructure for both the audio and the data delivery. Examples of the companies providing these systems are Centra ( eroom ( PlaceWare ( Using the definition of ALN, an Online Learning Network (OLN) is: a system that enables students learning via the Internet. OLNs facilitate interaction of professor-tostudents, student-to-professor and students-tostudents communication via asynchronous learning tools, i.e., anytime, anywhere learning or synchronous learning tools, i.e., real-time communication, or any combination of these two. The definition of OLN will be further extended beyond the course content, interaction and the use of technology to include the elements of systems management inherent in OLN. Specifically the configuration of the organization responsible for supporting the technology and the management of this organization are aspects of the online learning delivery system that should be included. Alavi & Leidner [3] argue that research in online learning must be extended to account for how technology can enhance learning; clearly online learning system configuration falls within this scope. Henceforth an Online Learning System (OLS) is defined to be an OLN as well as the technological, organizational and managerial infrastructure for the delivery of this service. From the instructor and learner perspective an OLN combines self-study with substantial, rapid, asynchronous interactivity with others and/or synchronous learning experiences. For managers OLNs provide challenges in service delivery and quality, systems management and security as well as personnel management. The definition of OLS is meant to encompass both aspects: learner, instructor, class, technological and system configuration, and management of the OLN. The definition of OLS can be specialized to both the synchronous and asynchronous learning systems, e.g., an asynchronous online learning system is the entire technological, organizational and management system that facilitates and delivers the ALN. Pure synchronous learning systems are probably rare in practice as most instructional courses consist not only of classroom activities and lectures, but also include outside the classroom discussions, homework assignments or other activities, which can be achieved in asynchronous mode. Certain educational institutions have begun to integrate both the synchronous and asynchronous online learning systems to form mixed OLS. By integrating the two systems, students are required to attend some live lectures via the Internet, and at later time, to submit assignments via a course website, communicate, collaborate, and contribute to discussions outside the classroom with classmates and the instructor utilizing asynchronous means, and to read the course notes prior to the live sessions. Alavi and Leidner [4] provide additional breadth and depth to the taxonomy of virtual learning environments. A diagram outlining the relations among the different learning networks presented here can be seen in Figure Student Values in Course Effectiveness Measuring and validating student perception of course effectiveness has received significant attention in the literature. Cashin [7] reports that more than 1500 books and articles debating the subject of utilizing student evaluations to measure course effectiveness in the classroom. A related line of studies has focused on comparing effectiveness between technology based and classroom based learning [5,8,9,10,15]. Much of this work contains references to student perceptions of the learning environment that are hypothesized to influence the overall effectiveness of a course.

3 /02 $17.00 (c) 2002 IEEE 3 Learning Systems and Methods Distance Learning Face to Face Learning Correspondance Learning Systems Online Learning Systems (OLS) Videoconfrencing and Satellite Broadcasting Learning Systems Online Assisted Courses In Class Lectures or Labs Synchronous Online Learning Systems (SLS) Mixed Asynchronous and Synchronous Online Learning Systems (MLS) Asynchronous Online Learning Systems (ALS) Figure 1: Proposed Learning Systems Hierarchy The current paper is not directed at building a model of course effectiveness, but instead focuses on generating a set of values that may influence course effectiveness in OLSs. Although there is no intrinsic value to the learner in the technology or the associated management system that delivers education over the Internet, the value of online education is the net gain to the student from both the education received and the means by which it was delivered, obtained and supported. Keeney [11] makes a parallel definition for the values of customers who make purchases using the Internet. Learners may view the value of online learning, even the same course, in many different ways. One might find it challenging to use the technology, while another may enjoy the freedom and flexibility that the technology provides. Given the flexibility afforded students engaged in online learning, two students may go about grasping a concept in a completely different manner; one may engage in the course content first, another may work to answer the homework questions before any other study and a third may interact with colleagues having learned that this is most expedient. Thus the values of students in learning online would depend in part on the perceived knowledge gained and in part on the means by which it was obtained. A number of studies of course effectiveness in a variety of settings have elucidated the values of students with respect to course effectiveness. Marks [12], in a thorough, empirical work on traditional classroom learning evaluations, suggests that there are five factors that influence a student's overall evaluation of a course. These include course organization, workload and difficulty, expected fairness of grading, liking and concern of instructor, and perceived learning. These factors are constructed based on similar answers to survey questions with respect to these five factors. Student attitudes towards course organization, course difficulty and workload and towards the course subject itself as well as the instructor were found to influence overall course effectiveness. In a book that focuses solely on distance education Moore & Kearsley [13] present a broad perspective of distance education theory, programs and administration. Drawing on a significant body of literature these authors outline many factors that should be considered in designing effective distance learning courses including technical support and course content. In addition studentprofessor and student-student interaction, learner control over the course, course difficulty, a student s interest in the subject as well as course cost are purported to affect the student s experience. Lifestyle factors including employment and family issues may also affect a student s desire for success in distance education that in turn may influence their opinion of course effectiveness. Webster & Hackley [17] studied teaching effectiveness in a videoconference oriented distancelearning setting. Two of the major findings were that content or medium richness, defined as the flexibility and variety of the delivery technology, and the degree of student-professor interaction positively affected student opinion of distance learning effectiveness. Lack of technology quality (i.e., number of interruptions, delays and errors,) and the student s lack of comfort with technology were found to negatively affect student opinion of course effectiveness. Carswell, et. al., [6], in a case study, investigate student opinions of Internet technologies in transforming the distance learning experience. These students, from the Open University, stated that online teaching brought them additional flexibility and rapidity of response. Carswell, et. al., [6] also found that interaction and comfort with technology influenced student attitudes toward the medium. In another case study of a synchronized videoconference course between two universities, Alavi, Yoo & Vogel [5] found content quality and instructor-student and student-

4 /02 $17.00 (c) 2002 IEEE 4 student interactions enhanced student s learning experience. Keeney [11] studied the values of customers making purchases on the Internet. In this survey, customers were asked what it was in their Internet purchase experience that added value. Like responses were categorized, and the results were organized in to ten fundamental value objectives related to Internet commerce. The objectives found in this study included minimizing cost, maximizing convenience, minimizing time spent, maximizing enjoyment and security. Since the medium in which the transaction is occurring is similar to that of online learning, the values underlying these objectives may be applicable in a distance learning setting as well. The results of these research papers both in distance learning and elsewhere are utilized to form a basic set of values that student may have for OLSs. These values from the literature were grouped according to three broad technological themes: technological, course and instructor and student. The result is shown in the table in Appendix A with twelve categories in all. The questionnaire will enhance those values found in the literature by surveying students enrolled in distance learning courses focusing on the system configuration aspect of OLSs. 4. Research Methodology To augment the set of student values with respect to OLSs, a qualitative questionnaire was utilized to gather values for online learners. The methodology and structure behind the questionnaire was guided by Kenney s [11] approach of gathering qualitative values of customers making Internet purchases. Reponses that are alike will then be grouped together to form the set of fundamental values that can be utilized in the subsequent stages of this work. These will be included in follow-up surveys validating the reliability of student values and in building a structural model for predicting course effectiveness. The questionnaire was delivered online to graduate students who are in online assisted as well as mixed synchronous and asynchronous online courses. Questionnaire participants were asked to identify specific values with respect to the online learning system and system configuration that were important to their educational experience. As in Keeney [11] the questionnaire is divided into parts, the first of which is completely open-ended allowing respondents the opportunity to express all possible pros and cons of the system. Subsequently the questionnaire follows a more structured approach, stimulating thinking with queries about specific aspects of the system. To eliminate duplications and insure the integrity of the data, the questionnaire was administered on a server that allowed access to registered students only. Each questionnaire was set to gather both the username and Internet protocol (IP) number of the computer used by the student to submit the data. No duplications of questionnaire submissions were found in the survey. A sample of the open-ended questionnaire can be found in Appendix B. All of the responses to each of the questions are then converted in a common form. For example, if one student suggests that system uptime is an important factor and another student suggests that availability of the system is holds value then these two responses may be grouped under the term system uptime. Once the like responses are converted to similar terms, these fundamental values will then be organized under the same categories as those from the literature. 5. Results and Discussion The initial questionnaire was implemented in May of 2001 to two cohorts of students enrolled in two different graduate programs in the college of business in a state university in southeastern United States. Response to the questionnaire was limited to 9 observations out of 11 students (or response rate of about 82%) using the mixed online learning system configuration. In the second program, which utilized the asynchronous online assisted learning system only, response was 27 observations out of 36 students (or response rate of 75%), which results in an overall response rate of 77%. The results of the questionnaire are shown in the last column of appendix A. It is not surprising given the focus of the survey most of the new values relate to aspects of the online learning system. These include the value of a wide variety tools available in the online learning system including chat and discussion forums as well the ability to access all courses from a single portal. Students also liked the homogenous interface for all courses. The ability to take quizzes remotely and to review course material at their leisure were also features of the online learning system that added value. On one hand this seems to indicate that students like the flexibility that a variety of technologies or tools provide within a single course, while at the same time the students also value the fact that each tool only needs to be learned once. This would seem to support standardizing online learning on a single technology platform across an entire college or school. The students also expressed some the challenges that they face in online learning settings. Not surprisingly, students stated that they were challenged by connectivity issues and in some cases by Internet service provider problems. Interestingly, however, students also desired to be supported via in additional to by telephone and in person. This indicates that while the last mile problem may still remain, students in this sample, while still sensitive to network reliability issues, tended to be

5 /02 $17.00 (c) 2002 IEEE 5 technologically savvy and comfortable with support. This questionnaire showed similar results with respect to interaction as several of the papers in the literature review. Students engaged in distance learning are much more likely to be distracted by whatever is happening in close proximity. In this survey students in synchronous learning situations reported being distracted by the surroundings in their office or home while in session, and even, by talking (i.e., text chat,) with other students in the course that was occurring during these class sessions. Further research into whether talking during class has a positive or a negative influence on the student s opinion of course effectiveness is warranted. The results of this initial questionnaire also supported many of the factors identified by Keeney [11], Marks [12], Moore & Kearsley [13] and Carswell et. al. [6]. For example, lifestyle values played a major roll for the mixed OLS students. This result has also been mentioned in the literature by Ponzurick et. al., [14] where it was found that a major reason for MBA students to engage in distance education was so that work, travel and family responsibilities could be met. System reliability and technical support were also found to play a roll when looking at the values of both mixed and asynchronous assisted students. No attempt was made here to present the relative strength of any of the responses by frequency or otherwise as this effort was directed only at enhancing the set of possible student values for course effectiveness in OLS. 6. Conclusions This paper extends the definition of asynchronous learning networks to online learning systems in order to include the systems configuration (technology and management) aspects of online learning systems. The literature review provided for empirical studies that had obtained student views concerning factors that added value to learning environments both traditional and online and consumer views about factors that add value to Internet purchases. These opinions concerning the costs and benefits associated with classroom learning and online transactions make up a proposed set of student values for online learning. Finally, these values were augmented through the use of a questionnaire to students engaged in online learning coursework. Future research in this area will include two major phases. Once the broad set of potential student values is found based on extensive literature review, the current questionnaire and additional research, a survey for online learners will be created to both validate the proposed values and to create constructs representing distinct grouping of those values along the line of the framework presented in Appendix A. This will be followed by a second survey and subsequent structural modeling that will result in a causal model to predict course effectiveness for OLS based on these constructs. The goal is to understand how student values in OLS affect course effectiveness from the learner s point of view especially with respect to OLS configuration. This work is in itself important for it both builds a framework for student values in OLS from a diverse set of literature and augments that framework using a simple open-ended questionnaire methodology to gather relevant student views. OLS configuration has not been studied widely in the literature, and it is hoped that this analysis provides some insight into the importance of systems and support in online learning initiatives. Managers of these initiatives should gain from a better understanding of the values of student stakeholders. References [1] ALN website ( [2] M. Alavi, Computer-mediated Collaborative Learning: An Empirical Evaluation, MIS Quarterly, 18(2), 1994, pp [3] M. Alavi, and D. Leidner, Research Commentary: Technology Mediated Learning-A Call for Greater Depth and Breadth of Research, Information Systems Research, 12(1), 2001a, pp [4] M. Alavi, and D. Leidner, Virtual Learning Systems, Unpublished Manuscript, 2001b. [5] M. Alavi, Y. Yoo, and D. Vogel, Using Information Technology to Add Value to Management Education, Academy of Management Journal, 40(6), 1997, pp [6] L. Carswell, P. Thomas, M. Petre, B. Price, and M. Richards, Understanding the 'Electronic' Student: Analysis of Functional Requirements for Distributed Education, Journal of Asynchronous Learning Networks, 3(1), 1999, pp [7] W.E. Cashin, Student Ratings of Teaching: The Research Revisited. Instructional Development and Effectiveness Assessment Paper No. 32. Manhattan: Center for Faculty Evaluation and Development, Kansas State University, [8] D.H. Clements, Enhancement of Creativity in Computer Environments, American Education Research Journal, 28(1), 1991, pp [9] C.P. Funkhouser, The Influence of Problem-Solving Software on Student Attitudes about Mathematics, Journal of Research on Computing in Education, 25(3), 1993, pp [10] C.M. Gardner, P.E. Simmons, and R.D. Simpson, The Effects of CAI and Hands-on Activities on Elementary Students Attitudes and Weather Knowledge, School Science and Mathematics, 92(6), 1992, pp

6 /02 $17.00 (c) 2002 IEEE 6 [11] R. Keeney, The Value of Internet Commerce to the Customer, Management Science, 45(4), 1999, pp [12] R. B. Marks, Determinants of Student Evaluations of Global Measures of Instructor and Course Value, Journal of Marketing Education, 22(2), 2000, pp [13] M. Moore, and G. Kearsley, Distance Education: A Systems View, Wadsworth, Belmont, Ca., [14] T.G. Ponzurick, K.R. France, and C.M. Logar, Delivering Graduate Marketing Education: An Analysis of Face-to-Face Versus Distance Education, Journal of Marketing Education, 22(3), 2000, pp [15] G.L. Reglin, CAI Effects on Mathematics Achievement and Academic Self-Concept Seminar, Journal of Education Technology Systems, 18(1), 1989, pp [16] T. Smalley-Bowern, E-learning tested, InfoWorld, [17] J. Webster, and P. Hackley, Teaching Effectiveness in Technology-mediated Distance Learning, Academy of Management Journal, 40(6), 1997, pp Yair Levy Yair Levy is instructor and online learning program director at the College of Business Administration at Florida International University. Mr. Levy is a Ph.D. candidate in MIS at Florida International University. He earned his Bachelor's degree in Aerospace Engineering from the Technion (Israel Institute of Technology). He has received his MBA with MIS concentration from FIU. His current research interests are in the area of online learning, elearning, web based training (WBT), and Internet based distance-learning systems. He served as a research reviewer for the International Conference on Information Systems (ICIS). He teaches Telecommunications and Networking, Web Management, and E-commerce Technologies for Managers in the Master of Science in MIS program and for undergraduate MIS majors. Between 1995 and 1998 he assisted NASA in developing e-learning platforms and management of Internet and Web infrastructures. Kenneth E. Murphy Dr. Kenneth E. Murphy (Ken) holds a Ph.D. from Carnegie Mellon University in Operations Research and has been employed by Florida International University since Dr. Murphy s research interests are varied spanning the quantitative methods and technology arenas. Specifically, he has worked and published in the areas of machine and personnel scheduling as well as in the organizational learning literature. More recently, his focus has shifted to the value of implementing large-scale packaged software in global organizations and the implications of this trend for education. Dr. Murphy has published in Operations Research, Naval Research Logistics and Communications of the ACM journals. He is an active member of The Institute for Operations Research and Management Science (INFORMS) and the Decision Sciences Institute (DSI).

7 /02 $17.00 (c) 2002 IEEE 7 Appendix A: Proposed OLSs Values and Categories Value Category 1. Technical Support 2. System Reliability 3. System Flexibility and Features Values (Literature) Phone access to support (Keeney 1999; Moore & Kearsley 1996 p.169) Support when needed (Moore & Kearsley 1996 p.170) Opportunity for technical support (Keeney 1999; Moore & Kearsley 1996 p.76,163) System and course availability (Webster & Hackley 1997) System errors (Webster & Hackley 1997) System security (Keeney 1999) Time flexibility of learning (Moore & Kearsley 1996 p.134-6,168) Ability to submit assignments from anywhere (Carswell et. al. 1999) 4. Network Reliability Worldwide and remote system and course(s) access (Webster & Hackley 1997) Network availability and congestion (Webster & Hackley 1997) 5. Course Content Syllabus, course objectives, assessment description, course schedule (Moore & Kearsley 1996 p ) Lectures via text and/or audios and/or videos and/or graphics (Moore & Kearsley 1996 p ) Satisfaction with course/lessons (Moore & 6. Course Organization and Quality 7. Interaction and Feedback Kearsley 1996 p.162) Ease-of-use (course content) (Carswell et. al. 1999, Keeney 1999) Speed of information gathering (Keeney 1999) Organized course content (Keeney 1999; Marks 2000) Content quality (Alavi, Yoo, & Vogel 1997; Keeney 1999; Webster & Hackley 1997) Self test prior to graded test (Moore & Kearsley 1996 p.108) Amount and type of student and professor interactions (Alavi 1994; Alavi, Yoo, & Vogel 1997; Carswell et. al. 1999, Moore & Kearsley 1996 p.76, ,163) Minimize the feel of isolation (Keeney 1999; Moore & Kearsley 1996 p.162) Attitude toward instructor and classmates (Marks 2000; Moore & Kearsley 1996 p.162) Additional Values (Questionnaire) access to support System tools (chat, bulletin board or discussion forums, file drop box etc.) Required to install additional applications Portal access to all courses Ability to take quizzes remotely Ability to review recorded classes Connectivity ISP access problems Quantity of information Homogenous interface

8 /02 $17.00 (c) 2002 IEEE 8 8. Learner Control Learner's content pace control (Moore & Kearsley 1996 p.76,107,163) Time limited tasks (assignments due, quizzes taking, discussions, etc.) (Moore & Kearsley 1996 p.107,163) 9. Course Course difficulty (Marks 2000; Moore & Difficulty/ Kearsley 1996 p.162-3) Workload Course workload compared to other courses (Marks 2000) 10. Motivation/Attit Motivation to learn (Marks, 2000) ude Interest in subject matter (Moore & Kearsley 1996 p.162) Intent to complete the course (Moore & Kearsley 1996 p.162) Comfort with distance learning and technology (Carswell et. al, 1999, Webster & Hackley 1997) 11. Cost Course cost (Alavi 1994; Keeney 1999; Moore & Kearsley 1996 p.74) ISP and Internet access costs (Keeney 1999) Travel cost/time (for courses with required live sessions) (Keeney 1999) 12. Lifestyle Employer support and ability to work while learning (Moore & Kearsley 1996 p.162) Allow attendance of family responsibilities (Moore & Kearsley 1996 p.162) Family support (Moore & Kearsley 1996 p.162) Amount of "off-record" interaction during synchronous session Learner distraction during realtime session Allow business and other travel during study

9 /02 $17.00 (c) 2002 IEEE 9 Appendix B: A samples of the open-ended questionnaire Students experiencing Mixed (synchronous and asynchronous) Online Learning System: Please try to identify five main benefits that you find in the Centra system: Please try to identify five main benefits that you find in the WebCT system: Please try to identify five main drawbacks (or costs) that you find in the Centra system: Please try to identify five main drawbacks (or costs) that you find in the WebCT system: Students experiencing Asynchronous Online Learning System: Please try to identify five main benefits that you find in the WebCT system: Please try to identify five main drawbacks (or costs) that you find in the WebCT system: