Validation of an evaluation model for learning management systems

Transcription

1 Original article doi: /j x Validation of an evaluation model for learning management systems S.W. Kim* & M.G. Lee *Computer Education/e-Learning, Graduate School of Education, Ajou University, Suwon, South Korea Department of Education, Yonsei University, Seoul, South Korea Abstract Keywords This study aims to validate a model for evaluating learning management systems (LMS) used in e-learning fields. A survey of 163 e-learning experts, regarding 81 validation items developed through literature review, was used to ascertain the importance of the criteria. A concise list of explanatory constructs, including two principle factors, was extracted from the responses of 133 respondents by exploratory factor analysis. Factor I was labeled as instruction management, screen design, and technology and factor II, interaction and evaluation. Factors I and II consisted of four and three sub-factors, respectively. Confirmatory factor analysis, in turn, was carried out to assess the degree to which the plausibility of factor models was empirically confirmed. Overall fit measures (RMSEA, CFI, NNFI indexes) showed the model to be quite suitable for LMS evaluation. Reliability estimates using Cronbach s alpha indicated that the evaluation tool had good internal consistency. The overall results of the analyses suggest that this evaluation tool is highly instrumental for LMS. evaluation criteria, factor analysis, learning management systems, reliability, validity. Introduction With the rapid development of information and communication technologies has come the penetration of e-learning into almost all kinds of organizations, such as schools, companies and the military. E-learning environments, however, are radically different from those of face-to-face classrooms. This difference requires that educational practitioners in various fields search for effective and efficient mechanisms to prepare, operate and manage the e-learning process. The learning management system (LMS), one of the major e-learning platforms, provides such supporting services. Most of the studies related to LMSs have compared and analysed their functions and features (Barron 2000; Barron & Lyskawa 2001; Landon 2007), and have Accepted: 12 September 2007 Correspondence: Myung-Geun Lee, Department of Education, Yonsei University, 134 Sinchon-dong, Seodaemun-gu, Seoul , South Korea. mglwin@yonsei.ac.kr involved intuitive extraction of evaluation criteria. In particular, the comparison analysis method has been criticized for comparing platforms with completely different features and for having little applicability to the design and development of new LMSs (Maslowski et al. 2000). A system with a specific, unique function cannot be assumed to be better than another that does not maintain the same functionality (A Herran 2000). The so-called inventory-type evaluation framework may make evaluation impossible because the inventory of functions increases with time. That is, such research is no longer relevant, just as research about media-effect comparison lost its relevancy through time. Additionally, the method of intuitively extracting evaluation criteria has the weakness of focusing only on a small number of criteria, such as instruction, technology and institution, without establishing any compelling process for validating the criteria themselves. Obviously, then, research for developing and validating an evaluation framework for LMSs is in great need 284 Journal of Computer Assisted Learning (2008), 24,

2 Validation of an evaluation model for LMS 285 at this time. That is, a reliable and valid evaluation model for assessing and selecting LMSs needs to be developed to aid educational practitioners in various fields who are primarily engaged in e-learning. A criteria model for evaluating LMSs The criteria model for evaluating LMSs is based on the principle that e-learning platforms ultimately aim to effectively accomplish instruction (Yoo 2000; Song 2004; Uzunboylu et al. 2006). An e-learning platform needs to be optimized for supporting important instructional activities, such as instructional management, interaction, evaluation and information guidance. In other words, LMS should be a means to help teachers and learners achieve instructional goals through problem-solving teams, question and answer sessions, or online simulations, rather than be just another tool that merely provides users with the convenience of sending , distributing handouts or keeping an online grade book (Frey 2005). Analysis of related previous studies and literature offers the potential criteria model for evaluating LMS (see Fig 1). As seen in the figure, the seven aspects of the criteria that constitute the potential model are as follows: instructional management, interaction, evaluation, information guidance, screen design, technology and organizational demand. The first four aspects are directly related to instruction. Screen design, technology, and organizational demand support instructional activities specific to e-learning. Additionally, organizational demand is the basic element that should be satisfied before the other six aspects. All of the aspects are closely related to each other. Although the first four aspects constitute the core of an e-learning program, the remaining three aspects are also important. E-learning effectiveness, for example, is contingent upon network stability. Instructional management controls all aspects of e-learning administration, from class registration to completion. It includes management safety, user accessibility and easiness of course management. The subcategory management safety is related to user differentiation, access authentication and identification/ password management (Landon 2007). Also, every LMS should maintain user accessibility so that even the disabled may easily access and manage the platform. Users should be able to navigate learning materials using a variety of methods and be able to monitor the learning process through the LMS (Hansen & Frick 1997; Rosenberg 2001; Seo & Park 2004). Interaction represents communication among users. LMSs should be designed to accommodate a variety of communication types, such as learner-to-learner, Fig 1 A potential criteria model for evaluating learning management systems. LMS, learning management system.

3 286 S.W. Kim & M.G. Lee teacher-to-learner and learner-to-material, because active participation enhances user satisfaction and learning effectiveness (Rha & Kim 2000). Availability of communication tools and easiness of accessibility are also foundations of a successful LMS-user interaction (WBLS Architecture Team 1999). For example, LMS faculty should be able to communicate with persons or groups via without the use of specific software, such as Outlook Express (Tielemans & Collis 1999; Rosenberg 2001). Evaluation incorporates a set of criteria, such as the easiness of test management, a variety of test types and the reusability of items. Management of online tests should be easy. For instance, an LMS should be able to maintain online learning result records and search them with ease. Being afforded such interactive evaluation of their work may encourage learners to continue to study via an LMS. Evaluation suggests that learning materials, test types and feedback maintain some variety and be restorable and reusable. For example, multimedia such as video, audio and pictures can provide tests that emulate real-world situations (Straetmans & Eggen 1998). Information guidance includes three subcategories: easiness of furnishing information, searchability of information and accessibility of information. That is, guidance and tips regarding the platform structure must be adequate to enable users efficient navigation through a variety of information. Information guidance aids learners in finishing their learning effectively and efficiently (Ritchie & Hoffman 1997). The Screen of an LMS should be designed to have clarity of directions, consistency, unity, readability, easiness of navigation, learner control and appropriateness of multimedia. The elements of a screen, such as buttons, links and icons, should be designed to enhance intuitive user access and have a good feel and look. Also, users should be able to adapt screen designs to their tastes, and the level of multimedia must be appropriate. User control of the screen design prevents the instructional process from becoming tedious and monotonous (Hannafin & Peck 1988; Kang et al. 2000). Screen design should also improve readability and easy navigation. That is, the screen should be so designed as to enable users to predict where a specific link is connected to and to minimize the effort required to interpret learning material messages (Kim 2000). Technology incorporates interoperability, ease of installation and system stability. E-learning platforms should be interoperable with other platforms, so that learning materials created previously are reusable and all data are synchronously updated when integrating materials with other systems (Tielemans & Collis 1999; WBLS Architecture Team, 1999; Roberts 2005). For the purpose of ensuring interoperability and reusability, it is imperative that LMS be designed in accordance with e-learning standards such as Sharable Content Object Reference Model (Sanchez-Alonso & Frosch-Wilke 2005). System installation should be easy, and system stability should ensure no errors. The final area to be considered for LMS evaluation is the actual organizational demand for education and training. This has three subcategories of appropriateness to learning demand, economic validity and user support. An e-learning platform must meet the organizational demand required for its use. It is essential that the goals of education and training are of paramount importance in the development and selection of an LMS. There is also, however, the need to consider economic validity (Bersin 2005). If an e-learning platform is very effective but very expensive, it may go unselected as a result of its prohibitive costs. Also, the technical support offered by e-learning platform suppliers can be a critical factor affecting platform selection. Method To develop a potential criteria model for evaluating LMSs and to validate the evaluation model, an evaluation criteria framework was made based on a literature review. In total, 81 items were developed through the framework, and a survey was carried out among 163 e-learning experts to ascertain the importance of each item in the evaluation of LMSs. Responses from 133 respondents were used for factor and reliability analyses that confirmed the fitness of the criteria framework. The 133 evaluators were experts with more than 1 year of experience with e-learning at universities, corporate training centres and e-learning system companies. The evaluation group consisted of educational technology faculty (3%), e-learning education managers (17.3%), e-learning planners (20.3%), e-learning consultants (18.8%), programmers and web designers (31.6%), and others (9%). About 89.4% of the respondents had relatively substantial experience in the

4 Validation of an evaluation model for LMS 287 Table 1. Mean and standard deviation (SD) for the variables. Item no. Mean SD Item no. Mean SD Organizational demand A A A A A A A Instructional management B B B B B B B B B B Interaction C C C C C C C C C C C C C C C C C C Evaluation D D D D D D D D D D D D D Information guidance E E E E E E E E Screen design F F F F F F F F! F F F F Technology G G G G G G G G G G G G G e-learning field ranging from 1 to 3 years. Data were collected by visitation, mail and . A total of 149 evaluation items were developed based on the potential criteria framework, and the items were examined and revised by seven graduate students majoring in educational technology and education measurement/evaluation at a private university. Finally, an educational technology faculty member reviewed the face validity of the evaluation items three times. The final criteria framework comprised of 81 items: instructional management (10 items), interaction (18 items), evaluation (13 items), information guidance (8 items), screen design (12 items), technology (13 items), and organizational demand (7 items). A 5-point Likert-type

5 288 S.W. Kim & M.G. Lee Fig 2 Model I. scale was used. By considering means (M 4.5, M 2.0), standard deviations (sd 7.0), correlation between item-total score (r 0.30), and correlation between items (r 0.60), the number of items was reduced to 69. To derive a concise list of explanatory constructs from the 133 responses 1 collected, exploratory factor analysis was performed. In the initial factoring step, principal component analysis was employed to extract initial factors and the varimax of orthogonal rotation in the rotation step. And confirmatory factor analysis was conducted to assess the degree to which factor model plausibility was empirically confirmed. Reliability was assessed to confirm the internal consistency of the selected items. Results The scores from the survey regarding the importance of items for evaluating LMSs ranged from 3.05 to 4.34, indicating that most of the e-learning experts agreed that all of the preparatory items were important (Table 1). To examine the validity of the evaluation criteria framework, a factor analysis of survey responses was conducted. Exploratory factor analysis was used because the links between the observed and latent variables were unclear. The minimum factor-loading criterion was set to 0.30; items loading below 0.30 were not used in the interpretation. To determine the number of factors to be extracted, factor interpretability was considered the most important, although scree test and Kaiser criterion results were also considered. Initially, a 2-factor model (model I in Fig 2) was suggested, as the first two factors accounted for 29.34% of the total variance (Table 2). The items were roughly clustered into two categories: instruction management/technology/ screen design (factor I), and interaction and evaluation (factor II). As a result of the second exploratory factor analysis, a 4-factor model (model II in Fig 3) and a 3-factor model (model III in Fig 4) were also suggested, as the first four and three factors accounted for 45% and 44% of the total variance, respectively. It was interpreted that factor I and factor II, respectively, comprised four sub-factors (suitability of design in screen and system, ease of course procedure, interoperability of the system and suitability of academy administration, easiness of instructional management and appropriateness of multimedia use) and three subfactors (flexibility of interaction and test and user control, test and communication variety, and user accessibility). Confirmatory factor analysis was conducted, using the suggested models to assess the degree to which factor model plausibility was empirically confirmed. The results, which are listed in Table 3, confirm the plausibility of the suggested models (N = 133; RMSEA ; CFI ; NNFI ). Table 2. Proportion of variance per eigenvalue. No. of factor Eigenvalue Difference of eigenvalue Variance explained Cumulative proportion of total variance

6 Validation of an evaluation model for LMS 289 Fig 3 Model II. Table 3. Fit statistics for models. Index RMSEA CFI NNFI Model I Model II Model III Model I Model II Model III Model I Model II Model III Value Finally, internal consistency reliability was analysed to test the correlation among items. The Cronbach s alpha statistics for factors I and II were 0.94 and 0.98, and those of the sub-factors ranged from 0.78 to 0.86, as shown in Table 4. These results confirmed the adequacy of the internal consistency of the LMS evaluation model. Based on the aforementioned results, the final revised criteria model, shown in Fig 5, maintains appropriate construct validity and reliability. This model consists of two factors (Instruction management/screen design/technology and Interaction/Evaluation). Factor I is most highly saturated with 41 variables. All of these items deal with suitability of design in screen and system, easiness of course procedure, interoperability of system and suitability of academy administration, and easiness of instruction management and appropriateness of multimedia use. Factor II is most highly saturated with 23 variables. All of these items measure the flexibility of interaction and test and learner control, variety of communication and test types, and user accessibility. Discussion and conclusion Fig 4 Model III. This study aims to generate a useful model for evaluating LMSs for use in various e-learning fields. To this

7 290 S.W. Kim & M.G. Lee Table 4. Internal consistency reliability analysis (n = 133). Factor Sub-factor Number of items Mean SD Cronbach s alpha Instruction management Screen design Technology Interaction Evaluation Suitability of screen and system design Ease of course procedure Interoperability of system and suitability of academy administration Ease of instruction management and appropriateness of multimedia use Flexibility of interaction and test and learner control Variety of communication and test types User accessibility Fig 5 The final revised criteria model for LMS evaluation. LMS, learning management system. end, factor and reliability analyses were conducted, and the implications of their results may be interpreted as follows: First, the statistically confirmed evaluation model did not include any other items related to organizational demand except the appropriateness of platforms to education and training (A4). This result is contrary to McGreal s (1998) view of integrated LMS evaluation, which emphasizes organizational environment characteristics. However, the organizational aspect should not be interpreted as unimportant in LMS evaluation, but rather as more intrinsic to the evaluation than any other factors. On the other hand, items related to information guidance were scattered among other factors of the evaluation model. Contrary to expected results, evaluators did not emphasize this category relative to others. Second, the LMS evaluation tool generated herein is based on a different format than any existing inventorytype evaluation framework. Until now, researchers, such as Lazenby (1999) and Learning Technologies Resource Centre (2006), have evaluated LMSs dichotomously, using yes or no responses to evaluate the effectiveness of LMS functions. This approach may

8 Validation of an evaluation model for LMS 291 only serve to lengthen function inventories because of the rapid emergence of information technologies. Also, although dichotomous methods may aid in the selection of existing LMSs, they cannot be applied to the design and development of new platforms (Maslowski et al. 2000). Third, the approach taken in this study overlaps to a certain degree with Collis s (1999) method, which tried to assess the inventory-type evaluation models. Collis objected to the checklist-type, function-centred method, as it cannot confirm the receipt of a complete evaluation by the decision-maker of a firm considering an LMS. By establishing an evaluation criteria framework, this study included items that concur with Collis emphasis on usability. In conclusion, seven factors for evaluating LMSs (suitability of design in screen and system, easiness of course procedure, interoperability of system and suitability of academy administration, easiness of instruction management and appropriateness of multimedia use, flexibility of interaction and test and learner control, variety of communication and test types, and user accessibility) can greatly benefit those engaged in e-learning as they seek theoretical guidance for their practices and transactions in real-world situations. The factors were thus developed to assess the effectiveness and efficiency of LMSs in support of e-learning, rather than focusing on the functions and features of LMSs. Consequently, despite the limitations of the present study in sample size and the relatively narrow room for generalization imposed by the limits of convenience sampling, the evaluation model herein proposed can play an instrumental role in assessing LMS effectiveness in situations where LMS features are not the primary concern. Future research needs to focus on the development of more elaborate models specific to various types of organizations that reflect the characteristics of education and training practices taking place in different institutional settings, including primary and secondary schools, universities and companies. Note 1 One of the most conservative approaches, regarding sample size, has been proposed by Boomsma (1982), who has recommended that a sample size of at least 200 be collected before any attempt at factor analysis. However, a more liberal estimation of appropriate sample size includes the formula a N-n-1 greater than or equal to 50, where N is the sample size and n is the number of variables (Ledakis 1999). In this study, N is 133 and n is 69 (items). Consequently, the sample size of the present study can be justified from the viewpoint of a liberal approach.

9 292 S.W. Kim & M.G. Lee Appendix Evaluation criteria for learning management systems Factor Sub-factor Item Instruction management Screen design Technology Suitability of design in screen and system A1 C1 F1 F2 F3 F4 G6 G8 G10 G11 G12 Is the platform appropriate for learning and training goals? Does the platform provide an environment that promotes learner-teacher interaction? Are the directions and terms clear? Is the interface (eg. buttons, icons) designed to accommodate easy access by anyone? Is the screen structure uniform? Does the screen have a good look and feel? Is the installation of the LMS easy? Does the platform make any errors? Is the process of exchanging data and information with other systems easy? Is the number of users appropriate that can simultaneously connect to the platform? Can LMS functions be easily updated? Ease of course procedure B6 B7 B8 B9 E5 E6 E7 F6 F7 F8 F10 F11 F12 Can learners find out specific learning and course contents by various search methods? Can instructors upload learning materials easily? Can learners print out learning materials easily? Can a learner s progress (eg. learning procedure, learning time) be easily tracked? Is an information guide provided for the entire structure of the platform? Are there various search methods? Can learners easily approach the information that they want to get? Does screen structure and content reflect the learner s level? Is there good character readability? Is the use of scroll bar minimized? Are link directions predictable? Can learners come back to the platform after using other software such as a word processor? Is the clicking minimized to arrive at a desired screen? Interoperability of system & suitability of academy administration B3 G1 G2 G3 G4 G5 G13 Is class management easy, such as student registration, training, and payment? Is the platform applicable in the existing H/W, S/W and networks? Can functions currently absent be added? Are instructional materials produced by other companies reusable? Can the platform be used in any operating systems such as Windows NT and UNIX? Is the platform operable in recent versions of internet browsers? Does the platform conform to international standards and specifications?

10 Validation of an evaluation model for LMS 293 Appendix (continued) Ease of instruction management & appropriateness of multimedia use B1 B4 B10 C4 D1 D2 D4 D6 D11 F9 Can a user easily change their indentification and password? Can the instructors and operators easily approach class management functions? Can the course be opened and altered comfortably? Does the LMS facilitate interaction between the learner and learning contents? Can the LMS store and maintain test results? Are test results easily provided? Is scoring done automatically? Can learners search and confirm the test results via the internet? Can items produced be stored and reused? Is multimedia used properly? Interaction Evaluation Flexibility of interaction &test and learner control C5 C7 C9 C10 C11 C15 C18 D3 D10 D13 E3 E4 F5 Can users communicate synchronously? Can users send and receive their without any software? Does the LMS provide newsgroup functions for sharing news and holding discussions? Does the LMS provide users with information to facilitate face-to-face interaction? Can users search for the messages which are sent and received? Does the LMS let users know that has arrived? Can learners send their anywhere when using the platform? Does the LMS provide test results in various ways including ? Does the LMS store reports of learners activities? Do learners have the opportunity to repeatedly solve the same items? Can users search for and find the online help that they want? Can learners move from the platform to related websites to search for other needed information? Can users change the screen design according to their tastes Variety of communication and test types C2 C13 C14 C16 D7 D8 D9 Does the platform promote interaction among learners? Does the platform promote cooperative learning? Does the platform provide space where learners can easily ask questions regarding learning materials? Can instructors simultaneously communicate with several learners easily? Does the platform provide a variety of item types? Can the instructor give a descriptive evaluation when assessing learning tasks? Does the platform provide for diagnostic evaluation? User accessibility B5 C17 E8 Can disabled persons approach class management functions anytime and anywhere? Can handicapped persons easily use communication functions? Can handicapped persons easily approach the information that they want?

11 294 S.W. Kim & M.G. Lee References A Herran A. (2000). The Search for an Online System for Teaching & Learning at JCU. Available at: tld.jcu.edu.au/general/survey_re/recs/recs3.html (last accessed 6 July 2005). Barron A.E. & Lyskawa C. (2001) Software tools for online course management and delivery. In Web-Based Instruction (ed. B.H. Khan), pp Educational Technology Publications, Englewood Cliffs, NJ. Barron T. (2000). The LMS Guess. Learning Circuits. Available at: barron.html (last accessed 7 November 2005). Bersin J. (2005) Evaluating LMSs? Buyer beware. Training 42, Boomsma A. (1982) The robustness of LISREL against small sample sizes in factor analysis models. In Systems under Indirect Observation: Causality, Structure, Prediction (eds G.J. Karl & W. Herman), pp North-Holland, Amsterdam. Collis B. (1999) A Perspective on the Usability of Evaluation Frameworks for WWW-Based Course-Support System. Available at: bruikb.htm (last accessed 30 October 2005). Frey B.A. (2005) Enhancing Face-to-face Courses with A Course Management System. ERIC Document Reproduction Service No. ED Hannafin M. & Peck K. (1988) The Design, Development and Evaluation of Instructional Software. Macmillan Publishing, New York. Hansen L. & Frick T.W. (1997) Evaluation guidelines for web-based course authoring systems. In Web-Based Instruction (ed. B.H. Khan), pp Educational Technology Publications, Englewood Cliffs, NJ. Kang S.G., Lee Y.H. & Kim S.S. (2000) Design of systematic concept learning model using computer education search engine. SchoolNet pp Kim M.R. (2000) Strategies on screen design of learnercentered web-based instructional systems. Journal of Educational Technology 16, Landon B. (2007) EduTools Course Management System Comparisons. Available at: info/static.jsp?pj=4&page=home (last accessed 3 May 2007). Lazenby K. (1999) Using WebCT at the University of Pretoria, South Africa. International Journal of Telecommunications in Education 5, Learning Technologies Resource Centre (2006) Request for Information (RFI # TG11-06/118). Available at: (last accessed 25 June 2007). Ledakis G. (1999) Factor analytic models of the Mattis Dementia Rating Scale in Dementia of Alzheimer s type and vascular dementia patients. Unpublished doctoral dissertation, Drexel University, Philadelphia, PA. Maslowski R., Visscher A.T., Collis B. & Bloemen P. (2000) The formative evaluation of a web-based coursemanagement system within a university setting. Educational Technology 40, McGreal B.R. (1998) Integrated distributed learning environments (IDLEs) on the Internet: a survey. Educational Technology Review Spring/Summer, Rha I. & Kim M.R. (2000) Developing a conceptual model of web-based instructional platform for corporate education. Journal of Educational Technology 16, Ritchie D.C. & Hoffman B. (1997) Incorporating instructional design principles with the world wide web. In Web-Based Instruction (ed. B.H. Khan), pp Educational Technology Publications, Englewood Cliffs, NJ. Roberts E. (2005) ADL/SCORM gets interesting: instructional design as a new focus. Educational Technology 45, Rosenberg M.J. (2001) E-learning: Strategies for Delivering Knowledge in the Digital Age. McGraw-Hill, New York. Sanchez-Alonso S. & Frosch-Wilke D. (2005) An ontological representation of learning objects and learning designs as codified knowledge. The Learning Organization 12, Seo C. & Park S. (2004) E-learning system design and implementation for small sized cyber lecturing. Information Systems Review 6, Song S.H. (2004) The functions and future directions of e-learning platforms. The Journal of Educational Information and Media 10, Straetmans G. & Eggen T. (1998) Computerized testing: what it is and how it works. Educational Technology 38, Tielemans G. & Collis B. (1999) Strategic requirements for a system to generate and support web-based environments in a faculty. In Proceedings of ED-MEDIA 1999 (eds B. Collis & R. Oliver), pp AACE, Chalottesville, VA. Uzunboylu H., Ozdamli F. & Ozcinar Z. (2006) An Evaluation of Open Source Learning Management Systems According to Learners Tools. ERIC Document Reproduction Service No. ED Web Based Learning Systems Architecture Team (1999) Web Based Learning Systems at University of Wisconsin Madison. Available at: architecture/pdfs/wbls_guideline.html (last accessed 13 July 2005). Yoo P.J. (2000) Evaluation criteria of cyber education. School- Net pp