Specific tool for evaluation of elearning

Specific tool for evaluation of elearning Jana Kapounova Department of Information and Communication Technology University of Ostrava Ostrava, Czech Republic Jana.Kapounova@osu.cz Katerina Kostolanyova Department of Information and Communication Technology University of Ostrava Ostrava, Czech Republic Katerina.Kostolanyova@osu.cz Abstract In the paper we deal with a problem of Evaluation of elearning from the aspect of adaptability. We consider elearning as an educational project. We have determined phases both of elearning project and phases of evaluation and we designed criteria for their evaluation. In case of adaptive elearning we need to extend the items. The study support evaluation form consists of four criteria groups: fundamental characteristic of study support, basic properties of the study support structure, feedback properties and special properties of layers. Keywords phases of elearning project; evaluation of elearning; adaptive elearning environment; evaluation form; aspects of adaptability I. INTRODUCTION Instruction has been supported by computers and the Internet since the last century. Information and communication Technologies (ICT) were initially used for communication between learners and instructors; later on study materials were prepared and delivered to students and finally, the whole study process was controlled and managed by Learning Management Systems (LMS) [9]. The conception of elearning has been introduced and it plays an important role both in distance education and in faceto-face studies. From many definitions of elearning we have accepted the following: elearning is an educational environment which uses information and communication technologies to achieve the educational goal: it includes creation of educational objects, distribution of study content, implementation, communication between participants of the educational process and management of studies. The greatest emphasis is placed on the term learning while the e- is the electronic-related prefix, notifying of using ICT. At our Department of ICT we deal with the issue of Evaluation of elearning system approach. There are many approaches on how to evaluate elearning in theoretical studies and examples of good practice. So what is our approach? [1]. We consider elearning as an educational project. In that sense elearning project is a such project within which an elearning subject, or subjects forming one complex (e.g. a study programme), or just a part of a subject is planned, developed, implemented and introduced in the form of instruction. An elearning subject includes relevant content, study system, Learning Management System (LMS). An elearning project represents creation of a set of elearning subjects or elearning components (most often) or construction of LMS functions, e.g. adaptive instruction option. Note: By the term subject we mean a school subject; we prefer the term subject to the word course. II. ELEARNING PROJECT AND ITS PHASES Methodological Approaches The elearning project consists of certain phases, proceeding one after another and interrelating. In bibliography we can find many approaches to determine system life cycle phases, e.g.: Statement of work (SOW) for pre-design phase Instructional System Design (ISD) System development life cycle (SDLC) Project Management Body of Knowledge (PMBOK) Instructional System Design For our modelling we apply the scheme of Instructional System Design (ISD). It is also sometimes known as an educational parallel to the software development process. It breaks down into five phases, whose acronym ADDIE is used as an equivalent to ISD. The education design process must contain all five phases, so that the result would be a product that teaches [6]. Analysis identify needs and constraints. Design define learning activities, assessment and media. Development produce, perform formative evaluation, and revise. Implementation deliver the instruction. Evaluation evaluate results. When considering our elearning project evaluation we established phases of an elearning project which enabled us to This work was supported by the Czech Science Foundation under the contract No. 406090242. Page 423

specify phases of evaluation, within which we have specified activities necessary to be carried out when evaluating and people responsible for their fulfilment, including inputs and outputs of activities. We have determined phases of elearning project, evaluation and activities, and now we try to find out input and outputs information for each of the phases. III. EVALUATION OF ELEARNING PROJECT Usually, a starting point is an idea or stimulus to implement, update or extend elearning project. At the beginning we arranged phases of elearning project and relevant information in the table. The content of individual data, their arrangement and the determination of items and criteria for evaluation were commented on by specialists who have been dealing with elearning education, development of study supports and design and management of complex projects for a long time. TABLE I. phases of el project EVALUATION OF ELEARNING PROJECT phases of evaluation activities responsible person Input information is required for each of the activities in order to provide output information. Phases of elearning project Assignment (an idea to implement a project) Development of the initial project/request Analysis of: human, financial and technical resources including time frame description of subjects: content, didactics, readiness of an institution Design of: input requirements for the persons responsible educational and teaching objectives concept of content, specification of study matter structure of education and of teaching units teaching materials staff involved in project development time schedule of project development plan of operation and maintenance method of conclusion / termination of teaching Development: assignment of roles (tasks) for persons responsible formulation and specification of assignment scheduling of dates development of appearance of teaching environment development of teaching objects set up of behaviour (subjects) control of individual parts completion of educational supports control of the unit (subjects) verification of functionality before pilot operation ensuring operation conditions administration (of subjects) tutoring evaluation of student activities routine operation update Evaluation (for each subject or course) project evaluation evaluation by users feed-back and revision Evaluation of the project as a whole financial balance analysis of collected data evaluation of by-products of the project Activity outputs Results of the evaluation should show among others, if the people preparing the project have considered all of issues, how thoroughly and in what depth. Evaluation activity outputs can be in the following form: questionnaire, review, checklist, evaluation report, survey record, evaluation statement, or data analysis conclusions. IV. THE EVALUATION OF ELEARNING STUDY SUPPORTS In the project entitled elearning Elements for Study Support in Special and Technical Subjects, which was developed at VŠB-Technical University of Ostrava between 2006 and 2008, elearning study supports were created for fifty-seven subjects presented at three technical faculties. The stated supports were evaluated both professionally and in terms of pedagogy [8]. The evaluation form was developed by the project developers due to the non-existence of a standard available for assessment of pedagogic quality of elearning study supports [7]. Evaluation form in terms of elearning education methodology The evaluation form is a structured questionnaire consisting of twenty-six questions split into four areas: I. Basic characteristics of text 1. guide through the course at the introduction and in individual chapters; 2. break-down of the course into chapters; 3. statement of objectives in student performance for the whole course and for each chapter; 4. well-arranged overview of the course; 5. comprehensibility of the communicated content for students; 6. overall design of the course influenced by the author (not LMS): pictures, schemes, graphs; 7. use of multimedia (interpretation, video, audio); 8. quality and adequacy of additional sources, reference of basic study literature; 9. imaginativeness. Page 424

II. Encouragement of students 10. applying instruction to think about directly in the text and mobilizing questions; 11. applying content of study in practical examples, case studies; 12. support of communication tools (discussion, chat, email); 13. work groups of students and development of common partial or long-term projects. III. Planning and organizing of study activities 14. syllabus; 15. requirements for students (calendar); 16. use of basic elements according to guide for individual chapters. IV. Feedback and evaluation 17. key words (new terms at the beginning of chapter); 18. summary at the end of the chapter; 19. repeating and mobilizing questions in the text; 20. solved tasks in the text; 21. solution tasks short tasks across the text; 22. solution tasks more demanding tasks with summary of chapter (correspondence tasks to be sent to lecturer); 23. tests; 24. auto-tests; 25. key to tasks in the text; 26. glossary. Each question was answered by opponents verbally and evaluated by a point of 0 to 10; the higher point, the higher quality. A series of analyses was conducted over obtained data: descriptive statistics; principal component analysis; analysis of associations; cluster analysis. The analysis meets the results; they are described in [3]. evaluation of pedagogical quality of study supports divided their authors into qualitative groups; evaluation of pedagogical questionnaire of study supports brought proposals for their improvement. V. ADAPTIVE ELEARNING STUDY MATERIALS elearning environment and high-quality LMS enable to individualize the teaching process according to study characteristics of learners. The basic motivation for design and implementation of adaptive elearning study materials is to increase effectiveness and delivery of information. The principle of possible application of an adaptive study material consists in respecting of diversity of individual users. The task of adaptive educational systems is their adaptation to various types of characteristics and levels of student knowledge. Adaptive educational systems can be viewed from two aspects: 1. educational what are student s needs, ambitions and other sensations to which the system should adapt. The point is adaptation of content of the submitted subject matter. 2. technical what methods and techniques are available for adaptation of the system to student s needs. The point is adaptation of user (environment) to student s needs. The content of the subject matter should adapt to student s capabilities; preferred learning style; student s interest; student s motivation; student s performance during the course. Generally applies that students are differently motivated to learning, have different family background, various habits on when and how to learn; have different initial knowledge of the subject studied; have different level of talent for various fields of study; have a different learning style; have a different type of memory and different level of its training; need various depth of knowledge, understanding, use and applying of acquired knowledge; are differently concentrated or tired, etc. Characteristics of students which can be considered during the development of adaptive elearning study materials: 1. sensual perception: visual type, auditive type, kinaesthetic type, verbal type; 2. social aspects: they prefer to learn with mates, assisted by teacher or alone; 3. affective characteristics motivation is the most important in that category; 4. tactic of learning, technology, methodicalness; 5. method of learning: theoretical, deriving, experimenting; 6. learning progress: in detail, holistic; 7. learning conception: in depth, strategic, superficial; 8. auto-regulation of learning: students require precise instructions or they control learning independently. The suitable adaptive text-book should consider characteristics of student featuring their learning style. Moreover, the system operating with such study adaptive materials should be able to respond on provided information about the style and progress in student s learning and the submitted study material should be designed so that it complied with preferred requirements of student. It is quite unrealistic to embrace all above mentioned characteristics and their combinations at the development of each study support. Therefore, we have developed the methodology of adaptive textbooks design, which should make the preparation of study supports more effective [4]. Structure of Study Materials Author of the study material for adaptive use divide the content of study to frameworks containing integrated information. Page 425

Subject is the highest integral component of educational support; the subject is then further broken down into lessons. Lesson is the educational unit corresponding to the classroom hour. The lessons are further broken down into frameworks. Framework is the elementary part of the lesson containing the unit educational information. Elementary information of the framework corresponds for example with a newly introduced term, it can contain frameworks containing motivation for its implementation, definition, explanation of applied terms, fixation of new terms by their contextualizing, their application as examples of use, verifying test questions and tasks for solution. Study support is split into frameworks. Author develops them so that various methods of subject matter interpretation could be implemented in individual variants of frameworks. Layers of the framework correspond to phases of the teaching process events of teaching, e.g. interpretation, fixation, transfer of subject matter, etc. [5]. Types of layers Explanation group of layers containing interpretation of the content of study: T Theoretical contains theory: definition, terms, rules, algorithms, etc. In terms of education, this is the most important type of layer. S Semantic explains the introduced terms, described theory, contains additional information to the theoretical layer, explains correlations, etc. F Fixation reinforces knowledge through repetition, alternative formulation and concepts. R Solved examples contains examples applying theory, examples from textbooks. P Practical contains solution of practical examples. Testing group of layers for continuous testing of acquired knowledge: O Questions theoretical questions from the explained curriculum (= from explained subject matter). U Exercises, tasks to be solved. X Practical exercises, case studies. Others group of layers containing special type of layers to assist learning: M Motivational information on the subject, lesson or framework, which would motivate a student. N Navigational didactic or organizational information, a kind of guide for lessons, recommended study methods, etc. C Goal goal of the framework. L Literature quality and adequacy of basic and additional study resources. The information about the form and depth of explanation and type of layer is recorded by means of metadata. The learning management system then selects and runs the proper sequential order of teaching for individual student. One of the basic tasks in preparing an adaptive elearning project is preparation of study materials according to the given methodology [6]. The evaluation questionnaire can be applied to find out if developed study supports meet requirements of adaptive textbooks. Evaluation form in terms of adaptive structure of study supports The characteristics which can be considered at the development of a study support for the adaptive method of education are evaluated in the questionnaire applied for the assessment of the study material level of adaptability. Each question is answered by opponents verbally and evaluated by a point of 0 to 10; the higher point, the higher the quality. I. Basic characteristics of the study support: 1. study support is suitably divided in chapters; 2. objectives of the subject as a whole are formulated at the beginning of support; 3. conclusions of chapters include summary; 4. the presented content is comprehensive for students in interpretation passages; 5. multimedia are applied (animation, video, sound ); 6. the hint of think is used directly in the text of the course and in mobilizing questions; II. Basic characteristics of the study support structure: 7. chapters are suitably divided in particular frameworks in terms of adequate size and integrity of information; 8. in terms of content, study frameworks are correctly divided in layers; 9. self-support of layers is ensured, their sequence in the framework can be interchanged; 10. applied multi-media correspond to individual sensual variants; 11. the depth of the interpretation of variants of frameworks corresponds to the methodology of the development of study supports; 12. variants of depth 1 and 3 do not repeat the same information from depth 2; 13. it is possible to build pre-frameworks and postframeworks, from specific layers of frameworks (C, N, T, M) pre-frameworks and post-frameworks can be built; III. Characteristics of interpretation layers: 14. all types of layers T, S, F, R are sufficiently 15. applications of subject matter P are used in examples from practice; IV. Characteristics of back-up (testing) layers: 16. all types of testing layers O, U, X are sufficiently 17. groups of questions are included being split into compulsory and facultative; 18. group questions are content-equivalent; Page 426

V. Characteristics of special layers 19. all types of layers C, N, M, L are sufficiently 20. objectives are formulated in student s performance students know, what they should handle and solve at the end of the framework, lesson and subject; 21. L layers contains the list of additional information resources; 22. M layer is sufficiently represented in the study material at all depth levels. VI. SUMMARY elearning projects have been increasingly implemented in distant and full time learning. Regarding requirements of effective use of resources (human, time, financial) it is necessary to use sophisticated tools at their development. One of the groups of such tools is that for evaluation of elearning. For an elearning project we define individual phases of its development and relevant evaluation methods and tools. These can be in different forms, e.g. questionnaire, review, checklist, evaluation report, survey record, evaluation statement or data analysis conclusions. Examples of certain tools applied in practice: evaluation of the overall project plan; review of project draft management; review of proposed of subject/subjects; evaluation of educational support by lay reader; expert opinion; pedagogical opinion; others. In the paper we evaluate the elearning study material in terms of methodology of elearning education. Existing elearning environment enables more than to deliver study material to all learners in the uniform form. It is possible to prepare such trajectory which would be tailor-made to individual characteristics of learners. It is essential to prepare such study supports which satisfy requirements of adaptable education. The level of success in the development of the adaptive elearning material can be assessed by the tool for evaluation of elearning study material in light of methodology of the adaptive education. Study supports developed as such were subject to evaluation via evaluation forms. Following evaluation of data from those forms we will get a feed-back both for trainers who prepare authors for the development of adaptive supports, and for authors of those study materials. REFERENCES [1] R. C. Clark and R. E. Mayer, e-learning and the Science of Instruction: Proven Guidelines for Consumers and Designers of Multimedia Learning San Francisco : Pfeiffer, 2002. [2] J. Kapounová, Approaches to the evaluation of elearning. Proceedings of the Computer Based Learning in Science conference, 2007, pp. 208 214. [3] J. Kapounová and J. Šarmanová, The Evaluation of elearning Study Supports, Proceedings of the Computer Based Learning in Science conference, 2010, pp. 103 109. [4] K. Kostolányová, J. Kapounová, O. Takács and J. Šarmanová, Personalisation of Learning. Research, Reflections and Innovations in Integrating ICT in Education, Portugal : University of Lisabon, 2009, pp.. 234-237. [5] K. Kostolányová, J. Šarmanová and O. Takács, Classification of Learning Styles for Adaptive Education, The New Educational Review, 2011, pp. 199-212. [6] K. Kostolányová, J. Šarmanová and O. Takács, Adaptation of teaching process based on a students individual learning needs, Journal on Efficiency and Responsibility in Education and Science, 2011, pp. 3-17. [7] E. Mechlová, Tvorba e-learningových kurzů pro technické obory, Ostrava : Technical University of Ostrava, 2008. [8] J. Šarmanová, E-learningové prvky pro podporu výuky odborných a technických předmětů, Ostrava : Technical University of Ostrava, 2009. [9] H. Zlámalová, Principy distanční vzdělávací technologie a možnosti jejího využití v pedagogické praxi na technických vysokých školách, http://virtual.cvut.cz/telel/zlamalova.html, 2002, [online]. Page 427