Development of a Model to Dynamically Elaborate and Innovate Sustainable e-courses

Project no. 2005-3868 / 001-001 ELE-ELEARN, Education and training Programme WP3: Final report Development of a Model to Dynamically Elaborate and Innovate Sustainable e-courses Prepared by: Tampere University of Technology Ragnar Granit Institute Kari Lindroos Juha Nousiainen Mediamaisteri Group Mats Rajalakso 2007

REPORT TITLE Report Development of a model to dynamically elaborate and innovate sustainable e-courses WORK PACKAGE 3. Development of a model to dynamically elaborate and innovate sustainable e-course RESPONSIBLE(S) Ragnar Granit Institute (RGI), Tampere University of Technology Mediamaisteri Group ABSTRACT For ensuring the dynamic and sustainable entity the process concerning a particular courses are divided into two cycles; external and internal cycles. External cycle will ensure the dynamic processing of the courses in the EVICAB and the internal cycle will ensure the sustainability and quality of the given courses. This report will give a guideline of the processes taking place in EVICAB and detailed information on the realization and execution of the different sections. The report is divided into two parts: (a) Modelling of dynamic evaluation, (b) Implementation of the model KEYWORDS Dynamic and Sustainable e-courses, Implementation, Evaluation AUTHOR NAME(S) Kari Lindroos, Juha Nousiainen, Mats Rajalakso DATE CLASSIFICATION Autumn, 2007 Public EC EDUCATION AND TRAINING Project: European Virtual Campus for Biomedical Engineering (EVICAB) Project no. 2005-3868 / 001-001 ELE-ELEARN, Education and training Programme 2

Table of Content MODELLING OF DYNAMIC EVALUATION... 5 1. Introduction...5 2. External Cycle... 7 Phase 1. Proposal... 7 Steering Group... 8 Phase 2. Implementation... 8 Phase 3. EVICAB Course... 9 3. Internal Cycle... 9 Phase 4. Evaluation... 10 Phase 5. Data Processing... 11 Phase 6. Analysis... 12 Phase 7. Conclusions... 13 IMPLEMENTATION OF THE MODEL... 15 1. Implementation of the External Cycle... 16 1.1 Course Description database... 17 2. Implementation of the Internal Cycle... 19 3. Evaluation Forms... 20 3.1 General Discussion... 20 3.2 Evaluation Forms for Students... 21 3.3 Evaluation Forms for Teachers and Assistants... 22 3.4 Evaluation Forms for Alumni... 23 4. Evaluation Center... 24 4.1 Evaluation Data... 24 4.2 Parameters... 25 5. Evaluation in Practice... 27 6. Responsible partners... 28 6.1 System Administrator... 28 6.2 Steering Group... 28 6.3 Teachers and assistants... 28 7. Results... 30 7.1 Management of the evaluation data... 30 7.2 Processing of the evaluation data... 30 7.3 Conclusions from the data... 30 3

Modeling of Dynamic Evaluation for E-courses in EVICAB 2006 Kari Lindroos, Researcher Juha Nousiainen, Assistant Professor Ragnar Granit Institute Tampere University of Technology 4

MODELLING OF DYNAMIC EVALUATION 1. Introduction EVICAB European virtual campus for biomedical engineering is a project funded by European Commission under the program Education and Training. In the project a virtual learning environment for biomedical engineering will be established. The common virtual curriculum and e-learning solutions will integrate the synergy of European knowhow to improve the teaching, learning and research activities and make the use of teaching resources more effective regionally and internationally. (www.evicab.eu) European virtual campus for biomedical engineering, EVICAB will be dynamic and sustainable elearning portal. The whole elearning entity will be dynamic in a sense that all its content will be dynamically developed and evaluated. The system will be systematically and continuously modified and upgraded. For ensuring the dynamic and sustainable entity the processes concerning a particular courses are divided into two cycles; external and internal cycles (Figure 1). External cycle will ensure the dynamic processing of the courses in the EVICAB and the internal cycle will ensure the sustainability and quality of the given courses. This report will give a guideline of the processes taking place in EVICAB and detailed information on the realization and execution of the different sections. 5

DYNAMICAL EVICAB New course proposal from outside the EVICAB External Cycle Approval Form in EVICAB Moodle Coordinator Approval Steering Group Implementation and execution of the course Evaluation Implementation to EVICAB Need for new courses EVICAB COURSE Course 2007-2008 Course 2008-2009 Course 2009-2010 -Re-election -Development -Suspension Evaluation Student Peer Self External Pedagogi cal Internal Cycle Coordinator Steering Group Analysis Dynamic database Criteria Statistics Parameters Figure 1. Dynamical EVICAB 6

2. External Cycle Before a course may be included to EVICAB several phases need to be passed. One of the objectives of the virtual campus is to give high quality education. In order to ensure the quality and sustainability the following steps are followed in the approving procedure. Phase 1. Proposal The procedure on an e-course begins with a proposal sent to the coordinator of EVICAB. The process on individual course offered by a university may be launched by filling a form in EVICAB platform (follow the links on www.evicab.eu). The idea of External cycle is to evaluate the e-courses the university is providing. The proposal will include all the specifics of the offered course, containing a summary on the course, objectives, level, execution, current status, platform used, teaching tools etc. Coordinator will compare the specifics with given criteria (criteria stated later in cooperation with other WPs) and make the first statement on the course. If the course is rejected the providing university is informed and the reasons specified. In case that the course meets all the criteria the information will be sent to Steering Group (persons defined later). Steering group is the institution that makes the final decision. The courses approved may proceed to the next phase, the implementation phase. Courses may be approved or disapproved in the first phase as follows, Approved Approved with conditions Disapproved 7

EXTERNAL CYCLE New course proposal from outside the EVICAB Approval Form in EVICAB Moodle Phase 1. Coordinator Approval Steering Group Phase 2 Implementation and execution of the course Evaluation Phase 3 Implementation to EVICAB EVICAB COURSE Figure 2. External cycle Steering Group EVICAB Steering Group is an institution that makes the final decision on actions made in the EVICAB. Steering group will evaluate the courses in different stages of the life-cycle in the EVICAB. Courses will be evaluated before they are implemented to the system but also during the execution of the course. Steering group will consist of experts on different fields of education; i.e. Pedagogical and technical. Phase 2. Implementation Approved courses and courses approved with conditions will receive instructions for implementing and making final adjustments. In this phase the responsible institutes are Ragnar Granit institute and Mediamaisteri Group. 8

After the implementation of the course to EVICAB the course, its functions and its contents will be first time evaluated by the Steering Group. If Steering Group approves, the course can be opened for the students. Phase 3. EVICAB Course Course becomes a part of EVICAB and may use the status of EVICAB COURSE. This is the final phase for external cycle. 3. Internal Cycle The last phase of external cycle in EVICAB can be considered as a starting point of internal cycle. All courses within EVICAB are under observation. Continuous evaluation will be made for ensuring the high quality and up-to-date information on the courses. INTERNAL CYCLE Need for new courses EVICAB COURSE Course 2007-2008 Course 2008-2009 Course 2009-2010 Phase 7 -Re-election -Development -Suspension Coordinator Steering Group Phase 4 Phase 6 Analysis Dynamic database Criteria Statistics Parameters Evaluation Student Peer Self External Pedagogical Phase 5 Figure 3. Internal cycle 9

Phase 4. Evaluation EVICAB courses will be systematically evaluated. Several different aspects are taken into a count when EVICAB courses are evaluated. The most important evaluation is done by students. EVICAB platform will provide systematic feedback system which students will use. The courses may be evaluated both during and after the course. The procedure will be done with the tools provided by the platform. The feedback data will be stored on the platform database. Teachers and assistants of all courses are asked to fill in self-evaluation forms after every course. This is an action for sustainability and development of the courses. New ideas and way of improvement confronted during the course are hence written down and stored. Courses are also evaluated by other teachers and experts. This peer-evaluation is not very common in conventional class room education but can be easily performed in e-courses. This is used for process that ensures the high quality of the material presented. To this evaluation the external experts such as alumni may give their contribution. A feedback form for peer-evaluation will be provided on the EVICAB pages. Pedagogical issues are evaluated already when the course is implemented to the EVICAB system. Pedagogical evaluation is performed by applying a set of criteria. These criteria measure how the information is transferred from teacher to student via the internet and what kind of tools support the process. Evaluation in EVICAB Student feedback Self-evaluation Peer evaluation External evaluation Pedagogical evaluation 10

Phase 5. Data Processing Evaluation data gathered in the previous phase will be stored in a separate database. The database will contain also information on the criteria and on different parameters measured from the evaluation data. Dynamic Database Criteria Statistics Parameters Criteria used in evaluation are relative criteria. Relative criteria can be defined as a set of changing criteria in relation to statistics gathered from previous years. If the number of students has been decreasing rapidly during the last years or the number of students is increasing, the criteria are different. From the evaluation data and log-files (number of visits, number of students etc.) statistical information is collected. Statistical data will be the basis for parameters that are calculated from the data. These parameters are used in evaluation for the next year. Example of parameters measured Average numerical feedback Average number of visits Average number of students 11

DATA PROCESSING Statistics Evaluation data Parameters from previous year Dynamical Database Criteria Parameters Figure 4. Data processing in dynamical database Phase 6. Analysis All the data in dynamic database will be processed. Coordinator and Steering group will get the summary of evaluation data, statistics, and the parameters. Coordinator and members of the steering group will be given access to all evaluation data but the processed parameters are sent automatically. Based on these parameters and feedback steering group will make conclusions. 12

Phase 7. Conclusions Coordinator will draw conclusion in co-operation with the steering group about the course. Coordinator will compare the statistics and parameters, and to development related to previous year. Based on the data Steering group and coordinator will make the decision from the following options: Re-election (Course is approved as such for the next year) Development (Course is approved but some modification need to be done) Suspension (Course is not ready for next year and lot of modifications is needed) Cancellation (Course is not approved. Course has to start the process from the first phase of approval process) Internal evaluation cycle is not performed every year if there is no specific need for that. The evaluation cycle will be done every second or third year. Based on the internal evaluation cycle a need for new courses may arise due to suspension or cancellation of a course or if some field of expertise will need more courses to cover the curriculum requirements stated in Biomedea. 13

Implementation of the Dynamic Evaluation Model in EVICAB 2007 Kari Lindroos, Researcher, Ragnar Granit Institute, Tampere University of Technology Juha Nousiainen, Assistant Professor, Ragnar Granit Institute, Tampere University of Technology Mats Rajalakso, Mediamaisteri Group 14

IMPLEMENTATION OF THE MODEL The model defined in the modeling phase in work package 3 will be implemented to learning management system (LMS). The Model, illustrated in figure1, has various phases and all of these needs to be supported in LMS. In the EVICAB Project Moodle LMS is used and for the implementation the variety of tools in Moodle will be used. This report contains detailed information on the realization of the model in the LMS. DYNAMICAL EVICAB New course proposal from outside the EVICAB External Cycle Approval Form in EVICAB Moodle Coordinator Approval Steering Group Implementation and execution of the course Evaluation Implementation to EVICAB Need for new courses EVICAB COURSE Course 2007-2008 Course 2008-2009 Course 2009-2010 -Re-election -Development -Suspension Evaluation Student Peer Self External Pedagogi cal Internal Cycle Coordinator Steering Group Analysis Dynamic database Criteria Statistics Parameters Figure 1. Model for Dynamical EVICAB 15

1. Implementation of the External Cycle Implementation of the external evaluation cycle (Figure 2) is launched by application form on the first page of the EVICAB LMS. The institutes in the consortium as well as those who are willing to join the EVICAB will fill in an application form Course description, illustrated in figure 3. The input provided to the form will be used as the basis of the evaluation. Coordinator and the Steering Group will receive the application and make preliminary conclusion whether the course is suitable for the EVICAB or not. The reason for rejection in this phase could be the case that there already is a course covering the provided topics in EVICAB. The analysis and requirements will be discussed in chapter 7. EXTERNAL CYCLE New course proposal from outside the EVICAB Approval Form in EVICAB Moodle Phase 1. Coordinator Approval Steering Group Phase 2 Implementation and execution of the course Evaluation Phase 3 Implementation to EVICAB EVICAB COURSE Figure 2. External cycle 16

1.1 Course Description database The information provided in course description form will be stored in course description database. The database will contain the following fields: Course name Credits Person responsible University, Faculty, Department Department Address, telephone number, e-mail address Lecturers/assistants Level of the course (bachelor, master, PhD) Course homepage Prerequisite courses/topics Objectives of the course Contents of the course Course assessment criteria (exam, assignments, other criteria) Distant learning tools used in the course The percentage of contact teaching, distance learning, students independent study Additional information Course description database will form a basis for EVICAB Study Guide. The approved courses and specific data will be published on the first page of the EVICAB LMS (www.moodle.fi/evicab) in the course information summary. 17

Figure 3. Part of the Course Description Form Applications will be approved or disapproved in the order of current status of the provided courses. Course providers will be informed and the reason will be indicated if the course is rejected. The course providers with approved course will receive information and instruction how to proceed. The first step will be the opening of the course home page in EVICAB LMS. All the information, learning material and supporting material will be transferred to EVICAB or the Links to the material will be created to the courses EVICAB page. There exists a course template in The EVICAB LMS for all courses. This will make the sites more convenient for student to use, because any given information about the course will be found from the same section on the course pages. The teachers of every course need to update the course sites and fill in the content. Responsible person should familiarize with the Moodle features site which contains instruction for creating and using different LMS tools. 18

2. Implementation of the Internal Cycle INTERNAL CYCLE Need for new courses EVICAB COURSE Course 2007-2008 Course 2008-2009 Course 2009-2010 Phase 7 -Re-election -Development -Suspension Coordinator Steering Group Phase 4 Phase 6 Analysis Dynamic database Criteria Statistics Parameters Evaluation Student Peer Self External Pedagogical Phase 5 Figure 4. Internal cycle All courses in EVICAB will be evaluated regularly. The evaluation will be performed by using the internal evaluation cycle, illustrated in figure 4. The cycle consists of various evaluations and statements from coordinator and steering group. Evaluation data is gathered with a set of evaluation forms implemented to EVICAB LMS. The used evaluation forms will be presented in Chapter 3. The evaluation of all courses will be made from different point of views: pedagogical, student, peer, and self evaluation will be applied. 19

3. Evaluation Forms The learning management system Moodle has several features and tools for supporting the evaluation process. Discussion forums are used for open discussion on several topics from the course level to system level. Forums are good way to gather first hand information and feedback from the users. Other tools supporting the evaluation process are the various form creation tools. With these Moodle tools (Feedback, Form, Questionnaire) there are set of evaluation forms created. One main problem turned out when using tools built in Moodle. It is possible to collect evaluation data from several courses in one common course/workspace, but it is not possible for instance compare feedback between different courses in different categories. There are no tools which could be used for collecting data from different courses and comparing the collected data within needed parameters. Therefore there was need for new more flexible tools for collecting, saving and printing data. Within the EVICAB project and work package 3 totally new type of forms and more flexible databases were created in the learning management system Moodle. The forms are more versatile and there are more possibilities to create varied question forms. For using the forms, both collecting and printing data, no separate programs are needed just the web browser (Internet Explorer, Firefox). The forms are based on PHP and in the background there is a MySQL database. In this chapter the different ways implemented to the LMS will be described and discussed. 3.1 General Discussion Discussion forums for general discussion and feedback have been opened to every course page. These forums will give the instant feedback for the teacher from the functionality of the course as well as the content of the material. This evaluation is mainly given by the students. Similar discussion forum that has been opened to course page can be found from the front page of the EVICAB. The purpose of the forum is to get system level feedback from the users. This is the way for getting ideas and feedback for creating more convenient environment for the students and other users. 20

Discussion forum that is dedicated to evaluation of the quality of the EVICAB system and related issues has been opened on the evaluation center pages (defined in chapter 4). 3.2 Evaluation Forms for Students The most important evaluation data is received from students. The main focus of the whole system is to provide attractive, effective and convenient environment for students to study. For this reason there is several evaluation forms provided for students. Students may evaluate the courses, learning management system and give other comments. Students may give feedback on course pages concerning the particular course or in evaluation center pages. Feedback given on the courses is mainly direct for the teacher/teachers on the course for improving the courses. Feedback given in evaluation center pages is more for general use for system administrators of EVICAB, the alumni and teachers in general. Figure 5. Evaluation form for students in evaluation center 21

3.3 Evaluation Forms for Teachers and Assistants One key issue in the evaluation process is the self-evaluation. Teachers and assistants of all courses are asked to fill in self-evaluation form after every course they have given. If there is no time limit in the course the form should be filled once a year. The self-evaluation form can be filled in staff room or in evaluation center, which ever the teacher prefers. The idea is to let the course personnel think through the specifics of the course and consider if some modifications are needed for the next year. Figure 6. Evaluation form for teachers 22

3.4 Evaluation Forms for Alumni A new aspect in evaluation that will be provided in EVICAB is the feedback from external evaluators. These evaluators are experts in the field of biomedical engineering, professors, alumni, or representatives from industry. The idea is to get first hand information how the biomedical engineering education supports the needs of the industry. A specific feedback form is tailor made for the external evaluation in the evaluation centre. These experts may enroll to the courses, browse through the material and take part in the general discussion. Figure 7. Evaluation form for external evaluators 23

4. Evaluation Center The core of all evaluation in EVICAB is in the Evaluation Center. Evaluation Center is a Moodle site for all evaluation. This evaluation will be stored and analyzed with evaluation tools build for Moodle platform and EVICAB. (See chapter 3) 4.1 Evaluation Data Evaluation data is stored to feedback database. For all feedback forms there is a database created as the feedback form has been created. From this database the data can be visualized in tables; Moodles own table, see table 1 and the new tailor made table, see figure 8. The evaluation tool calculates the percentage of one particular option in relation to all answers. Table 1. Example feedback data analysis 1.) How useful were the lecture? - Very useful (0): 16 (41.03 %) - Quite useful (0): 15 (38.46 %) - Some (0): 5 (12.82 %) - Not so useful (0): 1 (2.56 %) - Not at all (0): 0 - No opinion (0): 2 (5.13 %) Average: 0.00 A teacher can choose one of her/his courses and get the feedback given by students. The amount of feedback givers are possible to see and the mean value of the feedback on a specific course. The teacher will also be able to see mean value for all courses in EVICAB, thus it is easy to compare the own course with the rest. 24

Figure 8. Feedback data for teacher from students 4.2 Parameters Evaluation of the courses in the EVICAB will be performed with evaluation and feedback forms. Based on this evaluation data the courses will be graded. The parameters that are considered in this grading are: Number of visitors Number of students General feedback Based on the grading the courses will be evaluated by the steering group. Steering group will analyze the feedback and give further instruction for the responsible teacher. 25

Based on the data Steering group and coordinator will make the decision from the following options: Re-election (Course is approved as such for the next year) Development (Course is approved but some modification need to be done) Suspension (Course is not ready for next year and lot of modifications is needed) Cancellation (Course is not approved. Course has to start the process from the first phase of approval process) 26

5. Evaluation in Practice In practice the evaluation process is filling different feedback form. Especially the main users, students, will be able to evaluate the EVICAB from different levels; Education, pedagogical issues, usability, functionality, learning management system. Even though a number of forms are required the whole process will be systematic and easy, and hence it will be quite self-evident action in addition to other educational tasks. Feedback form for students will be found from each course site. The form will be automatically added every time a new course is added to the system. Evaluation in EVICAB: Student feedback Self-evaluation Peer evaluation External evaluation Pedagogical evaluation 27

6. Responsible partners The responsible partners can be divided into three (3) groups: System administrator Member of the steering group Teachers and assistants 6.1 System Administrator The responsibility of managing the feedback data is a task for system administrators. System administrator will make sure that all the needed forms are provided and functional. System admin will provide the gathered data for the steering group and calculate parameters if needed. 6.2 Steering Group The members of steering group will receive the feedback data and calculated parameters. Based on this data the steering group will grade the courses in scale 1 5 according to the table 2. 6.3 Teachers and assistants Course teachers and assistants are responsible for the actions and improvement suggestions received from the steering group. The requested modifications need to be done prior to the next execution of the course. 28

Table 2. Course evaluation grades Grade Explanation Action 1 Poor/inadequate Course Immediate actions needed for improving the course and producing more high quality course material. Possible suspension of the course. 2 Insufficient Course Improvements needed. Course may be given next time if the modifications have been done. 3 Acceptable Course All the components of e-course are provided. Still some work has to be done for updating and upgrading the course. 4 Good Course The quality of the course is good in both content and functionality. No actions necessary but some modifications could bring the course to a new level. 5 Excellent Course No need for actions 29

7. Management of the Evaluation 7.1 Management of the evaluation data The feedback data in evaluation database will be summarized and average numbers will be calculated automatically. All the data and calculated parameters are stored to databases under the Evaluation Center site. 7.2 Processing of the evaluation data The evaluation data concerning a particular course may be printed out and analyzed. The system will calculate automatically average numerical feedback from the evaluation data. 7.3 Conclusions from the data Conclusions and analysis will be based on the evaluation data and the log-data. Parameters that the system will give for steering group are: numerical feedback, number of students, and number of visits. 30