Rosa Arruabarrena, Tomás A. Pérez, J. López-Cuadrado, J. Gutiérrez, and J.A. Vadillo

Transcription

1 On Evaluating Adaptive Systems for Education Rosa Arruabarrena, Tomás A. Pérez, J. López-Cuadrado, J. Gutiérrez, and J.A. Vadillo Dpto. de Lenguajes y Sistemas Informáticos, Universidad del País Vasco (UPV-EHU) Apdo San Sebastián, Spain Abstract. In this paper we have gathered some interesting techniques to evaluate educative systems. Our main purpose is to evaluate HEZINET, an adaptive hypermedia System for education available commercially. Therefore, we also include a system evaluation plan to identify which types and techniques of evaluation will be needed in order to accomplish it. 1 Introduction HEZINET is an Adaptive Hypermedia System [1] to learn Basque language. It automatically compiles exercises adapted to the concepts covered by the nodes that the student has already visited using the Item Response Theory and AI techniques [2]. The system has been validated and more than 2000 copies have been sold by now. However, we think that we can improve HEZINET. In order to do that, we have planned to perform different evaluations. At the moment we consider two uses of the system that we want to differentiate. On the one hand, the system is used as a distance learning system available on Internet to subscribed students (we will call it HEZINET-D for short). On the other hand, it is also being used in intranets by students in Basque academia as an extra support to traditional Basque classes (and we will call HEZINET-C for short). Until now, we have already made a usability test using surveys. The conclusions lead to some changes of location of functionality in the interface and to include some other contents to adapt to very beginners in Basque [3]. Now, together with the company that sells HEZINET we are interested in evaluating the effectiveness of the system and the advantages of the adaptation it provides [4]. This paper presents a wide study of strategies for software validation, types of evaluation to be considered when assessing computer assisted learning and concrete techniques to perform them. More detailed techniques are presented in [5]. A detailed overview on techniques that focus mainly on adaptation can be found in [6]. Finally, we present a plan to evaluate both versions of HEZINET.

2 2 Validation Strategies Software validation focuses on demonstrating that programs meet their requirements. We call the processes that check this validation strategies. In [7], several validation strategies are presented, such as (a) Proofs of Correctness, that consist on validating and verifying programs using formal proofs of correctness; (b) Criterion Based Validation, that is commonly used in software engineering (here the system is considered successful if it displays no major inadequacies within its intended application environment); (c) Expert Review in which a person (the expert) observes the whole system, parts of it and/or its behaviour and identifies its deficiencies; (d) Certification, that is based on techniques of software quality and deterministic rules; (e) Empirical Proofs, that use experience of user with the system as a measure. 3 Evaluation The evaluation of an educational system is a process of data gathering to determine the quality or the value of the instruction, its weaknesses and its strengths [8]. In [7,9,10] different dimensions of evaluation of these systems are presented. Two of them are the Evaluation Paradigms, that monitor what the evaluation goals are and the Evaluated Elements that classify those parts of the educational system to be assessed. Both are presented in detail in the following paragraphs. There are mainly three evaluation paradigms: (1) Formative Evaluation, which is oriented to detect weak points at early stages of the development of the product in order to modify and to improve it; (2) Summative Evaluation, that addresses punctual assessments of effectiveness of the finished application. The evaluation parameters are goals, benefits and costs; (3) Integrative Evaluation, which measures the level of integration between the educational software and other learning materials and methods into the context overall. If one looks at the elements being observed, three kinds of evaluation could be distinguished: (1) Internal Evaluation that implies that the system itself is assessed (for instance, different components of the system can be evaluated, such as the architecture, the intermediate processes, as well as behaviours of and the relations among the previous); (2) External Evaluation, that means that only elements out of the learning system are assessed, such as, student achievements or the affective impact of the system; (3) Global Evaluation, that evaluates components and impact on users. 4 Evaluation Techniques Evaluation techniques are concrete methods to carry out the validation of the system. Four most common ones are [10,11]: (1) Comparison: is based on comparing the characteristics of the system versus some standard or other system (for example, there are methods, such as golden standard, theoretical corroboration, empirical corroboration, duplication, Turing test, sensitivity analysis and benchmarking). (2) Contact

3 with users whose objective is to collect data about how the user interacts with the system, his/her behaviour and attitude (for instance, one can use interviews, surveys, questionnaires, focus groups, nominal groups, and tests). (3) Data analysis, that consists on reviewing, studying and assessing groups of data about certain characteristics. The information collected, processed and reported should be systematically reviewed and handled, and if any errors found, those should be corrected [12]. (4) Pilot testing which involves studying the performance of the system with potential end users. Depending on the number of users that are evaluated, we distinguish oneto-one testing, small-group testing, field testing and beta testing. 5 Evaluation of HEZINET To evaluate the effectiveness of both versions of HEZINET and taking into account [6,13,14], we have established a plan to: (1) improve the tools associated to the system; (2) assure that the adaptation results in better performance; (3) check the impact of the system as a motivating element of learning; (4) to attract users affectivity and retain the interaction with them as long as possible; (5) achieve excellent performance of students from the knowledge acquisition point of view, either using the system alone or transferring this acquisition to other contexts different from the computer; (6) guarantee that the system does not mislead students, independently from their computational skills, and that students can locate the desired contents either in HEZINET or in other related pedagogical material available. We planned to perform several the interface evaluation loops (internal summative evaluation) to measure different features. Although it could be considered formative, because the results will improve the product, we consider that the product has been already developed, and that we are working on a new version of the system. The plan is to create two groups of students (those involved with HEZINET-D and the ones with HEZINET-C). First, we check how students use the elements of the interface by means of (1) questionnaires detect weak points of the interface; (2) contacts with experts in Human Computer Interaction for alternative solutions to the identified problems and (3) empirical corroboration to choose the best prototype by means of (4) pilot testing on both populations. Therefore, our aim is to check if the new design solves the problems detected without interfering the rest of users. Once the new interface is validated, and if it were required, it will be integrated into a new version. To assess the affective impact of HEZINET, we will use open-ended questionnaires, in which users should indicate which elements are more helpful and pleasant. This evaluation (external and summative) could result in a reorganisation of the contents or in an inclusion of more multimedia courseware. To evaluate the system s effectiveness, we are interested in finding the differences among the learning results obtained from three populations: students from HEZINET- D, from HEZINET-C and from those that learn attending regular classes at school but without any tool of this sort. In order to conduct this (external summative) evaluation, we should analyse the user model. We will compare the data obtained from each population to check if any type of correlation among them exists. Concretely, when-

4 ever it would be possible, we will handle data from the system logs, from lists of results in tests completed at schools, in an anonymous way, and from the list of marks obtained in official tests of linguistic capability. As a result of the analysis, the hypotheses that correspond will be established. To verify the suitability of HEZINET-C within an institution like an academy of languages, it is important to assess whether the educational methods include the application as one more tool inside the classes or as an additional supporting service. We will conduct an (external integrative) evaluation using (1) interviews to students and teachers and (2) data gathering by direct observation. We consider that it is very complicated to evaluate the integration of HEZINET-D students and the learning material they may handle. Nevertheless, we could give some advice about the work environment for a good learning. 6 Conclusions Although it is not usual to evaluate systems after they are already in used, the goals of the education can vary and, perhaps, it is necessary to make certain adjustments to the initial proposed topics. It is important not only to evaluate the system but also to value the integration within the context where the learning will take place. Some synergies can be obtained thanks to the integration, which can do the system s potential be even higher than the expected one if the application were considered only separately. In this paper we have presented a wide study of types of evaluations that appear in educational systems. In the same way, it focuses on, possibly, the most interesting elements to conduct an evaluation of an educational adaptive hypermedia system. We have also discussed different goals we can consider and the system elements on which centre the study. Moreover, we also report about the techniques to be used to obtain a successful evaluation. Among these techniques, types of evaluations and strategies we have picked out the most interesting ones to evaluate HEZINET. Future work, as well as conducting the proposed evaluation plan, include projects to adapt HEZINET to other languages such as Spanish and French since the system is successful. References 1 Brusilovsky, P. (2001). "Adaptive Hypermedia." User Modeling and User-Adapted Interaction.Special Issue on: Ten Year Anniversary Issue 11(1/2): Pérez, T.A., López, R., Gutiérrez, J., González, A.: Learning Basque in a Distance-adaptive way. Computers and Education in the 21st Century. Kluwer Academic Publishers, Dordrecht, The Nederlands (2000) Villamañe, M., Gutiérrez, J., Arruabarrena, R., Pérez, T.A., Sanz-Lumbier, S., Sanz- Santamaría, S., López-Cuadrado, J., Vadillo, J.A.: Use and Evaluation of HEZINET; A System for Basque Language Learning. In: Lee, C.-H., Lajoie, S., Mizoguchi, R., Yoo, Y.D.,

5 du-boulay, B. (eds). Proceedings of the 9th International Conference on Computers In Education (ICCE/SchoolNet2001), Seoul (South Korea) (2001) 4. Karagiannidis, C. and D. G. Sampson (2000). Layered Evaluation of Adaptive Applications and Services. International Conference on Adaptive Hypermedia and Adaptive Web-Based Systems: AH2000, Trento (Italy), Springer-Verlag Berlin Heidelberg. 5. Arruabarrena, R., T. A. Pérez, et al. (2001). Compendio de técnicas para evaluación de sistemas hipermedia adaptativos. Simposium Internacional de Informática Educativa, SIIE, Viseu, Portugal. 6. Weibelzahl, S.: Evaluation of Adaptive Systems. UM2001: Proceedings of the eighth International conference on User Modeling, Springer: Berlin (2001) 7. Mark, M.A., Greer, J.E.: Evaluation Methodologies for Intelligent Tutoring Systems. International Journal of Artificial Intelligence in Education, 4(2/3) (1993) Tessmer, M.: Planning and Conducting: Formative Evaluations. Kogan Page Limited. London (1993) 9. Draper, S.W., Brown, M.I., Henderson, F.P., McAteer, E.: Integrative evaluation: an emerging role for classroom studies of CAL. Computers and Education, 26, (1-3) (1996) Murray, T.: Formative Qualitative Evaluation for Exploratory ITS Research. International Journal of Artificial Intelligence in Education, 4(2/3) (1993) Harvey, J.: LTDI Evaluation Cookbook. Learning Technology Dissemination Initiative. Edinburgh (1999) 12 Frechtling, J., Sharp, L.: User-Friendly Handbook for Mixed Method Evaluations. NSF, Arlington, VA (1997) 13 Chin, D. N.: Empirical Evaluation of User Models and User-Adapted Systems. User Modeling and User-Adapted Interaction 11(1-2) (2001) Shute, V.J., Regian, W.: Principles for Evaluating Intelligent Tutoring Systems. International Journal of Artificial Intelligence in Education, 4(2/3) (1993)