Giving More Adaptation Flexibility to Authors of Adaptive Assessments

Transcription

1 Giving More Adaptation Flexibility to Authors of Adaptive Assessments Aimilia Tzanavari 1, Symeon Retalis 2 and Panikos Pastellis 1 1 University of Cyprus Department of Computer Science 75 Kallipoleos St. P.O Box Nicosia, CYPRUS 2 University of Piraeus Department of Technology Education and Digital Systems 80 Karaoli & Dimitriou Piraeus, GREECE aimilia@ucy.ac.cy, retal@unipi.gr, pastelis@cytanet.com.cy Abstract. In this paper, we present a tool for authoring personalized assessments, which gives the author significant flexibility in terms of the adaptation the he can incorporate in the assessments he builds. Furthermore, we introduce the user model that is used by the system, along with the philosophy of its manipulation, which differs to a degree from that of other systems. We demonstrate that this tool fully conforms to the IMS-QTI standard, serving the goal of interoperability. 1 Introduction Involving students in an educational assessment process is an essential part of the whole learning process. The process of educational assessment entails the estimation of the user s knowledge level on the topics being assessed, based on evidence derived from observation of what they say or do in selected situations [8]. In a non-adaptive Web-based Educational System, information derived from assessment, as well as from the user s interaction with the system can be stored for users to observe their personal learning progress and to decide how to proceed, and for instructors to be able to support users, but also evaluate the effectiveness of the material presented. However when the adaptive parameter is added, making it now an Adaptive Educational Hypermedia System (AEHS) [1], the assessment process leads to information about the user s knowledge, strengths, weaknesses, etc., which is essentially the fuel for the adaptation engine. When adaptation is available within the assessment itself, it can take the form of adaptive testing [12] and/or adaptive questions [2], thus providing individualized assessments. In this paper we present a web-based adaptive testing system, called AthenaQTI tool (Athena is a Learning Management System where the AthenaQTI tool is going to be integrated). The AthenaQTI tool has been designed taking into account three very important issues. Firstly, that the user model used by the system conforms to a widely

2 used standard so that other related applications are able to use it. Second, that the system conforms to IMS QTI specification [7]; IMS QTI provides the basic structure for the assessment (representation of questions, results, etc.). It also enables the interoperability and portability of the assessmements among either web-testing systems or even Learning Management Systems (like WebCT, Blackboard, etc.). Thirdly, the system should provide an authoring environment that will allow the author of an assessment, i.e. the educator and/or an instructional designer, to make decision on matters such as: the different types of questions: multiple choice, fill-in the blanks, etc.), when are questions considered easy or hard, whether more detailed difficulty levels are required, what is considered a qualifying grade and under which circumstances, how are grades to be interpreted in terms of the user s knowledge level, how is a new learner handled, how many questions are necessary to estimate the user s knowledge with confidence, how does the user s performance affect the user model and many more. The paper is organized as follows. In Section 2, a brief description of adaptive testing and adaptive questions adaptation techniques is provided. Section 3 briefly presents an existing adaptive assessment authoring system (one of the very few, up to our knowledge), called SIETTE. In Section 4, the description of the user model used by our tool is presented and in Section 5, the AthenaQTI tool itself is presented, mainly focusing the functionality of the authoring environment and its conformance to IMS- QTI. It should be noted that the presented system is the only adaptive testing system that i) conforms to QTI and ii) has an authoring environment for adaptation rules. Finally in Section 6, we conclude by summarizing and planning further research steps. 2 Types of Adaptation in Assessment Systems Adding the adaptation capability to the assessment process has been proven advantageous, primarily for the reason that users are presented with personalized tests, tailored to their needs, preferences and current knowledge, but also because the number of assessment items required can be adjusted, most of the times resulting in fewer items, which implies a shorter, less tedious assessment. Two adaptation techniques have been adopted by assessment systems. One type of adaptation technique is Adaptive Testing. This involves a computeradministered test in which, the selection/ presentation of each question and the decision to stop the process are dynamically adapted to the learner s performance in the test [12]. Computer-adaptive testing is based on Item Response Theory [11]. The criterion for selecting questions in the adaptive testing process is to match the question s difficulty level with the learner s estimated knowledge level. This is because it has been shown that these are the type of questions that are more informative in terms of conclusions one can draw on the learner s knowledge. The goal is to accurately estimate the learner s knowledge. Several approaches exploit the idea of adaptive test-

3 ing. Huang in [3] describes the CBAT-2 adaptive testing algorithm that generates content-balanced questions and SIETTE [10] is an adaptive assessment authoring tool. The other adaptation technique used in assessment is adaptive questions. This causes the generation of a dynamic sequence of questions depending on learner s responses. In this case, several predefined rules, in conjunction with the learner s responses, are those that lead to the selection of the question(s) to follow. This requires however highly structured pools of questions, so that rules have enough information to work with and select questions. Adaptive questions have been used mainly in computer-assisted surveys. Pitkow and Recker in [9] showed that Web-based adaptive questionnaires can reduce the number and complexity of questions posed to users. Furthermore, in CATES [2], adaptive questionnaires are used to assess Web users attitudes. The general philosophy of assessment in Educational Systems that deal with it either exclusively or partially is depicted in figure 1. Fig. 1. Philosophy of Assessment in Educational Systems 3 Existing Assessment Authoring Systems - SIETTE SIETTE [10] is a web-based tool used by educators to assist them in the assessment process. Educators can create assessments that students can take through the Web. SIETTE can be used either as a stand-alone assessment tool, or as a module in an Intelligent Tutoring System. SIETTE is based on the adaptive testing adaptation technique. Assessments are adaptive and questions are selected in such a way that they fit to each student s knowledge level. The termination criterion is set so that two different students may well receive two different tests, both in terms of questions content and number of questions posed. SIETTE mainly handles multiple choice questions. In SIETTE, question selection is based on a function that estimates the probability of a correct answer to a particular question, leading the student s level of knowledge. The question, amongst the pool of the questions that have not been posed yet, with the highest probability will be posed. This is performed iteratively, until the termination

4 criterion is met. Every time the student answers a question, a new proficiency level is estimated taking the given answer (whether correct or not) into account. While the test progresses, a temporary student model is updated. The temporary student model is a data collection required for SIETTE system to estimate with accuracy the real proficiency of a student on a specific subject of the whole curriculum. This model is used by the diagnostic module of the ITS in order to advance across the curriculum, to give advertisements, etc. The temporary student model differs from the student model of the ITS, which information is more complete. The temporary student model contains only information about the proficiency level of the student at a given moment of adaptive test. A temporary student model [10] is shown below: Finally, the SIETTE system has a learning profiles generator module that gives information about each evaluated student. These profiles are available for the teacher and the student and are a form of result processing, including statistical data about the student s performance. 4 The user model used A user model in an AHES is essentially the information that the system holds about the user and that is mainly related to the learning process. This information has to be such that the system can better adapt to the user s individual needs. When observing the learner-computer interaction, several types of adaptation can take place, some of which do so explicitly and some implicitly, based on the interactive events, the path that the learner has taken, the performance concerning some learning tasks, etc [4]. An adaptive or adaptable educational hypermedia enriches the application functionality by maintaining a representation of the user (or user model ) and providing customization mechanisms to modify application features in response to user model updates. For adaptive educational hypermedia, user model updates are automatically generated by the system (by monitoring and interpreting the user s interactions); for adaptable hypermedia, user model updates are under the user control. The adaptivity/adaptability feedback could be related to the organizational and presentational issues of the learning resources (permission to move forward, encouragement to read specific sections, undertake some tasks, etc.). The user model used by the AthenaQTI tool has a particular form and specifically consists of the following elements: Demographic data - data that remain unchanged, such as age, gender, etc. User goals, which are related to the long term and short term learning goals related to learning objectives of specific concepts to be learnt (e.g. to complete course X )

5 User preferences, with respect to the various dimensions of the learning opportunity (e.g. the mode of delivery, accessibility requirements, or assessment) User knowledge, which includes the knowledge level about concepts to be learnt and weaknesses and strengths on particular areas, sections or points of the concepts Usage data historical data which include information like which pages were viewed, in what order, for how long, etc. The stereotype that applies to the user, which essentially is the group of users s/he belongs to based on some predefined presuppositions in terms of knowledge level, learning and cognitive styles (e.g. the Novice User, the Expert User, the Acoustic user, the Activist user stereotypes etc.). The first three elements are considered to be changeable only by the user, whereas the last three are monitored and updated by the system. There is only one user model that describes a user of our system, contrary to SIETTE that uses two: a temporary student model with performance information and the student model of the ITS, which is more complete. 5 The AthenaQTI tool The ability of an authoring tool to work with other e-learning software and systems is referred to as interoperability. Successful interoperability is the result of software compliance to technology standards. Within the e-learning community, several sets of technology specifications and standards have been proposed most of which are still evolving. The ultimate vision is to have interoperability among the e-learning tools and integrated platforms. Until then, the e-learning community is fragmented into different systems adhering to various standards. In the area of self-assessement a well known, widely adopted and quite stable specification has been proposed by the IMS Global Learning Consortium [6], named IMS QTI (Question and Test Interoperability) [7]. IMS QTI permits the representation of assessment test in a standard XML format thus allowing the interoperability for questions and tests between different assessment tools. QTI s structure contains elements such as assessments, sections, and items. An item is the formal name for a question within QTI, and an assessment is the name used for a test within QTI. The terms "item" and "assessment" are considered more precise and useful for the formal specification, than the looser but more widely understandable "question" and "test" words used in the QTI title. A section is a sub-section of an assessment and/or a grouping of questions. An assessment can contain sections and/or items. And a section can contain other sections and/or items. Figure 2 shows the possible cases.

6 Fig. 2. Different assessments supported by IMS QTI (taken from [7]) With the AthenaQTI tool, authors are able to create the following type of questions: True/False (Fig. 3) Multiple choice single response (Fig. 4) Multiple choice multiple response Multiple choice ordered response Fill-in the blanks Multiple Image choice item (single or ordered response) Image hot spot item (multiple or ordered response) Fig. 3. Screen for creating a True/False Question

7 Fig. 4. Screen for creating a multiple choice question The assessments are represented in XML [7], so that they can easily be exported and used by other applications that are also IMS compliant. Furthermore, authors can later open and edit an existing assessment. The assessments are presented to users, who first have to log on, so that their user model is uploaded. Feedback is automatic and can be utilized in several ways. Multimedia objects can be embedded into assessments, so authors/educators can create better interfaces and users can have a more attractive learning environment. Assessments are structured exactly as QTI dictates so that the AthenaQTI tool fully conforms to the specification (Fig. 2). Moreover, we innovated in a number of ways, starting from the use of the qti-metadata element (which is used for describing the subject domain that an assessment, section and item concern). The QTI-meta data element should be related to a vocabulary of terms of the subject domain. For experimental use, we have carefully analyzed the curriculum structure for studies in computer science provided by IEEE/ACM [5], in order to create a vocabulary that could be widely adopted. This vocabulary assisted us in the task of describing the topics that an assessment or a section refers to. Figure 5 shows the screen for creating a new assessment, where the author can select the assessment s topic going down three levels if necessary. Based on the assessment s topic, all sections created (Fig. 6) will refer to a sub-topic. By allowing the author to select the appropriate topic(s) for his/her assessment, only from a given hierarchically structured vocabulary, we alleviate several future difficulties. A brief discussion of these difficulties requires that we first visit the issue of adaptation in the system. It can be easily understood that changing the vocabulary (e.g for another subject domain) the functionality of the tool does not alter. Since a lot of work is being done in the area of ontologies it is a matter of time to see such vocabularies arisen for other subject domains than Computer Science.

8 Fig. 5. Screen for creating a new assessment Fig. 6. Screen for creating a new section under a specific assessment The other innovative aspect of the Athena QTI tool is that it allows the author to create both adaptive and non-adaptive assessments. Currently, the tool only supports the adaptive questions technique. Questions (which correspond to items) are selected and given to the user, according to a set of rules that the author creates. These rules take the form of IF-THEN rules, where the condition refers to user model information (see Section 4), and the action refers to the resulting change in the assessment. When users log on to take the test, they will be described by their personal user model. This will most probably be carried around throughout their navigation of other educational (or even other type of) applications as well. As the user model has to form an accurate image of the user, it needs to be updated frequently to include any changes. The core attribute, which is the one (and in some the only) used by most AEHS for adaptation purposes, is the user s knowledge. Knowledge is directly related to educational applications, whose goal is to instruct the user. The clearer the system s view on what the user knows, the better it can adapt. However, apart from knowledge, we keep usage data and stereotype information, from the system-dependent category.

9 The user s knowledge is updated in our tool, based on the assessments/sections topics. The algorithm used to update the user s knowledge is again implicitly given by the author through the rules s/he creates. Examples may be If the user s score in section C was over 80% then increase the knowledge of the section s topic by factor X. This results in a user model, whose knowledge attribute consists of topics (along with a degree of knowledge) that originate from a very specific vocabulary. In this way, if the user model is later used by a different educational application that is aware of this vocabulary, it will automatically be recognized without difficulty. Imagine if a user model includes great knowledge of Artificial Intelligence and the application that tries to use it and possibly examine the user s knowledge in AI assumes that it does not exist (since it is looking for a match with AI )! Usage data includes historical information about the user s performance in the test. It is very useful to be able to keep track of the user through a sequence of sections or items. This information is required in cases where the author wants to apply rules such as If the user performed very well in section A and very poorly in section B, then give a new section with intermediate difficulty level. Stereotypes are widely used in personalization, especially to overcome the cold start problem: how is the user model initialized. In our tool, we allow the author to define the number of stereotypes s/he considers necessary for the particular assessment, as well as each one s characteristics. Following this, the author is able to employ stereotypes in the rules s/he builds, not only to be applied at the beginning of the assessment, but at any other point. Currently, stereotypes, once defined by the author, remain unchanged throughout the assessment. The algorithm used by our tool to deliver an adaptive assessment can be summarized in the following: Determine which stereotype the user belongs to Do Select appropriate item to present Receive Answer If ( rule applies) then If (rule is of UM_update type) then update UM accordingly Else if (rule is of item selection type) update item selection criteria While (NOT end of assessment) Update UM accordingly 6 Conclusions In this paper we presented a web-testing tool that conforms to the IMS QTI specification. Up to our knowledge, SIETTE tool is the most relevant tool with which AthenaQTI can be compared. SIETTE shares a number of features with our tool, primarily that they are both adaptive assessment authoring systems, however the former uses ICC to select the question that will most probably fit the user s knowledge, placing it under the adaptive testing adaptation technique category. Our tool does not use

10 functions to estimate any parameter; rather the author is given the flexibility to express his/her didactical philosophy and methods through the creation or appropriate rules. Furthermore, our tool supports a wider range of question types than SIETTE, which seems to handle mainly multiple-choice. The most important advancement of the tool presented in this paper, lies in the facts that a) it fully conforms to the IMS QTI standard, making it very powerful nowadays, when interoperability is a vital issue, and b) it has an authoring environment. The AthenaQTI tool is still in its infancy and we need to proceed to formal testing with real users since we have only performed laboratory tests focusing on usability and software quality issues. Near future plans consist of the evaluation of the adaptation features of the assessments that will be created. This will involve testing with real users, observing and measuring a number of parameters, e.g. students performance in non-adaptive tests and in adaptive tests. References 1. Brusilovsky, P.: Adaptive and Intelligent Technologies for Web-based Education. In: Rollinger, C., Peylo, C. (eds.): Special Issue on Intelligent Systems and Teleteaching. Kunstliche Intelligenz, Vol. 4 (1999) Chou, C.: Constructing a computer-assisted testing and evaluation system on the World Wide Web the CATES experience. IEEE Transactions on Education 43(3) (2000) Huang, S.: A Content-Balanced Adaptive Testing Algorithm for Computer-Based Training Systems. In: Frasson, C., Gauthier, G., Lesgold, A. (eds.): Intelligent Tutoring Systems, Third International Conference, Springer (1996) Kobsa, A.: Generic user modeling systems. User Modeling and User-Adapted Interaction, 11(49):49 63, (2001). 5. IEEE/ACM Computing Curricula 2001 CS Body of Knowledge, 6. IMS Global Learning Consortium Inc., 7. IMS Question and Test Interoperability Specification, 8. Pellegrino, J., Chudowsky, N., Glaser, R. (eds): Knowing what students know: The Science and Design of Educational Assessment. National Academy of Sciences, National Academy Press, Washington (2001) 9. Pitkow, J., Recker, M.: Using the Web as a Survey Tool: Results from the Second WWW User Survey. Computer Networks ISDN Systems 27(6) (1995) Rios, A., Pérez de la Cruz, J.L., & Conejo, R.: SIETTE: Intelligent evaluation system using tests for TeleEducation. In: Workshop "WWW-Based Tutoring" at Fourth International Conference on Intelligent Tutoring Systems (1998) 11. van der Linden, W.J., Hambleton, R.K. (eds.) Handbook of Modern Item Response Theory, Springer-Verlag, London (1997) 12. Wainer, H., Dorans, N., Eignor, D., Flaugher, R., Green, B., Mislevy, R., Steinberg, L., Thissen, D. (eds): Computerized Adaptive Testing: A primer. 2nd edition. Lawrence Erlbaum Associates, Hillsdate NJ (2000)