Enhancing SCORM metadata for assessment authoring in e-learning

Transcription

1 Original article Enhancing SCORM metadata for assessment authoring in e-learning Wen-Chih Chang, Hui-Huang Hsu, Timothy K. Smith & Chun-Chia Wang Multimedia Information NEtwork (MINE) Lab, Department of Computer Science and Information Engineering, Tamkang University, Tamsui, Taipei Hsien, Taiwan, ROC Department of Information Management, Kuang Wu Institute of Technology, Peitou, Taipei City, Taiwan, ROC Abstract Keywords With the rapid development of distance learning and the XML technology, metadata play an important role in e-learning. Nowadays, many distance learning standards, such as SCORM, AICC CMI, IEEE LTSC LOM and IMS, use metadata to tag learning materials. However, most metadata models are used to define learning materials and test problems. Few metadata models are dedicated to assessment. In this paper, the authors propose an assessment metadata model for e-learning operations. With support from assessment metadata, we can incorporate measured aspects of the following list into the metadata description at the question cognition level, the item difficulty index, the item discrimination index, the questionnaire style and the question style. The assessment analysis model provides analytical suggestions for individual questions, summary of test results and cognition analysis. Analytical suggestions provide teachers information about why a question is not appropriate. Summary of test results improves the teacher s view of student learning status immediately. Items missing from the teaching materials can be identified by cognition analysis. In this research, the authors propose an enhanced metadata model and an implemented system based on our model. With metadata support, metadata can help teachers in authoring examination. assessment, cognition level analysis, e-learning, metadata Introduction With the popularity of distance learning, learning materials and group communications are widely established on the Internet and other wireless environments. Whether in a distance-learning environment or a traditional classroom, instruction and assessment operate together as a complete learning cycle. However, the instructor may miss some course key points when editing an examinaton. On the other hand, examination plays an important role in learning cycle. Assume that students took a pretest first, and then they prepared posttest according to the pretest examination Accepted: 6 March 2004 Correspondence: Wen-Chih Chang, 151 Ying-chuan Road Tamsui,Taipei County Taiwan 251, Republic of China. g @tkgis.tku.edu.tw paper. A good assessment system provides a suitable method to gather student feedback. With our assessment analysis, the teacher can adjust his/her teaching strategy, and redesign or reorganize learning materials if necessary. In addition, students can also learn the key point of learning materials. The analysis of assessment provides the learners with the most important element of each subject and each course, individually. Good assessment can diagnose if a learning activity achieves its learning object and whether the learning content is good for the learner. Moving a course from one Learning Management System (LMS) to another can be a time-consuming problem due to the variety of their content databases. In order to get over the difficulties of portability, some standards aim to provide the specifications necessary to enable content developers. However, it is difficult for most instructors who 305

2 306 W.-C. Chang et al. are not familiar with distance learning standards, such as SCORM, to deliver learning content and examination according to the standard. In the following sections, we first discuss several e-learning standards. Then we propose the assessment metadata elements with their meanings. The problem analysis model and examination analysis model are defined to support the standard s weakness. In the architecture section, we discuss and implement a prototype and an authoring interface. Finally, we compare our model with other e-learning standards before Conclusion and Future Work. Related work Many e-learning standards exist. However, different standards emphasize different aspects. A brief summary of these e-learning standards follows: SCORM introduction The Sharable Content Object Reference Model (SCORM) is an aggregated specification for asynchronous distance learning, organized by the Advanced Distributed Learning Initiative (ADL) ( One important issue in SCORM 1.2 is the technique of packing course material sources, structure and metadata into an exchangeable object. A few changes of metadata definitions were introduced in SCORM 1.2. However, in SCORM 1.2, only a few issues related to interaction were discussed. The SCORM 1.3 draft specification was published in late One important change compared to SCORM 1.2 is the adoption of the IMS Simple Sequence Specification. Interactivity and tracking of the learning status of individual users were addressed in detail in SCORM 1.3. The data model of terminologies recognized by the SCORM 1.3 run-time environment is also revised in the SCORM 1.3 specification. The technical issues are presented in three parts: the content aggregation model (the format of courseware), the run-time environment (the protocol of running courseware) and the simple sequence specification (with learning status tracking, sequencing rules and application program interfaces). In order to make courseware reusable, a standard representation of contents and structures must be enforced. Content Aggregation Model (CAM) serves this purpose. CAM can be discussed in three parts: the Content Model, the Metadata and the Content Packaging. The Content Model defines the content components of a learning experience. That is how learning materials are organized into different levels of small sections. Metadata definition is a set of standard items, which is used to describe the Content Model. Metadata provide an efficient and effective mechanism for content searching. Content Packaging is a standard definition to allow the content model and the structure to be packed into a standard exchangeable file, known as Package Interchange File (PIF). Standard representation of course content only helps reusability if the contents section is found. This indicates that, for an instructor to find suitable contents for reuse, an effective and efficient search mechanism is necessary. The traditional searching mechanism is for normal purposes like the portal web site CAM provides a standard courseware representation format for content exchange. From the perspective of software systems, courseware should be available on different computers and software platforms. The purpose of the SCORM run-time environment is to establish a standard protocol for the courseware to communicate with its underlying Learning Management System (LMS) that is independent of both machine and the operating system. Learning Object Metadata (LOM) Metadata provide a common nomenclature for learning resources to communicate and exchange with others in a common way. A good metadata model needs completeness, carefulness and flexibility. The most famous metadata model is the IEEE LTSC LOM (Hodgins 2001). It provides nine categories to describe learning resources. The categories include General, Classification, Annotation, Lifecycle, Technical, MetaMetadata, Educational, Relation and Rights. Several international e-learning standards, such as IMS, ADL and ARIADNE are based on LOM. Course hierarchy and structure In an e-learning environment, the course structure affects the learning resource transformation and educational knowledge constitution. According to the AICC nomenclature, the course structure is divided

3 Enhancing SCORM metadata e-learning 307 into two elements. One is the assignable unit, which is used to present to students, for example, a HTML file, an image file, or a video file. The other is the block, which collects assignable units. However, in order to include some additional features and reference metadata formats, the course structure and hierarchy occur in several versions, which concern the relation, the semantic meaning, the metadata and the web environment needs (Dodds 2001a, b). IMS Content Packaging The IMS Content Packaging specification (Anderson & McKell 2001) defines the content packaging format. In Fig. 1, a course was package as a zip format. In the zip file, an imsmanifest.xml file existed, which is an xml file used to express organization and resource. Most standards follow the format. It would easier to edit the learning resource with a XML parser s authoring tool for all kinds of content. e-learning environment E-Learning needs a good system to support online service, such as course management, student management, learning resource delivery, tracking service, integration of course and teaching methods and common format for exchange. ADL proposed SCORM, which defined a completed environment to support e-learning. In the Run-Time Environment, there are the API, data model, SCO, Asset, Launch mechanism and LMS. Assessment metadata for e-learning There are other similar e-learning standards for examinations. We made a simple comparison between them. The metadata in QTI is very complex. There are eight question types. A question s metadata have more than 20 attributes. There are no standards concerned with cognition. In addition, the examination and single question analysis are ignored. IMS QTI is the only standard that designs the result report. IMS QTI designs many metadata to be used by the user, such as the passing score, the minimum score and the mastery score. It also has different question types. ULF has less description of assessment. It focuses on the question types. But there are only a few question types. A special point is that ULF allows questionnaires. SCORM does not support assessment very much. In the metadata catalog [Educational], it describes the difficulty level, the semantic level, the interactivity level, the typical learning time. MINE SCORM Metadata Model provides the cognition level, discrimination, instructional sensitivity and difficulty, and gives feedback about Fig. 1 Content packaging conceptual diagram.

4 308 W.-C. Chang et al. Table 1. MINE SCORM, SCORM, IMS QTI and ULF Compared table (, completed; }, partial;, empty) Output format MINE SCORM SCORM ULF IMS Question & Test XML, SCORM 1.3 XML, SCORM 1.3 XML XML Metadata for assessment and exam analysis Difficulty } } Discrimination Distraction Instructional Sensitivity } Cognition single problem and examination analysis. It also supports many question types. In Table 1, we compared ADL SCORM, IMS QTI, ULF and our assessment metadata. It is obvious that our system has strong assessment analysis and examination authoring functions. The MINE SCORM Metadata Model We define an assessment metadata for e-learning in this section. Our work is based on SCORM. This assessment metadata is named the MINE SCORM Metadata Model. The MINE SCORM Metadata are designed specially for assessment in distance learning. It includes the assessment record, the assessment analysis, the questionnaire and the cognition level. The whole MINE SCORM Metadata are represented in a tree-like structure in Fig. 2. A description of MINE Metadata Information Model is also defined in Table 2. The Instruction objective is very important in teaching. If the instruction objective is clear, it guides teaching activities and evaluation precisely and properly. Another important issue for education is cognition. Assume the learning content, which the instructor prepared, is suitable for the learner. Then learning can be efficient. We quoted cognition level in our metadata model to make it provide cognition information for learning content. Bloom proposed the taxonomy of educational objectives in three domains: the cognitive domain, the psychomotor domain and the affective domain (Bloom 1956; Kropp et al. 1966; Seddon 1978; Fleishnan & Quaintance 1984). In the cognitive domain knowledge, comprehension, application, analysis, synthesis and evaluation are included. I. Knowledge: Remembering basic and previously learned information or lessons, such as knowledge of the learning subject, and knowledge of date, places or event. II. Comprehension: When people understand some concepts or information, they can translate the knowledge into new context. III. Application: Applying knowledge to real-life situations, such as using skills or knowledge to solve problems. IV. Analysis: Realizing concepts or ideas, and seeing structures or organizations. V. Synthesis: Using existing ideas or components to create new ones. VI. Evaluation: Based on some evidence to compare and analyse, and then make judgments. The analysis model in assessment system Complete teaching could be divided into three parts. The first part is teaching strategy (e.g., game, direct, discussion and experimentation). The second part is learning content (e.g., learning material or handbook, music score and textbook). The last part is assessment (e.g., questionnaires, tests, examinations and quizzes). Assessment plays an important role in this model. A teacher may use a proper teaching strategy and good learning contents to teach students. However, it is hard to know whether students learn well or not. The only way to understand their learning is to hold a test. With the analysis of the test result, the teacher may know how well the students learn, what the students need, and how the learning content can be improved. A good assessment analysis model provides a blueprint for teaching. Teacher Side: (1) Provide individual question statistics and analysis (2) Provide whole test statistics and analysis

5 Enhancing SCORM metadata e-learning 309 Knowledge Comprehension Cognition level Application Analysis Question style Synthesis Evaluation Essay True False Item Multiple Item Question Hint Question Hint Question Assessment Match Item Completion Item Questionnaire Answer Subject Hint Question Resumable DisplayType Random Fixed Individual test Item Difficulty Index Item Discrimination Index Distraction Exam Average Time TestTime Instructional Sensitivity Index Fig. 2 Proposed MINE SCORM assessment metadata model. Student Side: System Side: Item analysis: (1) Receive auxiliary tests for practices (2) Provide hint (1) Deliver auxiliary tests for practices (2) Deliver questionnaires to students and teachers (1) Provide feedback to students. (Student Side) (2) Provide feedback to teachers. (Teacher Side) (3) Provide the basis for improving learning content. (Teacher Side, System Side) (4) Improve the teachers formulate questions for a test or an exam. (Teacher Side) Individual question statistics and analysis in assessment system In this section, an individual question is the basic element of an examination. With analysis and statistics, the system can find the blind spot of an instructor by number representation and signal representation. Item Analysis with number representation Item Analysis is performed by comparing the proportion of learners who pass an item in contrasting criterion groups. The instructor can see each question s status. Also the model can provide suggestions from the test questions. Upper and lower criterion groups are selected from the extremes of the distribution. In a normal distribution, the optimum point at which these two conditions balance out is 27%. In 1939, Prof. Kelly indicated that the best percentage is 27%, and the acceptable percentage is 25 33% (Kelly 1939). We tried to define the percentage 25% in this paper and our system. (1) 1st Step: Sort the students order according to the students score in the exam. (2) 2nd Step: P H is defined as the higher 25% of total students and P L is defined as the lower 25% of total students. (3) 3rd Step: count a student s correct answers and his/her percentage in the higher group and the lower group of each question. (4) 4th Step: Calculate the item difficulty index for each problem P 5 (P H 1P L )/2 (5) 5th Step: Calculate the item discrimination index for each problem D 5 P H P L (6) 6th Step: The following shows the information format we record. No: The question s Number P H : the higher 25% of total student as the higher group P L : the lower 25% of total student as the lower group D 5 P H P L P 5 (P H 1P L )/2

6 310 W.-C. Chang et al. Table 2. MINE SCORM Metadata Information Model No. Name Explanation Multiplicity Data type 1 Assessment This category groups the assessment information that describes the resource as a whole 1.1 Cognitionlevel MINE assessment Vocabulary: Knowledge Comprehension Application Analysis Synthesis Evaluation 1.2 Questionstyle MINE assessment Vocabulary: Essay True False Item Multiple Choice Match Item Fill in Blank Essay Defines the text of an open-ended essay question. You can also use it to represent shorter fill-in-the blank TrueFalseItem Defines a question whose answer is either true or false MultipleChoice Defines a question with multiple choice answers MatchItem Define a question with proper matched choice CompletionItem Design a question like fill-in blank or close Questionnaire A list of questions that are answered by a number of people so that information can be collected from the answers: Question The content could be text, graph, drawing a picture. In this metadata, we focus on text Resumable True means resumed and false means paused at a later time Displaytype Fixed Order For tests with a fixed number and order of questions. Random Order For tests with a random order 1 and only 1 Container 0 or 1 VocabularyType (Restricted) 0 or 1 VocabularyType (Restricted) 1 or More (smallest permitted maximum: 10) 1 or More (smallest permitted maximum: 10) 1 or More (smallest permitted maximum: 10) 1 or More (smallest permitted maximum: 10) 1 or More (smallest permitted maximum: 10) 1 or More (smallest permitted maximum: 10) LangStringType (smallest permitted maximum: 2000 characters) LangStringType (smallest permitted maximum: 2000 characters) LangStringType (smallest permitted maximum: 2000 characters) LangStringType (smallest permitted maximum: 2000 characters) LangStringType (smallest permitted maximum: 2000 characters) LangStringType (smallest permitted maximum: 2000 characters) 1 and only 1 String (smallest permitted maximum: 1000 characters) O or 1 String (smallest permitted maximum: 1000 characters) 0 or 1 String (smallest permitted maximum: 1000 characters) 1.3 IndividualTest This category defines individual test. 1 and only 1 Container Answer Correct answer for explaining and querying 0 or More LangStringType (smallest permitted maximum: 1000 characters) Subject Define each question a main subject for learning material 0 or 1 String (smallest permitted maximum: 1000 characters) Item Difficulty Index Item Discrimination Index The index is suitable for large amount of people after test. Then calculate the value for reference The index is suitable for large amount of people after test. Then calculate the value for reference Distraction With the analysis to define the distraction of students RESERVED RESERVED RESERVED String String String

7 Enhancing SCORM metadata e-learning 311 Table 2. Continued No. Name Explanation Multiplicity Data type 1.4 Exam This category defines the whole 1 and only 1 Container class or group exam AverageTime The average time for testing. Each people take different time answering questions, we use average 1 and only 1 String (smallest permitted maximum: characters) time for operation Test Time A default time limit for testing 0 or 1 String (smallest permitted maximum: 1000 characters) Instructional Sensitivity Index The index is suitable for large amount of people after test. Then calculate the value for reference. With the comparison between the test result before teaching and the test result after teaching to analysis Instructional Sensitivity Index RESERVED String Table 3. Problem attribute Option A Option B Option C Option D Option E High-score group HA HB HC HD HE Low-score group LA LB LC LD LE Item analysis with signal presentation It is difficult for people to understand the Item Analysis with number. We use signal presentation instead of number presentation. However, it is not easy to identify the status. We defined four rules to show the analysis result automatic. In Table 3, item attributes of a single problem are defined. HA, HB, HC, HD and HE are the number of students in the high-score group who select answer A, B, C, D and E, respectively. LA, LB, LC, LD and LE are the number of students in the low-score group who select answer A, B, C, D and E, respectively. A common example would be used in the four rules. Assume that the number of students in the high-score group is 20, and the number of students in the low-score group is also 20. Four rules are defined to analyse the problems. These rules are introduced in the following. Rule 1: If ðlajlbjlcjldjleþ ¼0; then the answer s allure is low. Assume that Answer C did not attract any student in the low-score group. So, when ðlajlbjlcjldjleþ ¼0, then we can find that the answer s allure is low. Rule 2: N 5 {A, B, C, D, E} If Answer N is correct and HNoLN, then the answer is not well-defined. If Answer N is wrong and HN4LN, then the answer is not well defined. It supposed to be the number of high-score group should answer correct. If the number of the students who choose correct answer in the low-score group is greater than the number of the students in the highscore group. Rule 3: LM ¼ MAX ðla; LB; LC; LD; LEÞ Lm ¼ min ðla; LB; LC; LD; LEÞ LS ¼ LA þ LB þ LC þ LD þ LE If jlm Lmj6LS 20%, then students in the lowscore group lack of related concepts Rule 3 happened when the low-score group students choose every answer equally. This means that these students miss the concept of the problem. Rule 4: HM ¼ MAX ðha; HB; HC; HD; HEÞ Hm ¼ min ðha; HB; HC; HD; HEÞ HS ¼ HA þ HB þ HC þ HD þ HE If jhm Hmj6HS 20%

8 312 W.-C. Chang et al. Then students in the high-score group lack of related concepts Rule 4 is similar to Rule 3, and the two rules can detect if students learn the knowledge or not. From the rules we can identify the status of the test (Table 4). The information is useful for correcting improper questions in the examination. It is also useful for the instructor to know the students learning. Table 4 shows the signal representation status. Let s see an example. Assume that the class size is 44 students, and the high-score group and the lowscore group are both 11. Taking the example of no.2 question, the correct answer is Answer C. People in high-score group selected as follows: Answer A, 0 person, Answer B, 0 person, Answer C, 10 persons and Answer D, 1 person. People in lowscore group selected as follows: Answer A, 3 persons, Answer B, 2 persons, Answer C, 4 persons and Answer D, 2 persons. With the item analysis, we can get the following information. P H ¼ 10=11 ¼ 0:91 P L ¼ 4=11 ¼ 0:36 D ¼ P H P L ¼ 0:91 0:36 ¼ 0:55 > 0:3: The signal will show according to Table 4. P ¼ðP H þ P L Þ=2 ¼ð0:91 þ 0:36Þ=2 ¼ 0:635: The problem does not match Rules 1 4. However, if the D (P H P L ) is lower than 0.19, the problem needs to be eliminated or fixed. Total test statistics and analysis in assessment system The assessment analysis should be presented in different aspects. A total test analysis result could show the whole status of students. p Figure representation (1) Time (cross axis) and Number of answered question (vertical axis) figure: This figure shows if the test time is enough or not. (2) Test score (cross axis) and degree of difficulty (vertical axis) figure: This figure shows the distribution of scores and difficulty. (3) Cognition level (cross axis) and learning content subjects (vertical axis). This figure shows the cognition level, the question number and the subject (See Table 5). p Definition (1) The cognition that Bloom proposed is mapping from A to F. For example, knowledge is A and comprehension is B. Assume that X is a universal set, and X 5 {A, B, C, D, E, F}. (2) Concepts in the test are named from 1 to I, e.g., Concept 1. (3) From Concept 1, a question that belongs to the Knowledge cognition level is designed. A1 is set to [TRUE] if more than one question that belongs to the Knowledge cognition level exist in Concept 1. A1 is set to [TRUE] to represent that there is at least one question of the knowledge level in Concept 1. If A1 is [FALSE], there is no question of the knowledge level in Concept 1. (4) SUM(Xi) is the question s sum of the cognition level X in Concept i. Ex. SUM(F3) 5 3 means that there are 3 questions of the evaluation level in Concept 3. Table 4. Advice and suggestions about questions Status Light signal D Rule 1 Rule 2 Good Higher than 0.3 N/A N/A Fix Eliminate or fix Lower than 0.19 N/A N/A Table 5. Two-way specification table Knowledge Comprehension Application Analysis Synthesis Evaluation Concept 1 A1 B1 C1 D1 E1 F1 SUM(A1 F1) Concept i Ai Bi Ci Di Ei Fi SUM(Ai Fi) SUM(A1 Ai) SUM(B1 Bi) SUM(C1 Ci) SUM(D1 Di) SUM(E1 Ei) SUM(F1 Fi)

9 Enhancing SCORM metadata e-learning 313 (5) SUM(Ai Fi) is the question s sum in Concept i. Ex. SUM(A10-F10) 5 8: there are 8 questions (From Knowledge to Evaluation level) in Concept 10. (6) SUM(X1-Xi) is the question s sum of the cognition level X (From Concept 1 to Concept i). Ex. SUM(C1 C7) 5 7: there are 8 questions from Concept 1 to Concept 7. p Analysis (1) Concept Lost If ða1jb1jc1jd1je1jf1þ ¼FALSE, Concept 1 is lost in the examination. (2) Cognition level and question s sum relation SUM(A1 Ai)^SUM(B1 Bi) ^SUM(C1 Ci) ^SUM(D1 Di) ^SUM(E1 Ei)^SUM(F1 Fi) (3) Distribution of cognition levels and question (Paint algorithm, see Table 6) With the analysis result, we can divide into several results for suggestions to instructors. In Table 7, there are six types of results. With students of different ages and cognitive levels, instructors had distinct assessment and examination styles. With the analysis feedback, instructors can correct their exam. In Table 7A, test focuses on knowledge, comprehension and application cognition levels. In Table 7B, test focuses on analysis, synthesis and evaluation cognition levels. In Table 7C, test loses some concepts. The test key point is not the same as the teaching key point. In Table 7D, test partially emphasizes some concept. And the test is inclined to high cognition levels. In Table 7E, test partially emphasizes some concept. And the test is inclined to low-cognition levels. In Table 7 (F), test s key point is too distributed for students. There is no real key point in this test. The architecture of the assessment authoring system The assessment authoring system architecture has several main components (Fig. 3). There are two databases. One is the internal problem and examination database, and another is the SCORM compatible external repository. The system includes problem search, examination authoring, problem authoring and SCORM format output service. Also there is an online examination monitoring subsystem for capturing pictures. Table 6. Paint algorithm procedure

10 314 W.-C. Chang et al. Table 7. Several suggestion type of distribution of cognition level and question (paint algorithm) Fig. 3 System architecture of the assessment authoring system. Authors, instructors and tutors can use the assessment authoring system to edit problems or examinations. They can search similar or specific subjects for or related problems from the problem and examination database. In addition to searching the database, the instructor may edit the problem and examination themselves. After authoring the problems, instructors can combine their own problems with the problems from the database. In order to share the material relating to problems and examinations, our system provides a SCORM format package output service. The service can package the original problem and examination files into SCORM compatible files. Other instructors may reuse the problem and examination files from the SCORM compatible external repository. The administrator controls the database and learning management (LMS) monitor function. Learners take the examination or the problems with the Internet browser. When learners take the exam, the monitor function captures the client picture for monitoring the examination progress. Problem types Problem authoring provides several problem types, and there are choice problems, fill-in blank problems and true-false problems. Problem metadata and content Problem attributes in our system have two parts, one is the metadata information, and the other is problem content. The metadata information is used to describe the object.

11 Enhancing SCORM metadata e-learning 315 Problem template A picture can be put in a problem. It is permissible to set the picture s position. Besides, we can set the question description and question selection items. We can set the presentation style by simply moving each item. When the instructor edits a problem, he/she might want to save the problem structure for reuse. A new template can be added. Also, an existing template can be deleted. Problem search and query The instructor can search the problem using the metadata. For example, an instructor wants to query [h.c.f. (highest common factor)] the concept in Math. The problem type is Multiple Choice. He/She can set the problem search as the following: Subject: Math; Keyword: h.c.f.; Problem type: Multiple Choice. Examination output compatible with SCORM format In the SCORM standard, each file has a descriptive XML file with the same level in the course structure. In addition to these descriptive XML files, a main description is also an XML file called imsmanifest.xml. With imsmanifest.xml, we can parse the whole course structure. Furthermore, Java script files to communicate with API and the learning management system are necessary following the SCORM standard. Without these Java scripts, the learning management cannot find the API to communicate with. Some API functions are used to set values (e.g., learner record, learner progress, learner status), get values, handle errors (e.g., error message transfer, error status record, error dialog) and course beginning and ending (e.g., course initial and course finish). To generate the imsmanifest.xml file, the parser has to pass through the whole structure. The traversal algorithm gathers the needed information to build a tree structure of this course. To insert the Java script file into a proper position of a tree structure, our system would insert two Java script files. One of the java script files is used to communicate with the learning management system with an API adapter. The other is used to find the API window that is responsible for SCO to initialize the LMS. Conclusion In this paper, we propose the MINE Assessment Metadata to support e-learning assessment. The assessment metadata strengthen the SCORM counter part. The elements are simple but effective for analysis of test results. They are also important for guaranteeing teaching quality. The analysis result can tell the instructor if the teaching goal is achieved. Also, the student can understand the key point of the course and the blind spot of the learning process. The files produced by the system are compatible with the SCORM standard and the authoring concept is also referenced to IMS QTI. Besides, the system provides a monitor function for capturing the learner s picture during an examination. Defining the metadata is the most difficult part of using the metadata model and the assessment system. In e-learning standards, there are too many metadata. Only required metadata attributes are added in our assessment system. For the future work, we can focus on the following aspects: firstly, extending the question types to support most assessment system and also, strengthening interaction for assessment and multimedia assessment; secondly, combining the assessment metadata to learning content; thirdly, establishing an automatic question and answering system for the instructor. Acknowledgements We will like to thank H.C. Chen and K.H. Huang for implementating our system. Without their work, our assessment would not have been tested. This project was supported by NSC Taiwan under grant number NSC S References Anderson Y. & McKell M. (2001) IMS Content Packaging Information Model. HTML format. Available at: cpv1p1p2/imscp_infov1p1p2.html [last accessed 20 November 2003]. Bloom B.S. (1956) Taxonomy of Educational Objectives: The Classfication of Educational Goals: Handbook 1: Cognitive Domain. Longman, New York, Available at: Dodds P. (2001a) Sharable Content Object Reference Model (SCORM). Version 1.2. Technical report, The SCORM Content Aggregation Model, Advanced Distributed Learning Initiative, October 1. Available at:

12 316 W.-C. Chang et al. Down [last accessed 20 November 2003]. Dodds P. (2001b) Sharable Content Object Reference Model (SCORM). Version 1.2. Technical report, The SCORM Run-Time Environment, Advanced Distributed Learning Initiative, October 1. Available at: [last accessed 20 November 2003]. Fleishnan E.A. & Quaintance M.K. (1984) Taxonomies of Human Performance: The Description of Human Tasks. Academic/Harcourt Brace Jovanovich, Orlando, FL. Hodgins W. (2001) Draft Standard for Learning Object Metadata (LOM) Specification. Proposed Draft 6.1, Institute of Electrical and Electronics Engineers, Inc. Electronic version available at files/lom_1484_12_1_v1_final_draft.pdf [last accessed 20 November 2003] Kelly T.L. (1939) The selection of upper and lower groups for the validation of test items. Journal of Educational Psychology 30, Kropp R.P., Stocker H.W. & Bashaw W.L. (1966) The validation of the taxonomy of educational objectives. Journal of Experimental Education 34, Seddon M. (1978) The properties of bloom s taxonomy of educational objectives for the cognitive domain. Review of Educationa Research 48,