A Knowledge Scoring Engine (KSE) for Real-Time Knowledge Base Generation Used in Intelligent Tutoring Systems

Transcription

1 A Knowledge Scoring Engine (KSE) for Real-Time Knowledge Base Generation Used in Intelligent Tutoring Systems Mark T. Jodlowski Mississippi State University Stephanie M. Doane Mississippi State University Abstract This paper provides a detailed account of a tool developed to facilitate the development of individual computational cognitive models to be used in intelligent tutoring systems (ITS). A generic tool that we will call the Knowledge Scoring Engine (KSE) reads eye tracking and human performance data in real-time, parses the data into discrete events, and uses a knowledge inference generator to build and update student knowledge bases. Student knowledge bases are generated by interpreting actions in realtime based on the context in which the actions take place. The knowledge bases generated by KSE are used by ADAPT, a comprehension-based framework construction-integration model of pilot action planning. This paper provides a detailed description of KSE, the current ADAPT architecture, how the ADAPT model interacts with KSE, and the potential for future developments. 1. Introduction The effort required to build knowledge bases when modeling individuals is one of the many challenges of developing computational cognitive models for intelligent tutoring systems (ITS). The effort is even more difficult when knowledge bases are built in real-time while the modeled individual is acting and reacting in a dynamically changing environment (e.g., piloting an aircraft). Such a task is part of an ongoing project in our laboratory, in which our goal is to develop a real-time ITS of instrument flight skills. Current tutors of complex task performance use automated knowledge base tools. For example, PAT (Pittsburgh Urban Math Project Algebra Tutor) generates a student knowledge base by estimating the probability that the student knows a particular rule [1]. The probability that a student has knowledge of a particular rule is based on an examination of a student s action when the student is given a chance to use a specific rule. Future Combat System (FCS) training [2] demonstrates an alternative method. The FCS uses a series of questions and the student provides answers to assess the student s knowledge and tailor the training to the individual. In the examples above, student knowledge acquisition does not occur in real time or in a dynamically changing environment. Adding a realtime component provides another level of complexity. To generate a student knowledge base in real time, access to real-time, time-synchronized performance data is required. These data provide an ITS the capability of generating and continually updating the student model knowledge base. Our tool generates and updates student knowledge bases by interpreting actions in real time based on the context in which the actions take place. Actions are defined as pilot control and eye movements during simulated flight maneuvers. This type of knowledge base generation is sensitive to dynamically changing environments. For example, in the aviation domain, the context in which a student performs an action is crucial to determining the implication of that action for the student model. In addition, even while the student is performing the action, the environment is changing and the implication of the student s next action is affected by the changing context. Accurate knowledge base generation and continual updating is critical to the success of our project. The overall goal is to demonstrate how an integrative comprehension-based cognitive architecture s learning mechanisms act as a student model of real-time complex task performance in a dynamic task environment. Additionally, the project involves an effort to improve the training of real-time dynamic task performance. To accomplish these goals, we need to develop tools that will not only help us reach our goals, but will augment modeling efforts in general. The focus of this paper is on a Knowledge Scoring Engine (KSE) developed to generate individual knowledge bases in real-time for /04 $17.00 (C) 2004 IEEE 1

2 computational cognitive models of complex dynamic task performance. In addition, the paper focuses on KSE s integration with ADAPT, a constructionintegration model of cognition. 2. KSE Overview KSE s first task is to time synchronize incoming data (e.g., eye movements, flight maneuvers, and flight situation). This aspect of KSE is most closely tied to the ITS environment. The second task is to parse the continuous flow of data into meaningful discrete event segments (discrete event segments are meaningful units of information provided by the system and user). In our case, an event is a user action (either eye movements or control movements) combined with the current flight situation data. The current flight situation data (e.g., airspeed, heading, and altitude) provide contextual information about the environment in which the user performs the action. Tools that classify data into events are not new, and many are capable of reading and parsing real-time data. These tools look for numerical changes in the input data and parse the data according to the detected changes. Similar to other tools, KSE contains a parser that breaks the incoming data into discrete events. However, the next process is unique to KSE, which was developed as a general tool that makes inferences based on the parsed discrete event data. The inferred knowledge is added to an individual s knowledge base as a function of the parsed discrete events. That is, in our parser, events are always interpreted as a function of the context in which they occur. In addition, the inferences regarding knowledge can only be drawn in conjunction with the environmental context. What follows is a detailed description of KSE and ADAPT. After describing both components, an integrated working example of KSE and ADAPT will be discussed. 3. Knowledge Scoring Engine KSE is a generic tool that can be easily modified and used by any experimenters who design their computational models using LISP. Although our implementation of KSE is focused on creating declarative and procedural knowledge bases for ADAPT [3], a construction-integration model, the system could be adapted readily to other architectures, such as ACT-R. As can be seen in Figure 1, KSE consists of two components and a resulting knowledge base. The first component is a parser that translates incoming real-time data. The second component is a knowledge inference generator based on predefined Figure 1. Flow of data into the Knowledge Scoring Engine and out to an individual knowledge base. condition action rules provided by the experimenter. The knowledge inference generator creates and updates an individual s knowledge base. Each of these components will be discussed in the following sections Parser The parser is designed to examine the incoming flow of both static and real-time data, which can enter KSE from many sources. Some examples of incoming data are eye movements captured by an oculometer, flight stick controls, and performance data provided by a flight simulator. As the real-time data enters KSE, it is parsed into predefined discrete event segments, defined as the combination of the user action data and environmental data entering the system. For example, if eye-tracking data are collected while an individual is flying a flight simulator, the combination of the oculometer data and the current flight situation (e.g., airspeed, heading, and altitude) would be a discrete event (see Figure 2). The control movement data combined with the current flight situation would be considered a separate discrete event. The discrete events are then stored in a time-sensitive buffer. The current parser requires the experimenter to hard code combinations of elements (e.g., eye fixation combined with current flight situation) to allow KSE to identify events. In other words, the number of events that can occur is infinite, but constrained by the predefined elements. KSE is currently coded to identify three element combinations: pilot flight performance, actions, and eye movements. Although it was designed specifically for interacting with a flight /04 $17.00 (C) 2004 IEEE 2

3 Figure 2. Sample discrete event (combination of flight simulator and eye movement) data. simulator, the parser is generic enough to adapt easily to other types of environments (e.g., driving simulators) Knowledge Inference Generator The knowledge inference generator is a KSE component that creates and continually updates individual knowledge bases. An individual s knowledge base is determined by checking events against experimenter-defined condition action rules written in LISP. An example of a condition action rule for oculometer and flight simulator data would be: IF the goal is to decrease airspeed And the pilot looks at the airspeed indicator And the pilot pulls back the throttle THEN update the pilot s knowledge base to indicate that the pilot knows that decreasing the throttle decreases airspeed. Taken together, the predefined condition action rules and discrete events stored in the time-sensitive buffer generate and update an individual s knowledge base while the individual is performing the task. The process continues as long as data continue to enter KSE. To summarize, the discrete events are compared to the knowledge inference rules that reside in the knowledge inference generator. If a discrete event matches a rule in the knowledge inference generator, the associated knowledge updates the individual s knowledge base Tutoring Instrument Flight Skills As noted previously, KSE was developed to help transform a cognitive theory-driven constructionintegration model of pilot action planning during simulated flight (ADAPT) into a real-time intelligent tool for tutoring instrument flight skills. Given access to real-time control movement, eye movement, and flight situation data, the model uses KSE to build and continually update an individual s knowledge base. Given the individual s knowledge base, the ADAPT model has been shown to be capable of predicting individual novice, intermediate, and expert pilot eye fixations and control movements during simulated flight [3]. In addition, ADAPT has learning mechanisms to acquire both procedural and declarative knowledge. These mechanisms have recently been found to be capable of predicting individual student learning from a computer-based tutor for computer command skills [4]. We will now provide a history of the ADAPT architecture, the current ADAPT architecture, and how ADAPT uses KSE to generate a pilot s knowledge base. 4. ADAPT ADAPT is a comprehension-based framework construction-integration model of pilot action planning. We will now describe the constructionintegration theory and architecture of ADAPT /04 $17.00 (C) 2004 IEEE 3

4 Figure 3. ADAPT construction-integration model Construction-Integration Theory ADAPT is based on the premise that comprehension-based mechanisms identical to ones used to understand a list of words, narrative prose, and algebraic word problems constrain problem-solving episodes as well. ADAPT rests on Kintsch s [5, 6] construction-integration theory of comprehension. Specifically, Kintsch s theory presumes that low-level associations between incoming contextual information (e.g., task instructions) and background knowledge (e.g., domain knowledge) are constructed and used to constrain knowledge activation via a constraint-based integration process. The resulting pattern of contextsensitive knowledge activations is referred to as a situation model and represents the current state of comprehension. The construction-integration model [5] was initially developed to explain certain phenomena of text comprehension, such as word sense disambiguation. The model describes how we use contextual information to assign a single meaning to words with multiple meanings. For example, the appropriate assignment of meaning for the word bank is different in the context of conversations about paychecks (money bank ) and about swimming (river bank). In Kintsch s view, this can be explained by representing memory as an associative network in which the nodes contain propositional representations of knowledge about the current context or task: general (context-independent) declarative facts, and If/Then rules that represent possible plans of action [7]. The declarative facts and plan knowledge are similar to the declarative and procedural knowledge contained in ACT-R [8]. When the model simulates comprehension in the context of a specific task (e.g., reading a paragraph for a later memory test), a set of weak symbolic production rules construct an associative network of knowledge interrelated by superficial similarities between propositional representations of knowledge without regard to task context. This associated knowledge network is then integrated via a constraintsatisfaction algorithm that propagates activation throughout the network, strengthening connections between items relevant to the current task context and inhibiting or weakening connections between irrelevant items. This integration phase results in context-sensitive knowledge activation constrained by inter-item overlap and current task relevance. Kintsch s theory in particular has been used to explain a wide variety of behavioral phenomena, including narrative story comprehension [5], algebra story problem comprehension [5], the solution of simple computing tasks [7], and completing the Tower of Hanoi task [9]. This approach has also proved fruitful for understanding human-computer interaction skills [e.g., 10, 11, 12], predicting the impact of instructions on computer-user performance [13, 14], and predicting novice and expert pilot action planning [3] ADAPT Architecture Figure 3 displays the current architecture of ADAPT. The flow of cognition in the system is in response to the current goal and current state of the world (perceived environment), currently active information in declarative and plan element /04 $17.00 (C) 2004 IEEE 4

5 knowledge, information attended to in perceptual models (visual and auditory), and the current state of the motor modules (hand and speech). Initially, in response to the constructed state of all of these modules, knowledge and action are integrated and a plan element is selected and executed. This plan element can cause changes in the current goal, make retrieval requests of declarative memory, shift attention, or call for new motor actions. The action modules can cause changes in the simulated environment (flight simulator that uses algorithms to update the state of the aircraft (e.g., heading, airspeed, altitude) in response to actions and to the passage of time). The environmental changes are perceived by the perceptual modules. ADAPT follows the lead of Anderson [14] and uses a serial bottleneck theory of cognition in which parallel cognitive, perceptual, and motor modules must interact through a serial process of plan element selection and execution. ADAPT currently performs actions, looks-at displays, and moves flight controls, but it does not possess a perceptual motor model as sophisticated as that in EPIC [16]. ADAPT represents the three major classes of knowledge proposed by Kintsch [5]: world knowledge, general knowledge (e.g., declarative facts), and plan element knowledge (e.g., procedural knowledge represented as If/Then rules). Table I shows examples of each class of knowledge along with the abbreviated propositional representation in ADAPT. Each of the three types of knowledge is discussed below World Knowledge. The first class of knowledge represents the modeled pilot s current state of the world. Examples of world knowledge in ADAPT include the pilot s knowledge of the current and desired states of the airplane, determined relationships between the current and desired states (e.g., altitude is higher than desired value), and flight segment goals. Airplane status is represented as the current values, rates of change, and direction of change of each flight axis. Current values represent the current status of the airplane shown on display instruments (e.g., 3000 ft); direction of change represents increase, decrease, or holding axis status; and rate of change represents the speed of change to an axis (e.g., 500 ft per minute). These facts are contextually sensitive and fluid, changing as the modeled task and simulated performance progresses through a flight maneuver. If the ADAPT pilot detects a change in the state of the airplane, then plane status is updated, and the relationships between current and desired plane states recalculated. In addition, if the ADAPT pilot receives new flight goals, then goals are updated in the modeled pilot world knowledge. It is noteworthy that ADAPT constrains the number of propositions that can be retained in world knowledge to model working memory limitations crudely. Working memory capacity limitations are modeled by limiting the number of propositions retained in world knowledge from cycle to cycle using a predetermined capacity threshold. Decay limitations are modeled by limiting the length of time (modeled as cycles ) a given in-the-world proposition can be retained in memory using a predetermined decay threshold General Knowledge. General knowledge refers to factual (declarative) knowledge about piloting (see Table I). In ADAPT, general knowledge represents facts about the relationships between control inputs and plane performance, and knowledge of flight dynamics, display instruments, and control movements Plan Elements. Plan elements represents executable (procedural) knowledge about piloting. Plan elements describe actions that can be taken inthe-world, and they specify conditions under which actions can be taken. Thus, pilots have conditionaction rules that they can consider and execute if conditions are correct. Plan elements are three-part knowledge structures that include a name, preconditions in-the-world or general knowledge that must be satisfied in order for the plan to fire, and outcomes that are added to world knowledge if the plan is executed (see Table I). For example, as shown in Table I, a plan element that decreases airspeed requires that the pilot know that the desired airspeed is less than the current airspeed, that power controls airspeed, that the tachometer indicates power, and that pulling back the throttle decreases power. When the decrease plan element is fired, its outcome propositions are added to the world knowledge to represent the current need to look at the tachometer and to pull back the throttle. 5. KSE and ADAPT Given the knowledge intensive nature of the ADAPT model, it is clear that a tool to simplify knowledge acquisition would facilitate its use. One of our main goals is to generate and update an individual s knowledge base real time in a complex dynamically changing environment. ADAPT uses KSE to generate and update the knowledge base of each individual pilot (see Figure 4). The knowledge /04 $17.00 (C) 2004 IEEE 5

6 Table I. Examples of knowledge and their formal representations in the ADAPT model. Type of Knowledge Abbreviated Propositional Representation World Knowledge Desired airspeed is 90 kts Current airspeed is 100 kts Desired airspeed is much less than current airspeed A task is to check airspeed General Knowledge Control-performance relationship Power controls airspeed Flight dynamics Pitch up causes airspeed decrease Primary-supporting display VSI supports altimeter Display instrument Airspeed indicator indicates airspeed Control movement Pulling back throttle decreases power Plan Knowledge Name: Decrease airspeed Preconditions: Desired airspeed is less than current airspeed Airspeed indicator indicates airspeed Power controls airspeed Pulling back throttle decreases power Outcome(s): Need to look at airspeed indicator Need to pull back throttle base is generated as the knowledge inference generator acts on the discrete events identified in an individual s performance. Recall that the discrete events are combinations of eye fixations or control movements in the context of the flight situation. The knowledge engine contains the predefined rules for declarative knowledge and plan elements. As discrete events enter the buffer, the inference generator reads the discrete events and matches them to the declarative knowledge and plan element knowledge stored in the inference generator. If the discrete event matches a rule stored in the inference generator, then the knowledge is inferred and added to the pilot s knowledge base. Before describing in detail an example of ADAPT s use of KSE, it may be helpful to review the data flow in Figure 1. The example described will follow in serial order the flow of data from start to finish. Imagine a scenario in which the pilot knows he/she needs to decrease the airspeed of the plane and maintain both a constant heading and altitude. For individuals not familiar with aviation, the change (Know airspeed^desired=90) (Know airspeed^current=100) (Know much^lt-airspeed^desired-airspeed^current) (Task check airspeed) (Know power control airspeed) (Know pitch up airspeed decrease) (Know VSI support altimeter) (Know airspeed^indicator indicate airspeed) (Know pull^back throttle decrease power) (Do decrease airspeed) (Know lt-airspeed^desired-airspeed^current) (Know airspeed^indicator indicates airspeed) (Know power control airspeed) (Know pull^back throttle decrease power) (Need airspeed^indicator) (Need pull^back throttle) required to reach the goal is simply to slow down. The current flight situation indicates that the plane is traveling at 90 knots (airspeed), flying at 3000 feet (altitude), and has a heading of 270 (West). If the pilot fixates on the tachometer (the instrument indicating the engine RPMs), the real-time data that enter KSE are the flight situation data and the pilot s fixation. Fixation on the tachometer is one of the predefined events in KSE. The combination of the flight situation data (airspeed, altitude, and heading) and the fixation data are parsed into a discrete event. This event is then entered into the time-dependent buffer. The knowledge inference generator acts on the event in the buffer and tries to match the data stored in the buffer with rules stored in the knowledge inference generator. Meanwhile, the pilot has performed two actions (started decreasing the throttle and stopped his/her adjustment) and is now fixating on the airspeed indicator. Three more discrete events are added to the buffer: decreasing the power (using the throttle) combined with the current flight situation data; /04 $17.00 (C) 2004 IEEE 6

7 Figure 4. Integration of the Knowledge Scoring Engine and ADAPT intelligent tutoring system. stopping the control movement (no longer decreasing the throttle); and fixating on the airspeed indicator, as always combined with the current flight data. The compilation of events stored in the buffer now matches one of the predefined rules in the knowledge inference generator. IF the goal is to decrease airspeed AND the throttle is pulled back AND there is a fixation on the airspeed indicator THEN infer that the pilot knows that decreasing the throttle decreases the airspeed The individual pilot knowledge base is then updated with the new knowledge (propositional representation of the student). Updating the knowledge base continues as data continue to enter KSE. This allows the system to not only identify prior knowledge but also to identify knowledge learned by the student while interacting with the ITS. ADAPT can then use the new knowledge in the construction and integration phases Time Savings Before the development of KSE, it took approximately 2 hours to hand score an individual student knowledge base using performance data from seven, 6-second windows (42 seconds) of simulated instrument flight (for a complete description of the performance data windows see [3]). There is a significant timesaving benefit when using KSE. Because KSE functions in real-time, KSE can generate the same knowledge base in 42 seconds. When KSE runs in an off-line environment, each 6-second window of performance data takes less than a second to process Validation One method of validating cognitive models is to compare modeled and student performance. In a previous study [3], using hand-scoring and 42 seconds of performance data to generate a knowledge base, the ADAPT model matched approximately 80% of pilot behavior. We have performed a preliminary validation of ADAPT using KSE rather than hand scoring of knowledge bases. Using a small sample size (N = 4) and approximately 42 seconds of performance data, the match between modeled and student performance ranged from 72 90%. These preliminary results are consistent with the match obtained using hand scoring, which suggests that KSE is just as accurate as hand scoring. Furthermore, it appears that KSE is more sensitive to the identification of pilot knowledge; i.e., KSE is more likely to identify all pilot knowledge for a given window of data. Future work involves identifying the appropriate sensitivity level for KSE /04 $17.00 (C) 2004 IEEE 7

8 6. General Discussion There are many possible benefits to using KSE. Perhaps the most important advantage is its ability to generate and continually update individual knowledge bases for a complex dynamic task in real time. This capability provides a necessary component for intelligent tutoring systems designed to model individual students. Although KSE was developed to work in real time, it is also able to generate knowledge bases from saved data files. The data files can be read as input files, and KSE will generate an individual knowledge base reflecting the data provided to it. The generic nature of the rule interpretation allows an experimenter to input rules into the knowledge inference generator and quickly assess student knowledge. The experimenter does not have to manually examine the parsed data and generate knowledge bases for each subject. Another benefit of KSE is that it was designed to be a generic tool that can be used in any type of environment that uses LISP. As with any tool, there are restrictions, however. In KSE, the primary restriction is the necessity of developing an interface between the tutoring environment and the timesensitive event buffer. In our current application, the identification of events is closely tied to the contents of our real-time data. A second restriction is the requirement that the experimenter code the condition action rules that infer knowledge from actions in a LISP interpretable format. However, in the future, we hope to develop a friendly interface that can be used for entering rules. As previously mentioned, KSE was developed to parse oculometer, flight simulator, and control movement data and to generate individual knowledge bases for piloting an aircraft. The type of events KSE parses can be modified using LISP. In addition, the knowledge inference rules can be defined in LISP as well. Future development of KSE will provide a userfriendly interface to define events and specific knowledge inference rules. Overall, we feel that KSE shows promise as a tool for use in computational modeling and that KSE is capable of supporting the critical goal of developing situation specific models [17]. KSE allows knowledge bases for complex and dynamically changing tasks to be generated and updated both quickly and accurately based on contextually sensitive experimenter-defined rules. In addition, there is no reason KSE cannot be adapted for various types of data in multiple domains and for various cognitive modeling architectures. 7. References [1] Corbett, A.T., K.R. Keodinger, and J.R. Anderson, Handbook of Human Computer Interaction (2 nd ed.), Elsevier Science, Amsterdam, [2] G.M. Burmester, D. Stottler, and J.L. Hart, Embedded Training Intelligent Tutoring Systems (ITS) for the Future Combat Systems (FCS) Command and Control (C2) Vehicle. Proceedings of the Industry/Interservice, Training, Simulation & Education Conference, National Academies Press, Washington DC, [3] S.M. Doane and Y.W. Sohn, ADAPT: A Predictive Cognitive Model of User Visual Attention and Action Planning, User Modeling and User Adapted Interaction, 10, Kluwer Academic, Dordrecht, Netherlands, 2000, pp [4] Y.W. Sohn, and S.M. Doane, Evaluating Comprehension-Based User Models: Predicting Individual User Planning and Action, User Modeling and User- Adapted Interaction, 12, Kluwer Academic, Dordrecht, Netherlands, 2002, pp [5] W. Kintsch, The Use of Knowledge in Discourse Processing: A Construction-Integration Model, Psychological Review, 95, American Psychological Association, Washington, DC, 1988, pp [6] Kintsch W., Comprehension: A Paradigm for Cognition, Cambridge University Press, Cambridge, MA, [7] S.M. Mannes, and W. Kintsch, Routine Computing Tasks; Planning as Understanding, Cognitive Science, 15(3), Elsevier Science, Amsterdam, 1991, pp [8] Anderson, J.R., Rules of the Mind, Erlbaum, Hillsdale, NJ, [9] F. Schmalhofer, and B. Tschaitschian, The Acquisition of a Procedure Schema from Text and Experiences. Proceedings of the 15th Annual Conference of the Cognitive Science Society, Erlbaum, Hillsdale, NJ, 1993, pp [10] S.M. Doane, D.S. McNamara, W. Kintsch, P.G. Polson, and D. Clawson, Prompt Comprehension in UNIX Command Production, Memory and Cognition, 20(4), Psychonomic Society, Austin, TX, 1992, pp [11] M. Kitajima, and P.G. Polson, A Comprehension- Based Model of Correct Performance and Errors in Skilled, Display-Based, Human-Computer Interaction, International Journal of Human-Computer Studies, 43, Elsevier Science, Amsterdam, 1995, pp /04 $17.00 (C) 2004 IEEE 8

9 [12] S.M. Mannes, and S.M. Doane, A Hybrid Model of Script Generation: Or Getting the Best of Both Worlds, Connection Science, 3(1), Taylor and Francis Group, Philadelphia, PA, 1991 pp [13] S.M. Doane, Y.W. Sohn, D.S. McNamara, and D. Adams, Comprehension-Based Skill Acquisition, Cognitive Science, 24, Elsevier Science, Amsterdam, 2000, pp [14] Y.W. Sohn, and S.M. Doane, Cognitive Constraints on Computer Problem Solving Skills, Journal of Experimental Psychology: Applied, 3, American Psychological Association, Washington, DC, 1997, pp [15] Anderson, J.R., and C. Lebiere, The Atomic Components of Thought, Erlbaum, Mahwah, NJ, [16] D.E. Meyer, and D.E. Kieras, A Computational Theory of Executive Cognitive Processes and Multiple-Task Performance: Part 1. Basic Mechanisms, Psychological Review, 104, American Psychological Association, Washington, DC, 1997, pp [17] Pew, R.W., and A.S. Mavor, Modeling Human and Organizational Behavior: Application to Military Simulations. National Academy Press, Washington D.C, Acknowledgments This research was supported by a grant from the Office of Naval Research #N The authors wish to thank Dr. Susan Chipman for her support of this effort /04 $17.00 (C) 2004 IEEE 9