Intelligent Visualization and Exploration of Time-Oriented Clinical Data

Transcription

1 Intelligent Visualization and Exploration of Time-Oriented Clinical Data Yuval Shahar and Cleve Cheng Stanford Medical Informatics 251 Campus Drive, MSOB X-215, Stanford University, Stanford, CA , USA tel: ; fax: Abstract We describe a conceptual architecture and software implementation specific to the task of interpretation, summarization, visualization, explanation, and interactive exploration of time-oriented clinical data and the multiple levels of meaningful concepts that can be abstracted from these data. We build on our work on abstraction of timeoriented clinical data using a knowledge base, acquired from clinical experts, of temporal properties of the data. We call the new framework KNAVE (Knowledge-based Navigation of Abstractions for Visualization and Explanation). The visualization and exploration operators, whose semantics are domain independent, access the domain-specific knowledge base. Exploration exploits key relations (e.g., the abstraction hierarchy) in each clinical domain. Preliminary assessment of the prototype with several clinical users has been encouraging. The KNAVE methodology has broad ramifications for reducing the load that large numbers of time-oriented clinical data put on practicing physicians. 1. Temporal abstraction and information visualization In this paper, we present a conceptual and computational knowledge-based framework for interactive summarization, visualization and exploration of timeoriented clinical data and their multiple levels of temporal abstractions (domain-specific, meaningful temporal patterns). Thus, we solve the problem of overloading the attending physician s capability for handling large numbers of clinical individual-patient data by summarization of these data (typically raw data, measured at particular time points) into temporal abstractions (which typically characterize time intervals). These abstractions can be readily visualized graphically in response to specific queries by the physician. We then enable the physician to explore the resultant abstractions and raw data using tools that have access to the same knowledge that gave rise to the derived abstractions (as well as to the patient record). These tools, as we show, are not specific to any clinical domain; rather, they are specific to the task of visualization and exploration of time-oriented data and to the semantics of the computational method that derives the abstractions. In the framework we describe in this paper, we build on our previous studies and computational tools. These studies concerned the nature of knowledgebased temporal abstraction, the automated acquisition of that knowledge, and the computational implementations of the temporal-abstraction reasoning methods. The temporal-abstraction task [29] is the task of creating context-sensitive interpretations of time-stamped clinical data in terms of higher-level concepts and patterns that hold over time intervals (see example in Figure 1). Interval-based abstractions are useful for planning therapy and for monitoring the application of clinical guidelines. [35] Temporal abstractions are a prerequisite for creation of high-level summaries of time-oriented clinical databases, such as electronic medical-record databases, thus reducing the information overload on care providers. As we show in this paper, a disciplined method for creation and maitenance of temporal abstractions is crucial for effective visualization of and exploration through these abstractions. Temporal abstractions are also helpful for explanation purposes by decision-support systems, and support interactive data-mining of time-oriented clinicalrecord databases. Visualization and exploration of information in general, and of large amounts of time-oriented clinical data in particular, is essential for effective decision making. Examples include deciding whether a patient had several episodes of bone marrow toxicity of a certain severity and duration, caused by therapy with a particular drug; or deciding if a clinical intervention, such as a specific insulin-therapy regimen, has been effective /99 $10.00 (C) 1999 IEEE 1

2 . Platelet counts ( ) 150K 100K BMT PAZ protocol Expected CGVHD M[0] M[1]M[2]M[3] M[0] M[0] Time (days) Granulocyte counts ( ) 2000 Figure 1. Typical inputs to and outputs of the temporal-abstraction task in a clinical domain. The figure presents abstractions of platelet and granulocyte values during administration of the PAZ protocol for treating patients who have chronic graft-versus-host disease (CGVHD), as a complication of a bone-marrow transplantation (BMT) event. = event; = platelet counts; = granulocyte counts; = open context interval; = closed abstraction interval; M[n] = myelotoxicity (bone-marrow toxicity) grade n. Different care providers require access to different types of time-oriented clinical data, which might be distributed over multiple databases. Furthermore, effective abstraction, visualization and exploration of clinical data and meaningful concepts and temporal patterns in them requires knowledge of domain-specific properties of the particular data, such as meaningful classifications into more abstract patterns, knowledge of whether similar patterns can be joined or should be considered as separate episodes, and an indication of how data should be displayed. We have previously shown [29, 34] that such temporal-abstraction knowledge can be acquired and represented formally within knowledge bases specific to each clinical domain. Thus, knowledge based summarization, visualization, and exploration of time-oriented clinical data and their abstractions requires an integrated solution to three subtasks: temporal abstraction, knowledge acquisition, and information visualization. A new solution must also consider the distributed nature of databases, knowledge bases, and problem-solving processes. An integrated framework will have direct implications by supporting multiple clinical tasks facing practicing physicians (e.g., therapy planning, chronic patient monitoring). The integrated architecture we are developing (most of whose individual components have been implemented) is presented in Figure 2. The new architecture comprises three types of components connected over the Web or a local-area network (see Figure 2). Several of the components, such as the temporal and statistical abstractions mediator, have been a topic of our previous research; others, such as the visualization and exploration modules, are the focus of our current work. Users (e.g., physicians) send complex temporal and statistical queries at varying levels of 1000 abstraction from local workstations (through the visualization module) to the temporal- and statisticalabstraction (TSA) server. The TSA server is a database mediator [44], that will access multiple domain-specific time-oriented databases to provide answers to the queries, using the RÉSUMÉ temporal- abstraction system we have developed in previous research (see Section 2.2). Typically, time-oriented databases include only raw timeoriented data, and no abstractions of those data; examples include databases of electronic medical records. To interpret the data correctly, the TSA server uses knowledge about temporal properties of the data by accessing the appropriate domain-specific knowledge base (through the ontology server). Thus, the TSA server is a temporal mediator to the timeoriented databases. The TSA server is an extension of the Tzolkin temporal mediator [Nguyen et al., 1997] (see Section 2.2). Clinical domain experts maintain the domain-specific temporal-abstraction knowledge bases, using graphical knowledge-acquisition tools that we have developed. [38] Domain-specific knowledge includes also visualization knowledge and user preferences. The clinical domain experts, the TSA server, and the local Temporal- and statistical-abstraction server (Tzolkin) Résumé Chronus DB Care providers DB Controller DB KNAVE visualization module Graphical Interface Expert physicians Domain ontology server Exploration Module KA Tool Figure 2. The overall KNAVE architecture. End users (care providers) interact with the visualization module to ask temporal queries about patient records. These queries are answered by the temporal- and statisticalabstraction mediator service, using domainspecific data from the appropriate clinical database, and temporal-abstraction knowledge from the appropriate temporal-abstraction knowledge base. The visualization module allows end users to explore the resultant abstractions in a manner specific to their clinical domain, using the temporal-abstraction and visualization knowledge from the appropriate knowledge base. Arrow direction indicates data flow. DB = database; KB = knowledge base; KA = knowledge-acquisition. KB KB KB /99 $10.00 (C) 1999 IEEE 2

3 visualization and exploration processes access the domain-specific knowledge bases through the domain ontology server. The visualization and exploration local client is an innovative computational module we are developing, which we call KNAVE (Knowledge-based Navigation of Abstractions for Visualization and Exploration). KNAVE will be used by all care-provider end users (e.g., physicians who need to summarize and explore patient data), and will coordinate the distributed services. We are currently developing a Java version of the KNAVE modules; however, we have tested their feasibility and usefulness through a preliminary Visual Basic prototype with highly encouraging results [4]. The KNAVE module includes two components: (1) a computational (query, browsing, and exploration) module, which will process information obtained from the TSA and domain-ontology servers to support tasks such as interactive semantic navigation (e.g., What data is this bone-marrow-toxicity pattern abstracted from? ), dynamic explanation (e.g., What classification knowledge was used to create this abstraction? ) and interactive exploration (e.g., What would the abstract pattern look like if the patient s platelet count were, in fact, lower on 1/6/98? ), and (2) a user interface visualization module, which displays the various windows and widgets that make up the KNAVE user interface and which drives the computational module. The KNAVE module filters the information coming from the TSA server to construct an internal representation which is efficient for the task of interactive exploration. This allows the representation of abstractions and relations between them in RÉSUMÉ to be separated from the process of displaying the information. 2. Information visualization and temporalabstraction In this section, we review the essential background work in temporal abstraction underlying the specific framework described in this paper. Visualization of information in general and of large amounts of time-oriented data in particular is essential for effective decision making. Much effort had been put in the past into creation of effective visualizations for information; an excellent example is the classic series of books by Edward Tufte on methods to display information [40, 41, 42]. Previous work had typically focused on exploring separately three different subtasks of the problem we are tackling: temporal abstraction, information visualization, and knowledge acquisition. Several approaches have been applied to the task of abstraction of time-oriented data into higher-level concepts, especially in medical domains, in which both large amounts of data and considerable knowledge are available [9, 10, 13, 15, 16, 19, 28]. None of these approaches, however, emphasized the need for a formal representation that facilitates acquisition, maintenance, sharing, and reuse of the required temporal-abstraction knowledge; this emphasis is the focus of our previous and current research. Furthermore, previous temporalabstraction approaches were typically not geared for use in a runtime system for visualization of and navigation through the domain-specific abstractions, and would not support a domain-independent interface. Research on information visualization in the areas of presentation and display techniques in general [40, 42], visualization of time-oriented clinical data [6, 14, 25], and human-computer interfaces [17, 27, 45] has developed useful visualization techniques for static presentation of raw time-oriented quantitative data and for browsing information, using various statistical and graphical methodologies such as scattergrams, pie charts, bar charts, three-dimensional representations [2] and their derivative techniques. These display methods, however, typically emphasize the display of raw data and do not focus on visualization of domain-specific temporal abstractions and on the issue of dynamic, interactive, navigation, using a domain-independent method, through multiple levels of these abstractions using domain-specific knowledge. The reason for that omission is that such capabilities require formal, domain-independent representations of the domain-specific temporal-abstraction knowledge, considerable effort in modeling the visualized domain, and the availability of computational mechanisms for creation of the abstractions. The past decade has witnessed considerable advances in semiautomated methods for knowledge acquisition and knowledge representation, based on approaches that operate at the knowledge level [23] and that assume taskspecific but domain-independent problem-solving methods [3, 5, 11, 20, 21, 43] which often succeed in alleviation of the knowledge-acquisition bottleneck. However, these methods often are not associated with runtime, interactive end-user applications, and focus on use by knowledge engineers and domain experts. Thus, visualization of time-oriented abstractions of clinical data using domain-specific knowledge is an important task which requires an integrated solution to all three subtasks. Such an integrated framework will have broad implications for multiple clinical tasks, such as therapy planning, patient monitoring, data interpretation, explanation of automated interpretations, and interactive data mining for clinical and research purposes. As we show in the following sections, we are employing techniques that we have previously developed to solve the temporal-abstraction task and its knowledgeacquisition requirements, augmenting them with a new computational structure for visualization of and navigation through the resulting abstractions /99 $10.00 (C) 1999 IEEE 3

4 2.1 Knowledge-based temporal abstraction of clinical data We have defined a general problem-solving method [11] for interpreting data in time-oriented domains, with clear semantics for both the problem-solving method and its domain-specific knowledge requirements: the knowledge-based temporal-abstraction method [29]. This method comprises a knowledge-level [23] representation of the temporal-abstraction task and the knowledge required to solve that task. The knowledge-based temporal-abstraction method decomposes the temporal-abstraction task into five parallel subtasks: temporal context restriction, vertical temporal inference, horizontal temporal inference, temporal interpolation, and temporal pattern matching. The five subtasks of the knowledge-based temporalabstraction method are solved respectively by five temporal-abstraction mechanisms (nondecomposable computational modules). The temporal-abstraction mechanisms produce output abstractions of several abstraction types: state (e.g., LOW), gradient (e.g., INCREASING), rate (e.g., FAST), and pattern (e.g., QUIESCENT-ONSET). The context-forming mechanism creates at runtime context intervals, induced by the presence of certain context-forming propositions, such as certain external events (not necessarily with the same temporal scope), over which hold interpretation contexts [31]. Context intervals create a relevant frame of reference for interpretation and enable the temporal-abstraction mechanisms to focus only on abstractions relevant for particular contexts, thus creating interpretations that are context-specific and avoiding unnecessary computations. The contemporaneous-abstraction mechanism abstracts one or more parameters and their values, attached to contemporaneous time points or time intervals, into a value of a new, abstract parameter. Thus, it performs a classification of a given parameter s value or a computational transformation of the values of several parameters, using a classification function. An ABSTRACTED-INTO relation exists between input and output parameters. The temporal-inference mechanism performs two subtasks: (1) inference of specific types of interval-based logical conclusions, given interval-based propositions, using a deductive extension of Shoham s temporalsemantic properties [36] (e.g., unlike two consecutive periods of anemia, two episodes of 9-month pregnancies cannot be summarized as an episode of an 18-month pregnancy, since they are not concatenable, a temporalsemantic property [36]), and (2) determination of the domain value of an abstraction created from two meeting abstractions (e.g., for a gradient abstraction, DECREASING SAME = NONINCREASING). The temporal-interpolation mechanism bridges gaps between temporally-disjoint propositions of similar types, using domain-specific temporal-dynamic knowledge about the dynamic behavior of the parameters involved [29]. The temporal-interpolation mechanism uses local and global truth-persistence functions to join temporally disjoint abstractions when values for direct determination of the abstractions are missing [30]. The temporal-pattern matching mechanism matches predefined temporal patterns and runtime temporal queries with data and concluded abstractions. The output is a pattern-type parameter which holds over an interval. The temporal-abstraction mechanisms require four domain-specific knowledge types for any particular domain: (1) structural knowledge (e.g., ABSTRACTED- INTO relations); (2) classification knowledge (e.g., definition of a parameter range as LOW); (3) temporalsemantic knowledge (e.g., the concatenable property); and (4) temporal-dynamic knowledge (e.g., persistence of a proposition over time when data is unavailable). The input to the temporal-abstraction task is a set of measured time-stamped parameters (e.g., temperature), external events (e.g., insulin injections), abstraction goals (e.g., diabetes therapy), and domain-specific temporal-abstraction knowledge. The output of the temporal-abstraction task is a set of interval-based, context-specific parameters at the same or at a higher level of abstraction and their respective values (e.g., a period of 5 weeks of severe anemia in the context of therapy with AZT ). An abstraction of a parameter (e.g., the state of the hemoglobin-level) is also a parameter. Time intervals are constructed from pairs of time stamps; time points are zero-length intervals. The structure {<, v, ξ>, I} is a parameter interval; it denotes that the parameter proposition parameter has value v given context ξ holds during time interval I. Propositions hold only over time intervals, although the time primitives are points (unlike pure interval-based approaches) [1]. If the parameter is abstract (derived), such a structure is called an abstraction. A temporal-abstraction ontology [29] includes (1) parameter ontology a theory of the relevant parameters and their temporal properties in the domain and the relations among these parameters (e.g., IS-A, ABSTRACTED-INTO), (2) an event ontology, which includes external events (e.g. medications), their interrelations (e.g., PART-OF relations) and properties, (3) a context ontology, which includes interpretation contexts (e.g., the temporal context defined by the effect of a drug) and relations (e.g., SUBCONTEXT) among interpretation contexts, (4) an abstraction-goal ontology, which includes all abstraction goals (which can induce contexts; e.g., monitoring of diabetes therapy) and their IS-A relations; and (5) all relations between inducing propositions and induced contexts /99 $10.00 (C) 1999 IEEE 4

5 2.2. RÉSUMÉ: an implementation of knowledgebased temporal abstraction We have implemented the knowledge-based temporalabstraction method as the RÉSUMÉ system [33, 34]. RÉSUMÉ generates temporal abstractions, given timestamped data and events, and the domain s temporalabstraction ontology. We tested the RÉSUMÉ system in several different clinical and engineering domains: protocol-based care (experimental therapy of AIDS patients, therapy of chronic graft-versus-host disease, and prevention of AIDS-related complications) [33]; monitoring of children s growth [18]; therapy of patients who have insulin-dependent diabetes [34], and even monitoring of traffic and evaluation of traffic-control actions [32]. The experiments in the traffic-control domain emphasized the generality of our methodology and its potential applicability not only to time-oriented data, but also to abstraction of data measured over any linear distance measure, and in particular, over linear space (e.g., spatial abstraction of data from traffic sensors along a highway). We also have incorporated the RÉSUMÉ system within a domain-independent temporal mediator [8, 44], the Tzolkin system [24], which combines the RÉSUMÉ temporal-reasoning system with the Chronus temporalmaintenance system [7]. Tzolkin answers temporalabstraction queries referred to a database by analyzing the query, retrieving the relevant data from the database and knowledge from the domain s temporal-abstraction ontology, and returning the appropriate abstractions The temporal-abstraction knowledgeacquisition tool As part of our previous research, we have constructed a graphical knowledge-acquisition tool for automated acquisition of temporal-abstraction knowledge from domain experts [38], using the PROTÉGÉ-II framework. The PROTÉGÉ II project [26, 39] develops a library of highly-reusable, domain-independent, problem-solving methods. One advantage of the PROTÉGÉ II approach is the production, given the relevant problem-solvingmethod and domain ontologies, of automated knowledgeacquisition tools, tailored for the selected problem-solving method and domain. Preliminary evaluation of the knowledge-acquisition tool for usability of the tool and reusability of the knowledge, using several domain experts, was quite encouraging. 3. The KNAVE project We employ an integrated knowledge-based approach to the visualization of time-oriented data and their multiple levels of temporal abstractions, by combining our previous work on knowledge-based temporal-abstraction and on mediators to time-oriented databases with developments in knowledge-acquisition tools. We call our evolving visualization and exploration system Knowledge-based Navigation of Abstractions for Visualization and Explanation (KNAVE). We have implemented and evaluated a small prototype [4] and are now in the process of consolidating and generalizing the overall computational architecture into a distributed one (see Figure 2). We are achieving our goals by capitalizing on (1) the domain-independent RÉSUMÉ temporal-abstraction system, (2) the temporal-abstraction knowledgeacquisition tool, (3) the domain-independent Tzolkin temporal-mediator module, which coordinates the RÉSUMÉ system and the Chronus temporal-maintenance system to mediate temporal queries to a time-oriented database, and (4) the use of a set of semantic-navigation operators that access the knowledge base that our graphical knowledge-acquisition tool acquires from domain experts for the purpose of computing domainspecific temporal abstractions in their respective domains. Thus, the operators defining the temporal query, visualization, and navigation processes embody, conceptually, the domain-independent semantics and ontology of our knowledge-based temporal-abstraction method; but these operators will use, for any given domain, the domain-specific temporal-abstraction knowledge of that particular domain. The basic module of the KNAVE system is the static temporal-abstraction visualization interface. This module enables the user to query a given clinical database in a domain-specific manner for time-oriented raw data, interventions, abstractions, and patterns, and to visualize graphically a static representation of the results of the temporal query. Thus, the user can ask show me periods of bonemarrow toxicity of Grade II or more in the past 80 days, in the context of therapy by a Prednisone-Azathioprine (PAZ) protocol (a guideline for treatment of chronic graft-versus-host disease following bone-marrow transplantation). The query defines the parameter (bonemarrow toxicity), parameter value (GRADE II or higher), the context (therapy by the PAZ protocol), and the time span (past 80 days) (Figure 3) The dynamic knowledge-based semanticexploration operators We believe that care providers need to be able to tap into the full knowledge contained within the clinical domain model while interacting with the system. Thus, we have implemented a module that performs knowledgebased exploration of the visualized data and its temporal abstractions. We enable users to change the focus of the /99 $10.00 (C) 1999 IEEE 5

6 Figure 3: The query interface of the KNAVE original Visual Basic prototype. The care provider is asking for all periods of bone-marrow toxicity of any value in the past 80 days, in the context of therapy by the PAZ protocol (therapy guideline). On the right, the user is browsing interactively the context and parameter ontologies of the medical domain of guidelinebased care. visualization by exploiting knowledge that has been acquired from domain experts to support temporal abstraction. In particular, we are using semantic links in the domain s temporal-abstraction ontology of parameters, interpretation contexts, and external events [29]. The knowledge-based semantic-exploration operators are demonstrated in Figure 4 as a set of browser trees. The dependency relation is split into the ABSTRACTED- FROM (derived from, as the layout customization prescribed in this case) and its inverse, ABSTRACTED- INTO (supports) relations to facilitate browsing. All browser trees are linked to the domain s temporalabstraction ontology (in this preliminary prototype, in a hardwired fashion) and enable dynamic exploration of that ontology. Figure 5 demonstrates the results of a motion along the ABSTRACTED-FROM (derived from) semantic link, following an initial query that displayed several bonemarrow-toxicity abstraction intervals. We have found as useful at least six major semanticexploration operators (see Figures 4 and 5): (1) generalization/specialization of the parameter (e.g., from the white blood-cell count parameter to the class of hematological parameters and vice versa) using the IS-A semantic links among parameters; (2) functional dependency among parameters (e.g., from a bone-marrow toxicity abstraction in the PAZ context to the platelet-state and granulocyte-state abstractions defining it) using ABSTRACTED- FROM relations (see Figure 5); (3) generalization/specialization of the interpretation context (e.g., from the preprandial (before meal) to the prebreakfast context in diabetes therapy) using the IS-A semantic links among interpretation contexts; (4) relations among contexts (e.g., from the context induced by the PAZ-therapy event to the context induced by a specific phase of that PAZ event, not necessarily contemporaneously) moving along the SUBCONTEXT relation; (5) generalization/specialization of the external event (e.g., from the insulin-administration event to regular-insulin administration) using the IS-A semantic links among events; (6) relations among events (e.g., from the PAZ-therapy event to a specific phase of the PAZ protocol in which a particular drug had been administered) moving along the SUBPART relation among events. Figure 4: The interface to the dynamic semanticnavigation operators in the original Visual Basic KNAVE protoptype. The abstracted-into relation had been split into the Derived-From Hierarchy and the Supports Hierarchy navigation operators. Figure 5: The result of an ABSTRACTED-FROM exploration query in the original Visual basic KNAVE prototype, given the bone-marrowtoxicity abstractions in Figure /99 $10.00 (C) 1999 IEEE 6

7 In the screen shot shown in Figure 5, the user chose to perform a depth-first exploration into the granulocytestate supporting abstraction, and into the granulocytecount raw data supporting that abstraction. The user could also display the parameters supporting the bone-marrowtoxicity abstraction in a breadth-first manner, thus displaying the granulocyte-state and platelet-state abstractions from which the bone-marrow-toxicity abstraction is derived. Once the user focuses on a clinical parameter within a specific time period, whether raw data, such as hemoglobin values, or abstractions at any level, such as bone-marrow toxicity in a particular context, she can also explore other presentations of the parameter values. For example, she can query for statistical descriptions of the parameter over the current time period, such as distribution (Figure 6) Temporal-syntactic and temporal-semantic browsing operators We are building into the KNAVE system purely syntactic (graphical) browsing operators that facilitate browsing. For instance, a temporal-zoom: Selection of one panel in a visualization window (e.g., just platelet counts) expands the display of its data to fill the whole screen, enabling the user to inspect data or abstractions during that time interval more closely. We can thus provide a more accurate definition of several categories of composite navigation operators, each of which is a specific combination of semantic-navigation and/or syntactic temporal-navigation operations. For instance, an operator comparable to drilling down the data in information systems, would include a syntactic temporal zoom (magnification) combined with moving along an ABSTRACTED-FROM semantic link. A rather unique combined syntactic and semantic operation is the enablement of users to change the temporal granularity of the display (beyond changing the scale) dynamically. Often, users need to examine data at different temporal resolution levels (e.g., days instead of hours). However, certain temporal-granularity levels are not useful for visualization of certain data types (e.g., heart rates in the intensive-care unit context need to be shown at the level of minutes). Thus, each data type has certain ranges of temporal granularity within which it is meaningful to visualize it and beyond which it should be visualized differently, using a data-specific aggregation operator (e.g., using descriptive statistics such as mean and standard deviation, or distribution; or moving to an abstraction of the parameter). The knowledge, which temporal-granularity ranges are meaningful for each parameter proposition, and what aggregation operators should be used automatically when the granularity chosen is out of that range, is part of the domain s visualization knowledge; it is acquired from the domain experts. The need for explicit representation of relevant temporalgranularity ranges was noted by Cousins and Kahn [6], although their main goal was the development of a syntactic visualization model, rather than exploration of data abstractions using domain knowledge Enhancement of the temporal-abstraction and knowledge-acquisition modules Figure 6. Display of a statistical description of an abstracted (derived) clinical parameter. In this case, the user asked for the distribution of the values of the bone-marrow toxicity abstraction (during a specific time period), as that abstraction is defined in the context of the PAZ protocol for therapy of chronic graft-versushost disease. We have found it necessary to make several enhancements to the implementation of the knowledgebased temporal-abstraction method in the RÉSUMÉ system and to the temporal-abstraction knowledgeacquisition tool, to support the KNAVE requirements: 1. We are enhancing the RÉSUMÉ system to enable creation of more complex abstraction patterns, and in particular, patterns comprising both temporal and statistical aspects (e.g., an increasing gradient of the weekly variance of the blood-glucose in the nonconvex post-breakfasts context over the past 6 weeks). 2. We enhanced the graphical temporal-abstraction knowledge-acquisition tool [38] so as to acquire more complex temporal patterns involving temporal-distance, temporal-relations, and parameter-value constraints. We enhanced the /99 $10.00 (C) 1999 IEEE 7

8 RÉSUMÉ runtime temporal-pattern matching mechanism accordingly. 3. We have added the ability to specify periodic temporal patterns in the knowledge-acquisition tool, to abstract these periodic patterns from clinical time-oriented data by the RÉSUMÉ system, and to visualize them within the KNAVE module. An example is detecting a pattern of administrations of regular and intermediate insulin drugs (with certain dose ranges) in the morning and in the evening, 30 to 45 minutes before breakfast and dinner, respectively, occurring 5 to 7 times each week. 4. We are enhancing the knowledge-acquisition tool to enable acquisition of temporal-visualization knowledge, such as the relevant temporalgranularity ranges that should be used for visualization of each parameter, the default abstraction or statistical functions that should be used when the visualization granularity is outside of that range, and domain-specific and individual user-interface preferences Dynamic interaction within the KNAVE architecture There are other advantages to the modular KNAVE architecture and its dynamic access to the domain s ontology Enabling care providers to set the goals of the abstraction process Often, the context for the abstraction process can be induced automatically from the data and the domain s temporal-abstraction ontology (e.g., therapy by the PAZ protocol induces a PAZ-therapy interpretation context). However, often the appropriate context for the abstraction process cannot be derived automatically from the data, such as when the goal of the abstraction process (e.g., therapy of diabetes) is only in the mind of the user and needs to be stated explicitly, so that it can be part of the database (at least temporarily) and thus can induce the appropriate interpretation context. Thus, we are enabling users, during the visualization session, to dynamically add abstraction goals [29], whose main object is to set the overall context(s) and relevant time span(s) Support of hypothetical (What If) queries Users of data mining, visualization, and exploration tools, such as KNAVE, often wonder if another specific datum would have changed the overall view of some segment of the data, if added of removed. We are developing a capability to add or retract at runtime data to and from the KNAVE dynamic memory. Thus, we are enabling the user to ask hypothetical ( what-if ) queries by hypothetically asserting or retracting data (e.g., what if the platelet count during the previous visit was in fact 50,000 and not 80,000? ) and by visualizing the resulting abstractions (e.g., the bone-marrow-toxicity grade might change to Grade III during the time interval corresponding to that visit, while certain abstraction might disappear). Both the abstraction capability and the what-if queries are supported by the RÉSUMÉ system and its specialized truth-maintenance system, which maintains logical dependencies among data and their abstractions [33, 34], and supports the nonmonotonic, defeasible nature of temporal abstraction Provision of explanations for temporal abstractions Care providers using decision-support tools often require meaningful explanations for either data interpretations or action recommendations, such as when a physician examines the visual results of a temporal query to a patient s record. Using the tight link between the KNAVE core visualization client and the domain ontology server (see Figure 2), we provide users with various types of context-sensitive explanations. For instance, we may need to answer a query such as why is this interval characterized as bone-marrow toxicity grade 3? (in addition to the query from which data is it abstracted? which can be answered by semantic navigation along the ABSTRACTED-FROM link). To answer that query, we retrieve from the domain s ontology and display the classification knowledge (here, a mapping function represented as a table) that defines the abstraction of bone-marrow toxicity grades from the toxicity levels of platelets and granulocytes in a context-sensitive fashion (e.g., specific to the PAZ-therapy context in which the abstraction was created) (Figure 7). Figure 7: Explanation in the KNAVE prototype. An explanation is being provided to a query about the bone-marrow-toxicity abstractions shown in Figure 4, by displaying a classification (functional-dependency) table from the temporalabstraction knowledge base. This function was acquired from the clinical expert who defined the the bone-marrow-toxicity abstraction in the PAZ context /99 $10.00 (C) 1999 IEEE 8

9 3.5. Explicit representation of domain-specific and individual user models The KNAVE framework enables a customization of the interface terms and behavior for each domain, including the listing of the relevant types of users in each domain (e.g., physicians, nurses, etc., in the case of medical domains) and their default task-semantics (e.g., whether to show just top level abstractions or all intermediate ones when returning the top-level abstraction) and userinterface (e.g., how should events be called in this domain) preferences, and by also enabling customization by individual users in that domain. These customizations are part of the domain s visualization model. This model also specifies a set of user profiles. These profiles further customize the display to accommodate needs of particular user groups in the domain. This differs from the more common concept of user preferences, which we represent as a further, individual, specialization on a particular type of a user profile. 4. Evaluation of the KNAVE framework We have performed a preliminary assessment of the original prototype version of the KNAVE core modules in a medical domain [4]. Seven users with varying medical and computer-use backgrounds were requested to answer within 20 minutes three complex temporal queries about the particular set of data we used, using only the new interface and a brief introduction to the navigation interface. The users were not otherwise familiar with the KNAVE system. The results were highly encouraging with respect to the users subjective enthusiasm and their objective capability to provide visual answers (mostly within 20 to 180 seconds, in total) to all queries. One of the lessons we learned, which we will emphasize in the final implementation, was the importance of redundancy. Somewhat surprisingly, answers were typically found using several (up to four) different paths to get to the same visualized set of abstractions (e.g., through the semanticnavigation interface, by going back to the initial temporalquery interface, by using the navigation drop menu, or by using a short-cut mouse right-button function). After the final reengineering is done (the preliminary Visual Basic prototype is being reimplemented in Java), we will perform a more formal evaluation of the KNAVE framework. The design will be an enhanced version of the feasibility study, with more users, qualitative and quantitative questionnaires, thinking aloud experiments, inclusion of well-defined target user behaviors, and application of standard methods for interface usability assessments [12] (e.g., quality of answers and time to get answers). 5. The New KNAVE Implementation We are reimplementing the KNAVE architecture using the Java language for the interface and by using the common object request broker architecture (CORBA) standard for communication among the KNAVE three types of core components (see Figure 2). In the current new prototype, the initial query interface requires the user to select the patient and the medical domain name (e.g., protocol-based care) (Figure 8). The patient data exists on a relational database of time-oriented patient records, which is accessed when necessary by the temporal mediation module. The medical domain knowledge is imported as a set of relevant classes and instances from the ontology server. Once the patient and the domain are known, the user can open one or more new frames, each of which is used in uniform fashion for query and exploration (Figure 9). Each frame includes domain-specific browsers for the main four temporal-abstraction classes (parameters, events, contexts, and patterns). Using the browsers (which access in dynamic fashion the domain s temporalabstraction knowledge base), the user defines her query. The results of the query are then displayed in the left hand portion of the frame. Changing the browser selection in the right hand panel changes the display in the left hand panel. Alternatively, the user can explore the display by creating a new frame for each exploration result. Thus, exploring the abstracted-from relation in Figure 5 would imply either changing the display to visualize three different views of the data, or creation of three separate frames that can be compared. Users are also allowed to cut and paste among frames. Figure 8: Starting a session in the new, Javabased KNAVE prototype. The user selects a patient and a domain knowledge base that will be used to interpret and explore that patient s data /99 $10.00 (C) 1999 IEEE 9

10 semantic navigation operators are identical, the resultant browsing process is specialized to the clinical domain s parameter, event, context, and abstraction-goal (sub) ontologies, defined by the knowledge-based temporalabstraction ontology, and acquired from experts physicians. By using intelligent summarization techniques and knowledge-based visualization and exploration operators, we expect to considerably reduce the information load on physicians and other care providers who need to handle large amounts of time-oriented patient data. Acknowledgments Figure 9: Browsing and exploring patient data in the new, Java-based KNAVE prototype. The user defines the query by selecting on the right hand panel one of four types of browsers, each of which has access to the appropriate domainspecific temporal-abstraction knowledge. The result is then displayed in the left hand panel. 6. Discussion We are developing a conceptual and computational knowledge-based framework for interactive visualization and exploration of time-oriented clinical data and their multiple levels of temporal abstractions. The approach we are using is quite different from existing ones, in several respects. For example, unlike the intriguing Lifelines [25] work, in which patient historical events and data are displayed as is over time, we strive to (1) interpret the data in a manner specific to the particular clinical domain and context in which the data has been acquired, and (2) visualize and interactively explore the data and its multiple levels of abstractions, using the same domainspecific knowledge. These goals imply dynamic access to the medical domain s interpretation and visualization knowledge, and the corresponding computational mechanisms that can exploit that knowledge. In our framework, the semantics of the querying, exploration, and explanation operators are defined by the corresponding formal semantics of the domainindependent ontology of the knowledge-based temporalabstraction problem-solving method and its computational mechanisms. The semantic-navigation operators allow the runtime user to navigate visually through the domainspecific temporal-abstraction (sub) ontologies, thereby leading to reciprocal visual navigation through the multiple levels of temporal abstractions of the particular time-oriented database that is queried. Thus, although the KNAVE interface is essentially the same (apart from domain- and user-dependent customizations) in every medical domain, and the semantics of the syntactic and This work has been supported by grants LM05708 and LM06245 from the National Library of Medicine and IRI from the National Science Foundation. Computing resources were provided by the CAMIS project, funded under grant No. LM05305 from the National Library of Medicine. We thank our domain experts, Drs. Stites, Kaizer, Basso, and Wilson for their helpful feedback during the development of the initial interface, and Dr. Puerta for his useful background references and suggestions regarding the evaluation of the preliminary prototype. References 1. Allen, J.F. (1984). Towards a general theory of action and time. Artificial Intelligence 23, Carpendale M.S.T., Cowperthwaite D.J., and Fracchia F.D. (1997). Extending Distortion Viewing from 2D to 3D. IEEE Computer Graphics and Applications 7/8, Chandrasekaran, B. (1986). Generic tasks in knowledgebased reasoning: High-level building blocks for expert system design. IEEE Expert 1, Cheng, C., Shahar, Y., Puerta, A.R., and Stites, D. P., (1997). Navigation and Visualization of Abstractions of Time-Oriented Clinical Data, Section on Medical Informatics Technical Report No. SMI , Stanford University, CA. 5. Clancey, W.J. (1985). Heuristic Classification. Artificial Intelligence 27, Cousins, S.B., and Kahn, M.G. (1991). The visual display of temporal information. Artificial Intelligence in Medicine 3, Das, A.K., and Musen, M.A. (1994). A temporal query system for protocol-directed decision support. Methods of Information in Medicine 33(4), Das, A.K., Shahar, Y., Tu, S.W., and Musen, M.A. (1994). A temporal-abstraction mediator for protocol-based decision support. Proceedings of the Eighteenth Annual Symposium on Computer Applications in Medical Care, pp , Washington, DC /99 $10.00 (C) 1999 IEEE 10

11 9. De Zegher-Geets, I.M. (1987). IDEFIX: Intelligent summarization of a time-oriented medical database. M.S. Dissertation, Medical Information Sciences Program, Stanford University School of Medicine, June Knowledge Systems Laboratory Technical Report KSL 88-34, Department of Computer Science, Stanford University, Stanford, CA. 10. Downs, S.M., Walker M.G., and Blum, R.L. (1986). Automated summarization of on-line medical records. In Salamon, R., Blum, B. and Jorgensen, M. (eds), MEDINFO 86: Proceedings of the Fifth Conference on Medical Informatics, pp , North-Holland, Amsterdam. 11. Eriksson, H., Shahar, Y., Tu, S.W., Puerta, A.R., and Musen, M.A. (1995). Task modeling with reusable problem-solving methods, Artificial Intelligence 79 (2), Gould J.D. (1988). How to design Usable systems, in M. Helander (ed), Handbook of Human-Computer Interaction. Elsevier Science Publishers. 13. Haimowitz, I.J., and Kohane, I.S. (1993). Automated trend detection with alternate temporal hypotheses. Proceedings of the Thirteenth International Joint Conference on Artificial Intelligence, pp , San Mateo: Morgan Kaufmann. 14. Kahn, M.G., Abrams, C.A., Cousins, S.B., Beard, J.C., and Frisse, M.E. (1990). Automated interpretation of diabetes patient data: Detecting temporal changes in insulin therapy. In Miller, R. A. (ed), Proceedings, of the Fourteenth Annual Symposium on Computer Applications in Medical Care, pp , Los Alamitos: IEEE Computer Society Press. 15. Kahn, M.G. (1991). Combining physiologic models and symbolic methods to interpret time-varying patient data. Methods of Information in Medicine 30, Kohane, I.S. (1987). Temporal reasoning in medical expert systems. Technical Report 389, Laboratory of Computer Science, Massachusetts Institute of Technology, Cambridge, MA. 17. Kolojejchick J., Roth S.F., and Lucas P. (1997). Information Applications and Tools in Visage. IEEE Computer Graphics and Applications 7/8, Kuilboer, M.M., Shahar, Y., Wilson, D.M., and Musen, M.A. (1993). Knowledge reuse: Temporal-abstraction mechanisms for the assessment of children s growth. Proceedings of the Seventeenth Annual Symposium on Computer Applications in Medicine, pp , Washington, DC. 19. Larizza, C., Moglia, A., and Stephanelli, M. (1992). M- HTP: A system for monitoring heart-transplant patients. Artificial Intelligence in Medicine 4, McDermott, J. (1988). Preliminary steps toward a taxonomy of problem-solving methods. In Marcus, S. (ed), Automating Knowledge-Acquisition for Expert Systems. Boston: Kluwer. 21. Musen, M.A. (1992). Dimensions of knowledge sharing and reuse. Computers and Biomedical Research 25, Musen, M.A., Tu, S.W., Das, A.K., and Shahar, Y. (1996). EON: A component-based approach to automation of protocol-directed therapy. Journal of the American Medical Association 3 (6), Newell, A. (1982). The knowledge level. Artificial Intelligence 18, Nguyen, J., Shahar, Y., Tu. S.W., Das, A.K., and Musen, M.A. (1997). A Temporal Database Mediator For Protocol- Based Decision Support. Proceedings of the 1997 AMIA Annual Fall Symposium (formerly the Symposium on Computer Applications in Medical Care), pp , Nashville, TN. 25. Plaisant C., Milash B., Rose A., Widoff S., and Shneiderman B., Lifelines: Visualizing Personal Histories. Proceedings of CHI '96 (Vancouver BC, April 1996) (ACM Press) Puerta, A.R., Egar, J.W., Tu, S.W., and Musen, M.A. (1992). A multiple-method knowledge-acquisition shell for the automatic generation of knowledge-acquisition tools. Knowledge Acquisition 4, Rohrer, R., and Swing E. (1997). Web-based information visualization, IEEE Computer Graphics and Applications 7/8, Russ, T.A. (1989). Using hindsight in medical decision making. Proceedings, Thirteenth Annual Symposium on Computer Applications in Medical Care (L. C. Kingsland, Ed.), pp , IEEE Comput. Soc. Press, Washington, D.C. 29. Shahar, Y. (1997). A framework for knowledge-based temporal abstraction. Artificial Intelligence 90 (1 2), Shahar, Y. (in press). Knowledge-based temporal interpolation. Journal of Experimental and Theoretical Artificial Intelligence. 31. Shahar, Y. (1998). Dynamic temporal interpretation contexts for temporal abstraction. Annals of Mathematics and Artificial Intelligence 22 (1-2) Shahar, Y., and Molina, M. (1998). Knowledge-based spatiotemporal abstraction. Pattern Analysis and Applications, 1 (2) Shahar, Y., and Musen, M.A. (1993). RÉSUMÉ: A temporal-abstraction system for patient monitoring. Computers and Biomedical Research 26, Reprinted in van Bemmel, J.H., and McRay, A.T. (eds) (1994), Yearbook of Medical Informatics 1994, pp , Stuttgart: F.K. Schattauer and The International Medical Informatics Association. 34. Shahar, Y., and Musen, M.A. (1996). Knowledge-based temporal abstraction in clinical domains. Artificial Intelligence in Medicine 8 (3), /99 $10.00 (C) 1999 IEEE 11

12 35. Shahar, Y., Miksch, S., and Johnson, P.D. (in press). A task-specific ontology for the application and critiquing of time-oriented clinical guidelines. Artificial Intelligence in Medicine. 36. Shoham, Y. (1987). Temporal logics in AI: Semantical and ontological considerations. Artificial Intelligence 33, Snodgrass, R., and Ahn, I. (1986). Temporal databases. IEEE Computer, 19, Stein, A., Musen, M.A., and Shahar, Y. (1996). Knowledge acquisition for temporal abstraction. Proceedings of the 1996 AMIA Annual Fall Symposium (formerly the Symposium on Computer Applications in Medical Care), pp , Washington, DC. 39. Tu, S. W., Eriksson, H., Gennari, J., Shahar, Y., & Musen, M. A. (1995). Ontology-based configuration of problemsolving methods and generation of knowledge-acquisition tools: Application of PROTÉGÉ-II to protocol-based decision support. Artificial Intelligence in Medicine 7 (3), Tufte, E.R. (1983). The Visual Display of Quantitative Information. Graphics Press:CT. 41. Tufte, E.R. (1990). Envisioning Information. Graphics Press:CT. 42. Tufte, E.R. (1997). Visual Explanations. Graphics Press:CT. 43. Weilinga, B., Schreiber, A.T., and Breuker, J. (1992). KADS: a modeling approach to knowledge engineering. Knowledge Acquisition 4, Wiederhold G. (1992). Mediators in the architecture of future information systems, IEEE Computer 3, Wright, W. (1997). Business visualization applications. IEEE Computer Graphics and Applications 7/8, /99 $10.00 (C) 1999 IEEE 12