1. BRIDGING FRAMES AND AXIOMS: INCREASING EXPRESSIVENESS WHILE MAINTAINING USABILITY WITH EZPAL

Transcription

1 EZPAL: Environment for Composing Constraint Axioms by Instantiating Templates Chih-Sheng J. Hou, Mark A. Musen, Natalya F. Noy Stanford Medical Informatics, Stanford University, Stanford, CA , USA Abstract: Many ontology-development tools allow users to supplement frame-based representations with arbitrary logical sentences. However, few users actually take advantage of this opportunity. For example, in the Ontolingua ontology library, only 20% of the ontologies have any user-defined axioms. We believe the difficulty of composing axioms primarily accounts for the lack of axioms in these knowledge bases: Many domain experts cannot translate their thoughts into abstract and symbolic representations. We attempt to remedy the difficulties by identifying groups of axioms that manifest common patterns, creating templates that allow users to compose axioms by filling in the blanks. We studied axioms in two public ontology libraries, and derived 20 templates that cover 85% of all the user-defined axioms. We describe our methodology for identifying the templates and present examples. We constructed an interface that allows users to create constraints on knowledge bases by filling in blanks; our usability testing shows that users could use templates to encode axioms with a success rate similar to that of experts writing directly in an axiom language. Our approach should foster the introduction of axioms and constraints that are currently missing in many ontologies. Key words: frame-based system, knowledge acquisition, knowledge representation 1. BRIDGING FRAMES AND AXIOMS: INCREASING EXPRESSIVENESS WHILE MAINTAINING USABILITY WITH EZPAL Frame-based representation systems (Karp, 1993) are a popular choice for knowledge representation. Because their taxonomic categorization of canonical concepts often bears close resemblance to the way humans describe knowledge, frame-based systems serve as efficient tools for knowledge acquisition and representation. With the proliferation of approachable user interfaces for frame-based knowledge acquisition, such as the Protégé ontology-editing environment (Protégé 2004), many domain experts actively participate in knowledge acquisition, often without the mediation of knowledge engineers. Domain experts enter information about classes and properties of concepts through a convenient GUI interface. For example, Error! Reference source not found. shows a form for defining a class in the Protégé environment. However, even with the cognitive simplicity of frame-based systems, the methodology does not completely eliminate the need for an

2 experienced developer to mediate the Figure 1. A screenshot taken from Protégé, representing classes, instances, and slots. The left column shows classes of the knowledge base as a taxonomic hierarchy; the middle column contains an instance of the class organization; and the right column presents relationships the instance encodes. process of the knowledge-base design. Most frame-based systems lack the ability to express disjunction, existential quantification about frames, or relationships between properties of the same frame or different frames. For example, one cannot restrict a value of one property based on that of another property. A remedy for this limitation is the introduction of a more expressive axiom language based on First Order Logic (FOL) developed specifically to express these more complex relationships. The traditional approach to specify such relationships involves composing and displaying logical functions in a form that is incomprehensible to many domain experts without training in FOL. An example in

3 Figure 2 shows a simple constraint, every employee has a unique ID, in an axiom in Protégé Axiom Language (PAL), which is an axiom language for the Protégé ontology-editing environment 1 based on Knowledge Interchange Format (KIF) (Genesereth and 1

4 Figure 2 Text-Based Axiom-Editing Environment in Protégé. The axiom in the Protégé Axiom Language (PAL) says that every employee must have a unique ID. Fikes, 1992). The axiom has the following equivalent English translation: For every two employees both of whom have IDs, if the two employees are not the same, their IDs cannot be identical. The difficulty in conceptualizing a thought in a symbolic representation is compounded by the fact there is no standard implementation of an axiom language. Therefore, even though the addition of an axiom language improves the expressiveness of a frame-based system, in practice this gain cannot be realized by developers unless they can overcome the peculiarities of a specific knowledge-based system to enter working axioms correctly. The lack of an abstracted representation for the logical expressions for a hybrid model, in contrast to the highly abstracted user-interface of the original model, does little to address the situation. This factor can lead to a failure to encode critical knowledge realizable only through axioms. We examined axioms in FOL that users included in ontologies in two large ontology libraries, the Protégé ontology library (Protégé 2004) and the Ontolingua ontology library (Ontolingua 1998). We noticed two important facts: First, many ontologies simply lack user-generated axioms, likely because axioms are very hard for developers to write. For example, in Ontolingua, only 20% of the ontologies provide manually generated axioms. Second, most of the axioms that do exist in the ontologies can be described by a small set of templates. In fact, we discovered that 20 templates covered 85% of user-defined axioms in the two libraries (Hou et al.,

5 202). We can use these templates to allow users to specify axioms by selecting an appropriate template and filling in the blanks rather than by having to conceptualize and express a complex axiom in first-order logic. Figure 3 shows the template that would allow users to create the aforementioned constraint every employee has a unique ID simply by populating the blanks of the templates. Figure 3 A template-based interface that allows users to encode the same statement as in Figure 2. Users create this constraint by specifying values in blanks, in this case Class Employee and Slot ID. We acknowledge that this approach will not encompass the modeling of every axiom possible, but it does allow users to specify common logical assertions through an abstracted interface. In case the template library does not cover certain axioms, or an advanced user desires additional flexibility, he can always fall back on the traditional mechanisms of specifying FOL sentences. Therefore, our approach augments, instead of replacing, existing features of a hybrid system. We constructed a knowledge base of templates, devised strategies to organize the templates based on referential patterns, and constructed a graphical user interface that permits users to instantiate constraint axioms by filling values into the templates. The paper begins the discussion with an outline of the underlying knowledge model (Section 2). We then describe our methodology for collecting templates (Section 3). In Section 4, we discuss EZPAL, a plug in for Protégé, which implements our approach (Section 5). We report on the methods and results of our usability experiments, which tested whether or not the templates allowed quick and accurate composition of the axioms similar to those found in various knowledge bases. We describe the evaluation protocols in Section 6 and results in Section 7. We conclude with the discussion of the experiment results (Section 8), related work (Section 9) and future work (Section 10). 2. KNOWLEDGE MODEL We apply our approach in the context of a general frame-based knowledge model. A frame represents a concept or object by encapsulating its characteristic values in various slots that taken together define a data structure. When a frame represents a general concept in the same way as a Platonic form, it becomes a class, whereas an individual realization or instantiation of that

6 concept is called an instance (Figure 1). A superclass is a superset of instances in its subclasses. For example, in Figure 1, the class Editor is a subclass of the class Author. Author is therefore a superclass of Editor, since Author is a more general concept. Chief Hancho is a specific instance of the class Editor. Each instance has a set of slots that contain attributes of the instance. A slot can contain a number, a string, a symbol, an instance of another class, or a class itself. For example, the salary slot of the Employee class contains an integer value. Subclasses inherit slots from superclasses. We can specify attributes of slots (facets) such as the value type, cardinality, default values, and a data range. For example, we can constrain the salary slot of every employee to be greater than 100. However, as we previously mentioned, we cannot usually specify relations between different slots. It is not possible to specify that the salary of every editor is greater than the salaries of those employees whom the editor is responsible for. These assumptions are consistent with the general knowledge model underlying the Open Knowledge-Base Connectivity (OKBC) protocol (Chaudhri, 1998), which was designed as a means of communication among different frame-based knowledge bases. 3. IDENTIFICATION OF TEMPLATES In order to identify axiom templates that would cover a large fraction of axioms that users have put into knowledge bases, we selected two ontology libraries that contained ontologies contributed by developers in many domains. We looked at axioms in ontologies from the Ontolingua ontology library (Ontolingua 1998) and Protégé ontologies. These sources contain knowledge bases from diverse fields and have ample axioms. Ontolingua and Protégé have very similar knowledge models (OKBC). The 18 ontologies that we examined covered such diverse subjects as biological processes, chemical agents, enterprise organization, and bibliographical data. For example, the VT-Domain ontology in Ontolingua describes the declarative concepts (such as component types) and parameters (such as standard physical dimensions) involved in assembling elevators. The ontology for the Athena project (Tu and Musen, 2001), a private ontology from which we derived several axioms, also contains declarative concepts, such as a controlled vocabulary for drugs, lab tests, and diseases, in addition to specific drug data for generating therapy guidelines. We did not consider upper-level and mathematical ontologies which contained specific, complex logical functions seldom reusable in other contexts. Most ontologies had 10 to 30 axioms. Our process for identifying templates consisted of the following steps: 1. We identified axioms that were identical except for the names of specific variables and

7 frames. We removed the variable and frame references to generate a pattern. 2. We used simple rules and judgment to identify groups of similar patterns and generalized them into templates. For example, if there were two patterns Slot A contains B and. Slot A does not contain B, we group the two patterns into the following template:. Slot A does/does not contain B. We introduce variations (see Section 4.3) to keep track of the subtle differences between patterns grouped when into a template (point 3). 3. We derived generic properties for categorization of the templates (see Section 4.2). Consider the following two axioms in a Protégé ontology describing clinical guidelines for hypertension treatment: 2 (forall?process (=> (slot-not-null inhibited_by_object?process) (forall?process (/= "cytochalazin" (name (inhibited_by_object?process))))) (=> (slot-not-null inhibited_by_object?process) (/= "neuraminadase" (name (inhibited_by_object?process))))) The first axiom states that No process instance contains the value cytochalazin in the slot inhibited_by_object. The second axiom states that No process instance contains the value neuraminadase in the slot inhibited_by_object. These axioms vary only in referred slot value cytochalazin and neuraminadase. Thus, we can generalize these two axioms into the following pattern: All instances of class do not contain the value in slot. We then generalize these patterns further, trying to balance the following tradeoff: On the one hand, we want to have a small number of templates in the final collection, simplifying the search for the appropriate template for the ontology developer. On the other hand, we want to ensure that the templates do not become so general as to be incomprehensible. Table 1 shows 4 examples of the 20 templates that we identified. 2 The ontology courtesy of Samson Tu.

8 Every instance of class that shares the same value in slot must/cannot have identical values in slot. Example: All staff members who share an office are assigned different desks. Every instance of Class Staff that share the same value in Slot office-number : Class Staff must share/not share values in Slot desk-number : Class Staff. Every instance of class appears at least once in slot for any instance of class. Example: Every student has at least one advisor. Every instance of Class Student appears at least once in Slot advisee : Class Professor of any instance of Class Professor. For every instance I 1 of class, there must be an instance I 2 of class that contains I 1 in slot. Example: Every student is an advisee of a professor For every instance I 1 of Class Student, there must be an instance I 2 of Class Professor that contains I 1 in slot advisee: Class Professor. For every instance of class, slots and cannot have the same value. Example: A student cannot both likes and dislikes an activity. For every instance of Class Student, Slot likes : Class Student and Slot dislikes : Class Student cannot have the same value. Table 1 Examples of templates with their sample usage. 4. MODELING TEMPLATES AS A PROTÉGÉ ONTOLOGY We developed an ontology of templates in Protégé, storing with each template information to help users understand what the template is about.

9 4.1 Components of a template Figure 4 demonstrates the implementation of a template, which is required to generate a user view (see Figure 3). Note that EZPAL users do not need to see these template definitions; rather, they see a user interface that EZPAL generates automatically based on these definitions (Figure 5). It is the template designer and usually a PAL expert who specifies these definitions.

10 Figure 4 Components of a complete template definition. Numbers in the Figure correspond to elements in the list in Section 4.1. Each definition includes a sentence with blanks to fill in, an example of sample usage, a list of properties, a prototypical axiom translation, a machine-interpretable expression of the template, and an estimate of the level of difficulty entailed in using the template. The form in the Figure is generated automatically from the Protégé ontology for template definition. 1. The English-language representation or the name of the template as a fill-in-the-blanks sentence. For the template in Figure 4, this representation is: Every instance of class appears at least once in slot of any instance of class 2. A set of properties describing the template (see Section 4.2). 3. A sample usage sentence in natural language and template-based English that includes a possible interpretation of the template with the blanks already filled in. Every student has at least one advisor Every instance of Class Student appears at least once in Slot advisee : Class Professor of any instance of Class Professor. 4. A generalized pattern that defines variations of the template and that drives translation into the target representation system. Here is a translation of the example template into Protégé Axiom Language (PAL): (defrange?{class A} :FRAME {Class A})

11 (defrange?{class B} :FRAME {Class B}) ^(forall?{class A} (exists?{class B} ({Slot B 0}?{Class B}?{Class A}))) The variables enclosed in curly brackets can be substituted with direct references to generate a working PAL axiom (see Section 5.1). 5. A computer-interpretable expression of the template for interface generation: The expression contains information that enables the automatic generation of a user interface to acquire fill in the blanks data. This information can in turn be used to translate a generalized pattern to a specific axiom in PAL. It also dictates what type of values can fill-in-the-blanks. This feature allows us to set additional user-interface controls on the types of the allowed frames for which users can fill in a particular blank. The example below limits user s choice of a slot to those slots of the second class B, thus effectively preventing users from composing potentially inappropriate axioms. Every instance of {Class A} appear at least once in {Slot B 0} of any instance of class {Class B} 6. Level of Difficulty: A subjective ranking between 1 and 10 indicating the complexity level of the template, where a lower value indicates the template is more easily understood by users. The template designer assigns the ranking based on his subjective view. However, in general, the ranking correlates positively with the number of blanks users need to fill in and negatively with the frequency with which the template appeared in the samples we examined. We model templates in Protégé as classes and associated information as slot values, thereby changing the paradigm of axiom-composition into one of axiom instantiation, Individual template-based axioms are thus instances of the respective template classes. 4.2 Generic properties for categorization of templates We considered several approaches to classifying templates. The first approach involved creating a hierarchy of templates with strict inheritance. However, we believe there are inherent problems associated with this approach. Empirically, we observed that there is no clear notion of an inheritance relationship between templates. Making a more complicated template an extension of another, simpler template often required overriding the majority of inherited features. This fact defeats the original purpose of inheritance. The second solution for structuring templates involved introduction of abstract super templates with multiple inheritance. Conceptually, the search for super-templates involves semantic abstraction of the templates, but this strategy resulted in templates that are weak and vague, whereas we designed the original templates to be unique and concise.

12 We adopted a third approach for organizing templates: We chose to organize the template classes through descriptions rather than through a hierarchical organizational scheme. Since each constraint axiom ultimately specifies how a particular slot value relates to another, we defined the following properties, which specify the structural or referential relationships between constrained or constraining slots: A condition constrains values of a slot for all instances of one class. This property applies when a constraint needs to apply a comparison across values of the same slot for instances of the same class. For example, slot must contain a unique value. Value(s) of a slot constrain slot values for related instances. This property applies when a slot value for one class restricts value for another slot that belongs to a different but related class. For example, if instance I 1 of class contains another instance I 2 of class in slot, then I 2 must contain value in its slot. Value(s) of a slot constrain other slot values for the same instances. This property applies when a slot value of one instance restricts the slot value of another slot for the same instance For example, for every instance, slot and slot cannot contain the same value. Multiple constraining slots or multiple constrained slots. - applies when the conditions need to invoke a particular constraint and/or there are more than one slot whose value needs constraints e.g. For every instance of class whose slot contains a value, slot must contain value and slot must contain value. These properties are not mutually exclusive, so it is possible to associate one template with more than one property. Following this strategy, it sufficed to create a flat list of templates in which all templates have the superclass template. We then assigned these properties to individual templates. 4.3 Variation in templates Each template can have several variations. Variations can result from using multiple conditions or resulting statements, or from applying different operations to slot values. In general, several variations are automatically associated with certain defined phrases or built-in to a template. For example, several templates have the phrase(s).that contains.. This phrase is short for that does/does not contain, where users will be able to choose either the positive or negative form through an interface in an attempt to generate a constraint axiom. For a different type of variation, which generalizes the number of conditions for a particular

13 slot, consider the following example: There exists an instance of class that contains (value in slot.) (*N) This variation specifies that both the following patterns are legal expansions of the template: There exists an instance of class that contains value in slot. There exists an instance of class that contains value in slot and contain value in slot. These patterns can give rise to the actual constraints found in knowledge bases. For example: There exists an instance of class Drug_Usage that has value Cardio-Selective-Beta-Blocker in slot Drug-Name. There exists an instance of class Current-Medicine that contains value hydrochlorothiazide in slot Drug-Name and value greater than 25 in slot daily-dose. Allowing such variations enables template designers to derive succinct yet flexible template expressions. Without this mechanism, we would have to store individually templates with very minor differences to ensure adequate coverage of the template library, and the template library would become unwieldy to browse and maintain. 5. EZPAL TAB 5.1 Composition of the EZPAL Interface We developed the EZPAL Tab, an extension to the standard Protégé ontology-editing environment (Protégé 2004) that allows the creation of PAL axioms by filling in blanks in a template, much like the process of creating classes and instances by filling in slot values in forms. While defining an ontology, the user can use the EZPAL tab to specify additional constraints on the ontology elements. For example, Figure 4 shows a snapshot of the tab, where a template represents the constraint Every editor s salary is higher than the employees he is responsible for. The EZPAL interface consists of four sections: (1) the Properties filter panel, (2) the Template list panel, (3) the Sample Usage panel, and (4) the Fill in the blank panel. 1. The Properties filter panel the panel modulates the search space or the size of the template list in display by showing only those templates that satisfy all the selected properties on the top panel. The filter panel also allows the users to choose from 3 modes: novice, intermediate, and expert modes. Each mode acts as another level of filtering by showing only the templates with their level of difficulty values under a certain threshold. 2. The Template List Panel The list displays the English representation of templates satisfying the property and the level of difficulty filter. Clicking on a specific template changes

14 Figure 5. Using the four components of EZPAL Tab to create a constraint Every editor s salary is greater than that of the employees for whom he/she is responsible. The user browses the template library by clicking on one in display on panel 2 the Template List Panel, examines sample usages on panel 3, and fills in the details on panel 4. the other two panels on the right to display the associated examples and fill-in-the-blanks form. 3. The Sample Usage Panel This panel displays the sample usage information for a template in two forms: as an example English sentence and as a template in which the blanks have already been filled. 4. The Fill in the blank panel This is the panel where user actually fills in the blanks and has a sentence about how this actually happens. When the user fills in the blanks, EZPAL limits his choices only to the ones that the ontology allows. For example,

15 Figure shows a permitted sequence of steps users can fill in values: one must specify the class type before specifying its slot, and subsequently the slot value. The tab displays the newly created axiom on the axioms based on this template section of the panel. Figure 6 Partial screenshots of a template to demonstrate the built in value selection constraint mechanism. (a) The user must specify a class type before it can specify the slot or slot value (b) User can choose from the slots of employee once the class is chosen. (c) The user can now define the slot value once the slot salary is chosen. 6. EVALUATION PROTOCOL AND RESULTS We performed user studies to evaluate formally the hypothesis that templates could help users to encode successfully simple axioms without any knowledge of an axiom language itself. More specifically, we evaluated how successful the users were in encoding constraints using the EZPAL tab. 6.1 Subjects and Training The participants in the experiment were four students, two staff members at Stanford Medical Informatics, and one student at University of Washington. All participants had one or more years of experience using the basic Protégé ontology-development environment for knowledge representation and knowledge acquisition, but they had little or no experience with the Protégé Axiom Language (PAL). All users had some Computer Science background. There was no formal training session to teach the participants how to use the EZPAL Tab. Instead, each participant received a link to a website containing a short description on the tab s features and a few examples. 3 We asked each participant to generate a set of constraints for the assigned knowledge base by using the EZPAL Tab. The success rate and time measurements served as the basis for our evaluation. 3 The experiment materials are available at

16 6.2 The Knowledge Base and Constraints for Evaluation We used an ontology describing a travel domain, which was available to us. 4 The ontology provided basic descriptions for modes of transportations, trip information, and facilities at destinations. The domain description contained mostly common-sense knowledge, and thus all participants were equally familiar with the domain. Furthermore, each participant received detailed written explanation of the ontology that elucidated the concepts and relations that were not immediately clear. The list of constraints that we asked the users to encode was derived primarily from the PAL constraints in the original ontology. We simplified several complex constraints into several simple ones and introduced a few more constraints that would enhance the integrity of the knowledge base describing the travel domain. Table 2 shows nine constraints that the participants were asked to encode. The constraints are written in plain English independently of the EZPAL templates to simulate a real world situation. The participants were given one hour to encode them. We intentionally included several constraints that could not be encoded with EZPAL (entries 5 and 9) to add more realism to the user studies. We informed the participants that they would need to decide whether to try to find a template for each axiom. We told then that even if a suitable template exists for an axiom, the wording is not necessarily equivalent. These conditions reflect real-world situations. 1. Every arrival city of a flight must have an airport. 2. For any lodging the number of available rooms cannot exceed total number of rooms. 3. Every destination of a flight must also have another flight departing from it. 4. No two transport systems can share the same make and model numbers. 5. The only possible mode of transportation between cities in different continents is Airplane. 4 We developed an ontology as part of the experiment on expressiveness of ontology-development tools at the EON2002 workshop (

17 6. All 5-star hotels have internet connections. 7. For every flight the price for the economy class must be lower than that of a business class. 8. Every route has customers. 9. The customers of a route must have matching expected departure and arrival cities. Table 2 Constraints in English. The subjects attempted to encode all 9 axioms within one hour. 6.3 Evaluation Criteria For each of the axioms, we asked the participants to indicate whether they (1) attempted to encode the axiom and believed they were successful ; (2) attempted to encode the axiom but could not find a suitable template; (3) did not attempt to encode the axiom. For all responses of category 1, we manually examined the knowledge base to determine how many constraints were encoded correctly. Based on the collected data, we used the following evaluation criteria to measure the participants performance: a) Ability to locate a template. We measured how often the participants could find a template for a constraint they attempted to encode. In other words, we divided total number of constraints that participants encoded by the total number of constraints that participants attempted to encode. This ratio demonstrates the subjects ability to locate their desired templates in the library, regardless of whether the templates identified could correctly express the logical statements in mind (we address the issue of correctness in Expectation Accuracy below). b) Expectation Accuracy. We measured the ratio of total number of correct guesses, which is the number of attempts, to look for templates for a constraint EZPAL can formulate and skipping ones EZPAL could not create, to the total number of axioms. This number implies how well the subjects allocated their limited time according to their expectations of the expressiveness of the template library. c) Correctness. We measured the fraction of axioms that the participants encoded that were actually correct. We divided the total number of encoded axioms that were correct by the total number of axioms that the users encoded. This number measures how well the participants understood the templates when using them to create constraint axioms.

18 d) Subjective opinion. We gave subjects usability questionnaires focusing on their opinions about the design of templates, properties, interface layouts, and the strategies that they used to identify templates. 7. EVALUATION RESULTS We now present the results of our evaluation. 7.1 Ability to locate templates The users attempted to encode on average 7.3 (S.D. = 0.94) out of 9 constraints with EZPAL. Of these, they were able to find suitable templates for an average of 5.2 axioms (S.D. = 0.75). These numbers constitute a success rate of 72.6%. Many subjects indicated that they did not have enough time to attempt all the constraints. 7.2 Expectation Accuracy Only seven of the nine constraints given to the subjects could be encoded directly using EZPAL. One of the remaining two axioms required information from a slot several layers of indirection away from the constrained class, and another required two EZPAL expressions due to the structure of the ontology. Of the 7 axioms that could be encoded with EZPAL, the users attempted to encode on average 6.4 times, or 92% correct (S.D. = 0.65). For the two axioms that EZPAL could not encode, the users did not attempt to use EZPAL 1.3 times, or 64% correct (S.D. = 0.47). In fact, overall, the weighted average of expectation accuracy is 86% (S.D. = 0.5). As the results suggest, the users tended to be optimistic by attempting to compose axioms that could not be expressed by any template. In fact, subjects were tricked by the deceptively simple axiom The only possible mode of transportation between cities in different continents is Airplane, which EZPAL could not encode due to the way the ontology was structured. 7.3 Encoded Axiom Correctness Out of the mean 5.2 axioms that each participant encoded, 4.5 or 87% of axioms were correct. Thus, most of the time users were successful in deciphering the relationships between the constraining classes, constraint classes, instances, and slots the templates represent as English sentences.

19 7.4 Expert s Performance in Encoding the Same Constraints Entered directly in PAL We also asked a PAL expert to encode the same set of nine constraints directly in PAL, without using the EZPAL tab. Given the same amount of time, the expert spent 55 minutes and attempted to encode all 9 constraints, 6 of which were correct. 8. DISCUSSION Our work demonstrates the feasibility of a template-based approach for acquiring logical constraints. The subjects, all of whom taught themselves EZPAL, were able to encode correctly 73% of the constraints under significant time pressure, 5 when dealing with unexpected complexities in various parts of the ontology, and with no prior experience with the template library itself. As expected, identifying the correct template, rather than populating the blanks with values, is the most difficult step in the process. Most of the constraints we supplied involved multiple slots and indirections, but users worked quite well within the framework by achieving high scores in the ability to locate templates and to encode axioms correctly. In fact, for the few simple constraints involving one slot or several slots without indirection, the percentage of attempted axioms encoded correctly was 92.9%. The chance of successfully encoding axioms in EZPAL correlates inversely with the difficulty level of the axioms. Comparison of performances of EZPAL users and that of the PAL user, who did not use EZPAL to encode the axioms, also revealed potential benefits of our approach. The PAL expert successfully constructed one constraint out of the two that could not be encoded with EZPAL. Interestingly, if one only considers the constraints EZPAL could express, the correctness levels are almost identical (5 by the expert using PAL directly vs. 4.5 on average by subjects using EZPAL). These results are especially promising, since the subjects, who composed the axioms without any knowledge of PAL, did almost as well as the PAL expert, who had extensive experience with composing PAL axioms. Further analysis of the results shows that 1) the expert spent a significant amount of time checking the syntax and logic of the axioms and 2) two axioms that the expert encoded correctly have a more succinct representation than is possible in EZPAL. EZPAL reduced the time needed for entry of axioms and ensured that the resulting PAL axioms where syntactically correct. In this study, we do not have sufficient data to conclude whether EZPAL users performed better or worse than PAL experts, since we evaluated only one of the latter. Because of the complexity of the language, we simply did not have enough PAL 5 Three out of seven subjects managed to examine all nine constraints before running out of time.

20 experts. In future work, we will conduct a full comparative study of PAL and EZPAL. Some subjects suggested that the wording for a few templates was slightly awkward, but in many cases this awkwardness was an artifact of standardization among sentence structures. This standardization contrasts with the variety of different ways in which the axioms in the libraries in our study expressed the same things. Some users found the standardized sentence structures greatly benefited their search for a working template. Beyond the sentence structures, the examples associated with each axiom provided perhaps the most significant assistance in the search process. Interestingly, opinions about the helpfulness of properties, which were descriptions about the referential patterns of templates, varied. Many users thought the properties were clear but some users also thought the properties were incomprehensible. Some participants did not use properties as a filtering mechanism at all. Some participants suggested that the properties are too abstract, but those who disagreed said that picturing the relationships among frames mentally or actually drawing pictures helped in choosing the properties. Another complaint was that sometimes a property encompasses too many templates; selecting only one property did not reduce the size of the viewable list of templates very much. However, we observed that the users who claimed they were comfortable with the properties were able to encode more axioms successfully than did those who did not make use of the feature. 9. RELATED WORK Research in classifying and representing axioms in a user-friendly way has been relatively sparse in the knowledge base-systems community. Staab and Mädche (Staab and Mäedche 2000) adopt an approach similar to ours: They note that many axioms that they encountered in ontology-engineering projects follow similar patterns. They abstract several classes of axioms, such as algebraic axioms (e.g., reflexivity and symmetry of properties), axioms dealing with composition of relations, axioms describing part whole relations and so on. Then the abstract representation of an axiom (the one that the user sees) is a predicate for which a user only needs to specify arguments. The user does not need to worry about the symbolic representation of the axiom in a specific language. For the limited set of axioms that the authors discuss, the predicate approach works well. However, it is unclear how the same approach would work for slightly different variations of axioms and axioms with varying number of conditions. In addition, for many users a look at the predicate name does not immediately clarify what the underlying axioms are about (e.g., consider the predicate PartonomicRolePropagation ). In the ontology libraries that we considered, most axioms did not fall in the categories identified by the authors. This approach does not provide a path from the axiom templates to a user-friendly interface that

21 naïve users can use to create the axioms. Despite these differences, it is important to see the parallels between this work and ours in that we try to facilitate the user s experience in defining axioms by allowing him to instantiate common patterns rather than write axioms from scratch. Researchers applied the above idea not only to specifying single axioms but also to specifying sets of axioms. In some cases, a set of axioms represents a complex knowledge-base component, such as a description of a physical process (Barker, 2001; Clark, 2000). In other cases, a set of parameterized axioms represents a new constraint that cannot be directly expressed in the corresponding language (Staab et al., 2001). As a result, the task of the knowledge-base designer is greatly simplified since he needs only to instantiate a particular pattern rather than write a set of axioms from scratch. In addition, the user creates the definition at a higher level of abstraction and the system can then translate it into several specific languages if necessary. In our work, we identify the recurring patterns by analyzing a large set of real-world axioms that users have created in their ontologies. The work on Generalized Directive Models (GDM) (van Heijst et al. 1992) also exploits the idea of using patterns for knowledge acquisition. With GDMs, the processes of creating abstract model of the domain and filling it in with domain knowledge are closely intertwined. GDMs use a grammar designed for a specific task, such as diagnosis, to acquire and generate information both for the abstract model and for the domain knowledge. For each of the tasks, a separate grammar must be designed. Tu and Musen (Tu, 2001) observe that most of the axioms required by developers of ontologies of clinical-practice guidelines need to write follow a limited set of patterns. Specifically, they consider axioms for describing decision criteria in clinical guidelines. Instead of writing an axiom in PAL, a guideline author fills in the form that looks exactly as a form for any regular instance in the knowledge base. For example, there is a check box to mark whether the user is looking for presence or absence of a particular criterion; a box to select from the list of time periods; a box to select a value from the list of conditions, and so on. Once the user fills in the form, the engine automatically generates an axiom in PAL. Again, in this approach, the user can specify a large fraction of axioms simply by performing a fairly intuitive task filling in blanks in a form. However, the templates are specific to defining guideline criteria. To use them for other domains, one will need to create new sets of templates. In our work, we studied ontologies from many different domains, to create domain-independent templates. Our methodology however can also be used in domain-specific context: a study of ontologies in a specific domain can produce a set of templates specific for the domain. The tools that we describe in this paper can then be used to assist users in creating axioms based on these domain-specific templates. Design patterns in object-oriented programming (Gamma, 1999) is another approach to using recurring representation patterns as the building blocks of design. A design patterns records,

22 explains, and evaluates patterns of code in object-oriented systems. Similar to our templates, each design pattern also encodes information for structuring, creating, and managing the design idioms. Our work also has some parallels with case-based reasoning (Leake, 1996). In case-based reasoning, a problem is solved by comparing it to a set of existing cases for which we know the solution. The case-based reasoning paradigm has been used in many different domains for diagnosis, design, planning, and so on. The underlying assumption in CBR is that situations repeat themselves and rather than solve a problem from scratch, we can look for a similar case that has occurred previously. In a similar way, our research suggests that in many cases, even such tasks as axiom design in knowledge-base systems, can be described by a set of patterns that have previously occurred. Where in a case-based reasoning setting, the system matches the new case to its library of cases, in EZPAL the user matches the axiom that he wants to express to the library of templates and adjusts the template to match the axiom more closely. And just as CBR will not work in every possible situation but has proved to be extremely useful and practical in plethora of applications, we believe that our methodology for defining logical axioms will enable many users to augment their knowledge bases with additional knowledge. 10. SUMMARY We developed an axiom-modeling methodology whereby we identified templates based on structural repetitions among existing axioms, and properties which were relational abstractions of the templates. Using the templates, which are available as fill-in-the-blank English sentences, users can fill in values to create axioms without previous knowledge of the underlying axiom language. We created a Protégé knowledge base of templates and properties derived from the constraint axioms in Protégé and Ontolingua ontologies. We also developed a constraint acquisition tab which allows users to filter the template list with known properties of the goal axiom before users identify and instantiate the associated template. Results of the user studies showed that even though a user s ontology layout affects coverage of the templates and the concept of property based on structural abstraction of templates required more time to understand, the approach demanded only a short training period and allowed its users to encode basic and intermediate axioms quickly and accurately. ACKNOWLEDGEMENTS We are very grateful to Samson Tu, Mor Peleg and Iwei Yeh for sharing their Protégé ontologies. We would also like to thank Monica Crubézy for suggestions about template designs and interface evaluation and Carolyn Ross for suggestions about writing. This work was supported in

23 part by a contract from the U.S. National Cancer Institute. Johnson Hou was supported by the Stanford President s Scholar Intellectual Exploration Grant for lower-division undergraduate students. This work was conducted using the Protégé resource, which is supported by grant LM from the United States National Library of Medicine. REFERENCES Barker, K., Clark, P. and Porter, B. A Library of Generic Concepts for Composing Knowledge Bases. In: the First International Conference on Knowledge Capture (K-Cap'01). Victoria, BC, Clark P., Thompson, J., and Porter, B. Knowledge Patterns. In: the Seventh International Conference on Principles of Knowledge Representation and Reasoning(KR-2000), pages , Eds: A. Cohn, F. Giunchiglia, B. Selman, CA:Kaufmann, Chaudhri, V.K., Farquhar, A., Fikes, R., Karp, P.D. and Rice, J.P. OKBC: A programmatic foundation for knowledge base interoperability. In: the Fifteenth National Conference on Artificial Intelligence (AAAI-98). Madison, Wisconsin: AAAI Press/The MIT Press, Gamma, E., Helm, R., Johnson, R. and Vlissides, J., Design Patterns: Elements of Reusable Object Oriented Software. Reading, MA: Addison-Wesley, Genesereth, M.R. and Fikes, R.E., Knowledge Interchange Format, Version 0.3,Reference Manual, Knowledge Systems Laboratory, Stanford University, Heijst, G.v., Terpstra, P., Wielinga, B. and Shadbolt, N. (1992). Generalised Directive Models. In: Proceedings of the 7th Banff Knowledge Acquisition for Knowledge-Based Systems Workshop, Banff, Alberta, Canada. Hou, C.J., Noy, N.F., Musen, M. A., A Template-Based Approach Toward Acquisition of Logical Sentences. In: the Proceedings of the Conference on Intelligent Information Processing (IIP-2002), Montreal, Canada: Kluwer Academic Publisher, Karp, P.D., The design space of frame knowledge representation systems, SRI AI Center: Menlo Park, CA, Leake, D., Case-Based Reasoning: Experiences, Lessons, and Future Directions. AAAI Press, Ontolingua, Ontolingua System Reference Manual, Knowledge Systems Lab, Stanford University, Protégé, The Protege Project, Stanford Medical Informatics, Stanford University, Staab, S. and Mäedche, A. Axioms are objects, too - Ontology Engineering Beyond the Modeling of Concepts and Relations. In: the 14th European Conference on Artificial Intelligence (ECAI 2000), Workshop on Ontologies and Problem-Solving Methods. Berlin, Staab, S. Erdmann M., Mäedche A. Engineering ontologies using semantic patterns. In: A. Preece & D. O'Leary

24 (eds.), Proceedings of the Seventeenth International Joint Conference on Artificial Intelligence(IJCAI-01), Workshop on E-Business & the Intelligent Web. Seattle, Tu, S.W. and Musen, M.A. Modeling Data and Knowledge in the EON Guideline Architecture. In: MedInfo London, 2001.