Robust Reading: Identification and Tracing of Ambiguous Names

Transcription

1 Robust Reaing: Ientification an Tracing of Ambiguous Names Xin Li Paul Morie Dan Roth Department of Computer Science University of Illinois, Urbana, IL Abstract A given entity, representing a person, a location or an organization, may be mentione in text in multiple, ambiguous ways. Unerstaning natural language requires ientifying whether ifferent mentions of a name, within an across ocuments, represent the same entity. We evelop an unsupervise learning approach that is shown to resolve accurately the name ientification an tracing problem. At the heart of our approach is a generative moel of how ocuments are generate an how names are sprinkle into them. In its most general form, our moel assumes: (1) a joint istribution over entities, (2) an author moel, that assumes that at least one mention of an entity in a ocument is easily ientifiable, an then generates other mentions via (3) an appearance moel, governing how mentions are transforme from the representative mention. We show how to estimate the moel an o inference with it an how this resolves several aspects of the problem from the perspective of applications such as questions answering. 1 Introuction Reaing an unerstaning text is a task that requires the ability to isambiguate at several levels, abstracting away etails an using backgroun knowlege in a variety of ways. One of the ifficulties that humans resolve instantaneously an unconsciously is that of reaing names. Most names of people, locations, organizations an others, have multiple writings that are use freely within an across ocuments. The variability in writing a given concept, along with the fact that ifferent concepts may have very similar writings, poses a significant challenge to progress in natural language processing. Consier, for example, an open omain question answering system (Voorhees, 2002) that attempts, given a question like: When was Presient Kenney born? to search a large collection of articles in orer to pinpoint the concise answer: on May 29, The sentence, an even the ocument that contains the answer, may not contain the name Presient Kenney ; it may refer to this entity as Kenney, JFK or John Fitzgeral Kenney. Other ocuments may state that John F. Kenney, Jr. was born on November 25, 1960, but this fact refers to our target entity s son. Other mentions, such as Senator Kenney or Mrs. Kenney are even closer to the writing of the target entity, but clearly refer to ifferent entities. Even the statement John Kenney, born turns out to refer to a ifferent entity, as one can tell observing that the ocument iscusses Kenney s batting statistics. A similar problem exists for other entity types, such as locations, organizations etc. A hoc solutions to this problem, as we show, fail to provie a reliable an accurate solution. This paper presents the first attempt to apply a unifie approach to all major aspects of this problem, presente here from the perspective of the question answering task: (1) Entity Ientity - o mentions A an B (typically, occurring in ifferent ocuments, or in a question an a ocument, etc.) refer to the same entity? This problem requires both ientifying when ifferent writings refer to the same entity, an when similar or ientical writings refer to ifferent entities. (2) Name Expansion - given a writing of a name (say, in a question), fin other likely writings of the same name. (3) Prominence - given question What is Bush s foreign policy?, an given that any large collection of ocuments may contain several Bush s, there is a nee to ientify the most prominent, or relevant Bush, perhaps taking into account also some contextual information. At the heart of our approach is a global probabilistic view on how ocuments are generate an how names (of ifferent entity types) are sprinkle into them. In its most general form, our moel assumes: (1) a joint istribution over entities, so that a ocument that mentions Presient Kenney is more likely to mention Oswal or White House than Roger Clemens ; (2) an author moel, that makes sure that at least one mention of a name in a ocument is easily ientifiable, an then generates other mentions via (3) an appearance moel, governing how mentions are transforme from the rep-

2 resentative mention. Our goal is to learn the moel from a large corpus an use it to support robust reaing - enabling on the fly ientification an tracing of entities. This work presents the first stuy of our propose moel an several relaxations of it. Given a collection of ocuments we learn the moels in an unsupervise way; that is, the system is not tol uring training whether two mentions represent the same entity. We only assume the ability to recognize names, using a name entity recognizer run as a preprocessor. We efine several inferences that correspon to the solutions we seek, an evaluate the moels by performing these inferences against a large corpus we annotate. Our experimental results suggest that the entity ientity problem can be solve accurately, giving accuracies (F 1 ) close to 90%, epening on the specific task, as oppose to 80% given by state of the art a-hoc approaches. Previous work in the context of question answering has not aresse this problem. Several works in NLP an Databases, though, have aresse some aspects of it. From the natural language perspective, there has been a lot of work on the relate problem of coreference resolution (Soon et al., 2001; Ng an Carie, 2003; Kehler, 2002) - which aims at linking occurrences of noun phrases an pronouns within a ocument base on their appearance an local context. (Charniak, 2001) presents a solution to the problem of name structure recognition by incorporating coreference information. In the context of atabases, several works have looke at the problem of recor linkage - recognizing uplicate recors in a atabase (Cohen an Richman, 2002; Hernanez an Stolfo, 1995; Bilenko an Mooney, 2003). Specifically, (Pasula et al., 2002) consiers the problem of ientity uncertainty in the context of citation matching an suggests a probabilistic moel for that. Some of very few works we are aware of that works irectly with text ata an across ocuments, are (Bagga an Balwin, 1998; Mann an Yarowsky, 2003), which consier one aspect of the problem that of istinguishing occurrences of ientical names in ifferent ocuments, an only of people. The rest of this paper is organize as follows: We formalize the robust reaing problem in Sec. 2. Sec. 3 escribes a generative view of ocuments creation an three practical probabilistic moels esigne base on it, an iscusses inference in these moels. Sec. 4 illustrates how to learn these moels in an unsupervise setting, an Sec. 5 escribes the experimental stuy. Sec. 6 conclues. 2 Robust Reaing We consier reaing a collection of ocuments D = { 1, 2,..., m }, each of which may contain mentions (i.e. real occurrences) of T types of entities. In the current evaluation we consier T = {P erson, Location, Organization}. An entity refers to the real concept behin a mention an can be viewe as a unique ientifier to a real-worl object. Examples might be the person John F. Kenney who became a presient, White House the resience of the US presients, etc. E enotes the collection of all possible entities in the worl an E = {e i }l 1 is the set of entities mentione in ocument. M enotes the collection of all possible mentions an M = {m i }n 1 is the set of mentions in ocument. Mi (1 i l ) is the set of mentions that refer to entity e i E. For entity John F. Kenney, the corresponing set of mentions in a ocument may contain Kenney, J. F. Kenney an Presient Kenney. Among all mentions of an entity e i in ocument we istinguish the one occurring first, ri Mi, as the representative of e i. In practice, ri is usually the longest mention of e i in the ocument as well, an other mentions are variations of it. Representatives are viewe as a typical representation of an entity mentione in a specific time an place. For example, Presient J.F.Kenney an Congressman John Kenney may be representatives of John F. Kenney in ifferent ocuments. R enotes the collection of all possible representatives an R = {ri }l 1 M is the set of representatives in ocument. This way, each ocument is represente as the collection of its entities, representatives an mentions = {E, R, M }. Elements in the name space W = E R M each have an ientifying writing (enote as wrt(n) for n W ) 1 an an orere list of attributes, A = {a 1,..., a p }, which epens on the entity type. Attributes use in the current evaluation inclue both internal attributes, such as, for People, {title, firstname, milename, lastname, gener} as well as contextual attributes such as {time, location, proper-names}. Proper-names refer to a list of proper names that occur aroun the mention in the ocument. All attributes are of string value an the values coul be missing or unknown 2. The funamental problem we aress in robust reaing is to ecie what entities are mentione in a given ocument (given the observe set M ) an what the most likely assignment of entity to each mention is. 3 A Moel of Document Generation We efine a probability istribution over ocuments = {E, R, M }, by escribing how ocuments are being generate. In its most general form the moel has the following three components: (1) A joint probability istribution P (E ) that governs 1 The observe writing of a mention is its ientifying writing. For entities, it is a stanar representation of them, i.e. the full name of a person. 2 Contextual attributes are not part of the current evaluation, an will be evaluate in the next step of this work.

3 John Fitzgeral Kenney e E House of Representatives E = argmax E EP (E, R M, θ) (1) = argmax E EP (E, R, M θ), (2) John Fitzgeral Kenney M i e i Presient John F. Kenney r i {Presient Kenney, Kenney, JFK} E House of Representatives R House of Representatives M {House of Representatives, The House} Figure 1: Generating a ocument how entities (of ifferent types) are istribute into a ocument an reflects their co-occurrence epenencies. (2) The number of entities in a ocument, size(e ), an the number of mentions of each entity in E, size(mi ), nee to be ecie. The current evaluation makes the simplifying assumption that these numbers are etermine uniformly over a small plausible range. (3) The appearance probability of a name generate (transforme) from its representative is moelle as a prouct istribution over relational transformations of attribute values. This moel captures the similarity between appearances of two names. In the current evaluation the same appearance moel is use to calculate both the probability P (r e) that generates a representative r given an entity e an the probability P (m r) that generates a mention m given a representative r. Attribute transformations are relational, in the sense that the istribution is over transformation types an inepenent of the specific names. Given these, a ocument is assume to be generate as follows (see Fig. 1): A set of size(e ) entities E E is selecte to appear in a ocument, accoring to P (E ). For each entity e i E, a representative ri R is chosen accoring to P (r i e i ), generating R. Then mentions Mi of an entity are generate from each representative ri R each mention m j M i is inepenently transforme from ri accoring to the appearance probability P (m j r i ). Assuming conitional inepenency between M an E given R, the probability istribution over ocuments is therefore P () = P (E, R, M ) = P (E )P (R E )P (M R ), an the probability of the ocument collection D is: P (D) = D P (). Given a mention m in a ocument (M is the set of observe mentions in ), the key inference problem is to etermine the most likely entity e m that correspons to it. This is one by computing: where θ is the learne moel s parameters. This gives the assignment of the most likely entity e m for m. 3.1 Relaxations of the Moel In orer to simplify moel estimation an to evaluate some assumptions, several relaxations are mae to form three simpler probabilistic moels. Moel I: (the simplest moel) The key relaxation here is in losing the notion of an author rather than first choosing a representative for each ocument, mentions are generate inepenently an irectly given an entity. That is, an entity e i is selecte from E accoring to the prior probability P (e i ); then its actual mention m i is selecte accoring to P (m i e i ). Also, an entity is selecte into a ocument inepenently of other entities. In this way, the probability of the whole ocument set can be compute simply as follows: n P (D) = P ({(e i, m i )} n i=1) = P (e i )P (m i e i ), i=1 an the inference problem for the most likely entity given m is: e m = argmax e EP (e m, θ) = argmax e EP (e)p (m e). (3) Moel II: (more expressive) The major relaxation mae here is in assuming a simple moel of choosing entities to appear in ocuments. Thus, in orer to generate a ocument, after we ecie size(e ) an {size(m1, size(m2 ),... } accoring to uniform istributions, each entity e i is selecte into inepenently of others accoring to P (e i ). Next, the representative r i for each entity e i is selecte accoring to P (r i e i ) an for each representative the actual mentions are selecte inepenently accoring to P (m j r j ). Here, we have iniviual ocuments along with representatives, an the istribution over ocuments is: P () = P (E, R, M ) = P (E )P (R E )P (M R ) E [P (e i )P (r i e i )] P (mj r j ) i=1 (r j,m j ) after we ignore the size components (they o not influence inferences). The inference problem here is the same as in Equ. (2). Moel III: This moel performs the least relaxation. After eciing size(e ) accoring to a uniform istribution, instea of assuming inepenency among entities which oes not hol in reality (For example, Gore an George. W. Bush occur together frequently, but Gore an Steve. Bush o not), we select entities using a graph base algorithm: entities in E are viewe

4 as noes in a weighte irecte graph with eges (i, j) labelle P (e j e i ) representing the probability that entity e j is chosen into a ocument that contains entity e i. We istribute entities to E via a ranom walk on this graph starting from e 1 with a prior probability P (e i ). Representatives an mentions are generate in the same way as in Moel II. Therefore, a more general moel for the istribution over ocuments is: E P () P (e 1 )P (r 1 e 1 ) [P (e i e i 1 )P (r i e i )] P (mj r j ). i=2 (r j,m j ) The inference problem is the same as in Equ. (2). 3.2 Inference Algorithms The funamental problem in robust reaing can be solve as inference with the moels: given a mention m, seek the most likely entity e E for m accoring to Equ. (3) for Moel I or Equ. (2) for Moel II an III. Instea of all entities in the real worl, E can be viewe without loss as the set of entities in a close ocument collection that we use to train the moel parameters an it is known after training. The inference algorithm for Moel I (with time complexity O( E )) is simple an irect: just compute P (e, m) for each caniate entity e E an then choose the one with the highest value. Due to exponential number of possible assignments of E, R to M in Moel II an III, precise inference is infeasible an approximate algorithms are therefore esigne: In Moel II, we aopt a two-step algorithm: First, we seek the representatives R for the mentions M in ocument by sequentially clustering the mentions accoring to the appearance moel. The first mention in each group is chosen as the representative. Specifically, when consiering a mention m M, P (m r) is compute for each representative r that have alreay been create an a fixe threshol is then use to ecie whether to create a new group for m or to a it to one of the existing groups with the largest P (m r). In the secon step, each representative ri R is assigne to its most likely entity accoring to e = argmax e E P (e) P (r e). This algorithm has a time complexity of O(( M + E ) M ). Moel III has a similar algorithm as Moel II. The only ifference is that we nee to consier the global epenency between entities. Thus in the secon step, instea of seeking an entity e for each representative r separately, we etermine a set of entities E for R in a Hien Markov Moel with entities in E as hien states an R as observations. The prior probabilities, the transitive probabilities an the observation probabilities are given by P (e), P (e j e i ) an P (r e) respectively. Here we seek the most likely sequence of entities given those representatives in their appearing orer using the Viterbi algorithm. The total time complexity is e1= George Bush e2= George W. Bush e3= Steve Bush 1 m1,r1=presient Bush m2=bush m3=j. Quayle Entities E 2 m4,r2=steve Bush m5=bush Figure 2: An conceptual example. The arrows represent the correct assignment of entities to mentions. r 1, r 2 are representatives. O( M 2 + E 2 M ). The E 2 component can be simplifie by filtering out unlikely entities for a representative accoring to their appearance similarity. 3.3 Discussion Besies ifferent assumptions, some funamental ifferences exist in inference with the moels as well. In Moel I, the entity of a mention is etermine completely inepenently of other mentions, while in Moel II, it relies on other mentions in the same ocument for clustering. In Moel III, it is not only relate to other mentions but to a global epenency over entities. The following conceptual example illustrates those ifferences as in Fig. 2. Example 3.1 Given E = {George Bush, George W. Bush, Steve Bush}, ocuments 1, 2 an 5 mentions in them, an suppose the prior probability of entity George W. Bush is higher than those of the other two entities, the entity assignments to the five mentions in the moels coul be as follows: For Moel I, mentions(e 1 ) = φ, mentions(e 2 ) = {m 1, m 2, m 5 } an mentions(e 3 ) = {m 4 }. The result is cause by the fact that a mention tens to be assigne to the entity with higher prior probability when the appearance similarity is not istinctive. For Moel II, mentions(e 1) = φ, mentions(e 2) = {m 1, m 2} an mentions(e 3) = {m 4, m 5}. Local epenency (appearance similarity) between mentions insie each ocument enforces the constraint that they shoul refer to the same entity, like Steve Bush an Bush in 2. For Moel III, mentions(e 1 ) = {m 1, m 2 }, mentions(e 2 ) = φ, mentions(e 3 ) = {m 4, m 5 }. With the help of global epenency between entities, for example, George Bush an J. Quayle, an entity can be istinguishe from another one with a similar writing. 3.4 Other Tasks Other aspects of Robust Reaing can be solve base on the above inference problem. Entity Ientity: Given two mentions m 1 1, m 2 2, etermine whether they correspon to the same entity by: m 1 m 2 argmax e E P (e, m 1 ) = argmax e E P (e, m 2 )

5 for Moel I an m 1 m 2 argmax e EP (E 1, R 1, M 1 ) = argmax e E P (E 2, R 2, M 2 ). for Moel II an III. Name Expansion: Given a mention m q in a query q, ecie whether mention m in the ocument collection D is a legal expansion of m q : m q m e m q = argmax e EP (E q, R q, M q ) & m mentions(e ). Here it s assume that we alreay know the possible mentions of e after training the moels with D. Prominence: Given a name n W, the most prominent entity for n is given by (P (e) is given by the prior istribution P E an P (n e) is given by the appearance moel.): e = argmax e E P (e)p (n e). 4 Learning the Moels Confine by the labor of annotating ata, we learn the probabilistic moels in an unsupervise way given a collection of ocuments; that is, the system is not tol uring training whether two mentions represent the same entity. A greey search algorithm moifie after the stanar EM algorithm (We call it Truncate EM algorithm) is aopte here to avoi complex computation. Given a set of ocuments D to be stuie an the observe mentions M in each ocument, this algorithm iteratively upates the moel parameter θ (several unerlying probabilistic istributions escribe before) an the structure (that is, E an R ) of each ocument. Different from the stanar EM algorithm, in the E-step, it seeks the most likely E an R for each ocument rather than the expecte assignment. 4.1 Truncate EM Algorithm The basic framework of the Truncate EM algorithm to learn Moel II an III is as follows: 1. In the initial (I-) step, an initial (E 0, R 0) is assigne to each ocument by an initialization algorithm. After this step, we can assume that the ocuments are annotate with D 0 = {(E 0, R 0, M )}. 2. In the M-step, we seek the moel parameter θ t+1 that maximizes P (D t θ). Given the labels supplie in the previous I- or E-step, this amounts to the maximum likelihoo estimation. (to be escribe in Sec. 4.3). 3. In the E-step, we seek (E t+1, R t+1) for each ocument that maximizes P (D t+1 θ t+1) where D t+1 = {(E t+1, R t+1, M )}. It s the same inference problem as in Sec Stopping Criterion: If no increase is achieve over P (D t θ t ), the algorithm exits. Otherwise the algorithm will iterate over the M-step an E-step. The algorithm for Moel I is similar to the above one, but much simpler in the sense that it oes not have the notions of ocuments an representatives. So in the E-step we only seek the most likely entity e for each mention m D, an this simplifies the parameter estimation in the M-step accoringly. It usually takes 3 10 iterations before the algorithms stop in our experiments. 4.2 Initialization The purpose of the initial step is to acquire an initial guess of ocument structures an the set of entities E in a close collection of ocuments D. The hope is to fin all entities without loss so uplicate entities are allowe. For all the moels, we use the same algorithm: A local clustering is performe to group mentions insie each ocument: simple heuristics are applie to calculating the similarity between mentions; an pairs of mentions with similarity above a threshol are then clustere together. The first mention in each group is chosen as the representative (only in Moel II an III) an an entity having the same writing with the representative is create for each cluster 3. For all the moels, the set of entities create in ifferent ocuments become the global entity set E in the following M- an E-steps. 4.3 Estimating the Moel Parameters In the learning process, assuming ocuments have alreay been annotate D = {(e, r, m)} n 1 from previous I- or E-step, several unerlying probability istributions of the relaxe moels are estimate by maximum likelihoo estimation in each M-step. The moel parameters inclue a set of prior probabilities for entities P E, a set of transitive probabilities for entity pairs P E E (only in Moel III) an the appearance probabilities P W W of each name in the name space W being transforme from another. The prior istribution P E is moelle as a multinomial istribution. Given a set of labelle entitymention pairs {(e i, m i )} n 1, P (e) = freq(e) n where freq(e) enotes the number of pairs containing entity e. Given all the entities appearing in D, the transitive probability P (e e) is estimate by P (e 2 e 1) P (wrt(e 2) wrt(e 1)) = oc# (wrt(e 2), wrt(e 1)). oc # (wrt(e 1 )) Here, the conitional probability between two realworl entities P (e 2 e 1 ) is backe off to the one between the ientifying writings of the two entities P (wrt(e 2 ) wrt(e 1 )) in the ocument set D to avoi 3 Note that the performance of the initialization algorithm is 97.3% precision an 10.1% recall (measures are efine later.)

6 sparsity problem. oc # (w 1, w 2,...) enotes the number of ocuments having the co-occurrence of writings w 1, w 2,... Appearance probability, the probability of one name being transforme from another, enote as P (n 2 n 1 ) (n 1, n 2 W ), is moelle as a prouct of the transformation probabilities over attribute values 4. The transformation probability for each attribute is further moelle as a multi-nomial istribution over a set of preetermine transformation types: T T = {copy, missing, typical, non typical} 5. Suppose n 1 = (a 1 = v 1, a 2 = v 2,..., a p = v p ) an n 2 = (a 1 = v 1, a 2 = v 2,..., a p = v p) are two names belonging to the same entity type, the transformation probabilities P M R, P R E an P M E, are all moelle as a prouct istribution (naive Bayes) over attributes: P (n 2 n 1 ) = Π p k=1 P (v k v k ). We manually collecte typical an non-typical transformations for attributes such as titles, first names, last names, organizations an locations from multiple sources such as U.S. government census an online ictionaries. For other attributes like gener, only copy transformation is allowe. The maximum likelihoo estimation of the transformation probability P (t, k) (t T T, a k A) from annotate representative-mention pairs {(r, m)} n 1 is: P (t, k) = freq(r, m) : vr k t v m k n vk r t vk m enotes the transformation from attribute a k of r to that of m is of type t. Simple smoothing is performe here for unseen transformations. 5 Experimental Stuy Our experimental stuy focuses on (1) evaluating the three moels on ientifying three entity types (People, Locations, Organization); (2) comparing our inuce similarity measure between names (the appearance moel) with other similarity measures; (3) evaluating the contribution of the global nature of our moel, an finally, (4) evaluating our moels on name expansion an prominence ranking. 5.1 Methoology We ranomly selecte 300 ocuments from New York Times articles in the TREC corpus (Voorhees, 4 The appearance probability can be moelle ifferently by using other string similarity between names. We will compare the moel escribe here with some other non-learning similarity metrics later. 5 copy enotes v k is exactly the same as v k ; missing enotes missing value for v k; typical enotes v k is a typical variation of v k, for example, Prof. for Professor, Any for Anrew ; non-typical enotes a non-typical transformation. (4) 2002). The ocuments were annotate by a name entity tagger for People, Locations an Organizations. The annotation was then correcte an each name mention was labelle with its corresponing entity by two annotators. In total, about 8, 000 mentions of name entities which correspon to about 2, 000 entities were labelle. The training process gets to see only the 300 ocuments an extracts attribute values for each mention. No supervision is supplie. These recors are use to learn the probabilistic moels. In the 64 million possible mention pairs, most are trivial non-matching one the appearances of the two mentions are very ifferent. Therefore, irect evaluation over all those pairs always get almost 100% accuracy in our experiments. To avoi this, only the 130, 000 pairs of matching mentions that correspon to the same entity are use to evaluate the performance of the moels. Since the probabilistic moels are learne in an unsupervise setting, testing can be viewe simply as the evaluation of the learne moel, an is thus one on the same ata. The same setting was use for all moels an all comparison performe (see below). To evaluate the performance, we pair two mentions iff the learne moel etermine that they correspon to the same entity. The list of preicte pairs is then compare with the annotate pairs. We measure Precision (P ) Percentage of correctly preicte pairs, Recall (R) Percentage of correct pairs that were preicte, an F 1 = 2P R P +R. Comparisons: The appearance moel inuces a similarity measure between names, which is estimate uring the training process. In orer to unerstan whether the behavior of the generative moel is ominate by the quality of the inuce pairwise similarity or by the global aspects (for example, inference with the ai of the ocument structure), we (1) replace this measure by two other local similarity measures, an (2) compare three possible ecision mechanisms pairwise classification, straightforwar clustering over local similarity, an our global moel. To obtain the similarity require by pairwise classification an clustering, we use this formula sim a (n 1, n 2 ) = P (n 1 n 2 ) to convert the appearance probability escribe in Sec. 4.3 to it. The first similarity measure we use is a simple baseline approach: two names are similar iff they have ientical writings (that is, sim b (n 1, n 2 ) = 1 if n 1, n 2 are ientical or 0 otherwise). The secon one is a state-of-art similarity measure sim s (n 1, n 2 ) [0, 1] for entity names (SoftTFIDF with Jaro-Winkler istance an θ = 0.9); it was ranke the best measure in a recent stuy (Cohen et al., 2003). Pairwise classification is one by pairing two mentions iff the similarity between them is above a fixe threshol. For Clustering, a graph-base clustering al-

7 All(P/L/O) Ientity SoftTFIDF Appearance Pairwise 70.7 (64.7/64.1/83.7) 82.1 (79.9/77.3/89.5) 81.5 (83.6/70.9/90.7) Clustering 70.7 (64.7/64.1/83.7) 79.8 (70.6/76.7/91.0) 79.6 (70.9/76.1/91.0) Moel II 70.7 (64.7/64.1/83.7) 82.5 (79.8/77.4/90.2) 89.0 (92.7/81.9/92.9) Table 1: Comparison of ifferent ecision levels an similarity measures. Three similarity measures are evaluate (rows) across three ecision levels (columns). Performance is evaluate by the F 1 values over the whole test set. The first number averages all entity types; numbers in parentheses represent People, Location an Organization respectively. gorithm is use. Two noes in the graph are connecte if the similarity between the corresponing mentions is above a threshol. In evaluation, any two mentions belonging to the same connecte component are paire the same way as we i in Sec. 5.1 an all those pairs are then compare with the annotate pairs to calculate Precision, Recall an F 1. Finally, we evaluate the baseline an the SoftTFIDF measure in the context of Moel II, where the appearance moel is replace. We foun that the probabilities irectly converte from the SoftTFIDF similarity behave baly so we aopt this formula P (n 1 n 2 ) = e 10 sim s(n 1,n 2 ) 1 e 10 1 instea to acquire P (n 1 n 2 ) neee by Moel II. Those probabilities are fixe as we estimate other moel parameters in training. 5.2 Results The bottom line result is given in Tab. 1. All the similarity measures are compare in the context of the three levels of ecisions local ecision (pairwise), clustering an our probabilistic moel II. Only the best results in the experiments, achieve by trying ifferent threshols in pairwise classification an clustering, are shown. The behavior across rows inicates that, locally, our unsupervise learning base appearance moel is about the same as the state-of-the-art SoftTFIDF similarity. The behavior across columns, though, shows the contribution of the global moel, an that the local appearance moel behaves better with it than a fixe similarity measure oes. A secon observation is that the Location appearance moel is not as goo as the one for People an Organization, probably ue to the attribute transformation types chosen. Tab. 2 presents a more etaile evaluation of the ifferent approaches on the entity ientity task. All the three probabilistic moels outperform the iscriminatory approaches in this experiment, an inication of the effectiveness of the generative moel. We note that although Moel III is more expressive an reasonable than moel II, it oes not always perform better. Inee, the global epenency among entities in Moel III achieves two-fole outcomes: it achieves better precision, but may egrae the recall. The following example, taken from the corpus, illustrates the avantage of this moel. Entity Type Mo InDoc InterDoc All F 1 (%) F 1 (%) R(%) P(%) F 1 (%) All Entities B D I II III People B D I II III Location B D I II III Organization B D I II III Table 2: Performance of ifferent approaches over all test examples. B, D, I, II an III enote the baseline moel, the SoftTFIDF similarity moel with clustering, an the three probabilistic moels. We istinguish between pairs of mentions that are insie the same ocument (InDoc, 15% of the pairs) or not (InterDoc). Example 5.1 Sherman Williams is mentione along with the baseball team Dallas Cowboys in 8 out of 300 ocuments, while Jeff Williams is mentione along with LA Dogers in two ocuments. In all moels but Moel III, Jeff Williams is juge to correspon to the same entity as Sherman Williams since their appearances are similar an the prior probability of the latter is higher than the former. Only Moel III, ue to the co-occurring epenency between Jeff Williams an Dogers, ientifies it as corresponing to an entity ifferent from Sherman Williams. While this shows that Moel III achieves better precision, the recall may go own. The reason is that global epenencies among entities enforces restrictions over possible grouping of similar mentions; in aition, with a limite ocument set, estimating this global epenency is inaccurate, especially when the entities themselves nee to be foun when training the moel. Har Cases: To analyze the experimental results further, we evaluate separately two types of harer cases of the entity ientity task: (1) mentions with ifferent writings that refer to the same entity; an (2) mentions with similar writings that refer to ifferent entities. Moel II an III outperform other moels in those two cases as well. Tab. 3 presents F 1 performance of ifferent approaches in the first case. The best F 1 value is only 73.1%, inicating that appearance similarity an global epenency are not sufficient to solve this problem when the writings are very ifferent. Tab. 4 shows the performance of ifferent approaches for isambiguating similar writings that correspon to ifferent entities. Both these cases exhibit the ifficulty of the problem, an that our approach provies a significant improvement over the state of the art similarity measure column D vs. column II in Tab. 4. It also shows that it is necessary to use contextual attributes of the names, which are not yet inclue in this evaluation.

8 Moel B D I II III Peop Loc Org All Table 3: Ientifying ifferent writings of the same entity (F 1 ). We filter out ientical writings an report only on cases of ifferent writings of the same entity. The test set contains 46, 376 matching pairs (but in ifferent writings) in the whole ata set. Moel B D I II III Peop Loc Org All Table 4: Ientifying similar writings of ifferent entities(f 1 ). The test set contains 39, 837 pairs of mentions that associate with ifferent entities in the 300 ocuments an have at least one token in common. 5.3 Other Tasks In the following experiments, we evaluate the generative moel on other tasks relate to robust reaing. We present results only for Moel II, the best one in previous experiments. Name Expansion: Given a mention m in a query, we fin the most likely entity e E for m using the inference algorithm as escribe in Sec All unique mentions of the entity in the ocuments are output as the expansions of m. The accuracy for a given mention is efine as the percentage of correct expansions output by the system. The average accuracy of name expansion of Moel II is shown in Tab. 5. Here is an example: Query: Who is Gore? Expansions: Vice Presient Al Gore, Al Gore, Gore. Prominence Ranking: We refer to Example 3.1 an use it to exemplify quantitatively how our system supports prominence ranking. Given a query name n, the ranking of the entities with regar to the value of P (e) P (n e) (shown in brackets) by Moel II is as follows. Input: George Bush 1. George Bush (0.0448) 2. George W. Bush (0.0058) Input: Bush 1. George W. Bush (0.0047) 2. George Bush (0.0015) 3. Steve Bush (0.0002) 6 Conclusion an Future Work This paper presents an unsupervise learning approach to several aspects of the robust reaing problem crossocument ientification an tracing of ambiguous names. We evelope a moel that escribes the natural generation process of a ocument an the process of how Entity Type People Location Organization Accuracy(%) Table 5: Accuracy of name expansion. Accuracy is average over 30 ranomly chosen queries for each entity type. names are sprinkle into them, taking into account epenencies between entities across types an an author moel. Several relaxations of this moel were evelope an stuie experimentally, an compare with a stateof-the-art iscriminative moel that oes not take a global view. The experiments exhibit encouraging results an the avantages of our moel. This work is a preliminary exploration of the robust reaing problem. There are several critical issues that our moel can support, but were not inclue in this preliminary evaluation. Some of the issues that will be inclue in future steps are: (1) integration with more contextual information (like time an place) relate to the target entities, both to support a better moel an to allow temporal tracing of entities; (2) stuying an incremental approach of training the moel; that is, when a new ocument is observe, coming, how to upate existing moel parameters? (3) integration of this work with other aspects of general coreference resolution (e.g., other terms like pronouns that refer to an entity) an name entity recognition (which we now take as given); an (4) scalability issues in applying the system to large corpora. Acknowlegments This research is supporte by NSF grants ITR-IIS , ITR-IIS an IIS an an ONR MURI Awar. References A. Bagga an B. Balwin Entity-base cross-ocument coreferencing using the vector space moel. In ACL. M. Bilenko an R. Mooney Aaptive uplicate etection using learnable string similarity measures. In KDD. E. Charniak Unsupervise learning of name structure from coreference atal. In NAACL. W. Cohen an J. Richman Learning to match an cluster large high-imensional ata sets for ata integration. In KDD. W. Cohen, P. Ravikumar, an S. Fienberg A comparison of string metrics for name-matching tasks. In IIWeb Workshop M. Hernanez an S. Stolfo The merge/purge problem for large atabases. In SIGMOD. A. Kehler Coherence, Reference, an the Theory of Grammar. CSLI Publications. G. Mann an D. Yarowsky Unsupervise personal name isambiguation. In CoNLL. V. Ng an C. Carie Improving machine learning approaches to coreference resolution. In ACL. H. Pasula, B. Marthi, B. Milch, S. Russell, an I. Shpitser Ientity uncertainty an citation matching. In NIPS. W. Soon, H. Ng, an D. Lim A machine learning approach to coreference resolution of noun phrases. Computational Linguistics (Special Issue on Computational Anaphora Resolution), 27: E. Voorhees Overview of the TREC-2002 question answering track. In Proceeings of TREC, pages