Robust Reading of Ambiguous Writing

Transcription

1 In submission; please o not istribute. Abstract A given entity, representing a person, a location or an organization, may be mentione in text in multiple, ambiguous ways. Unerstaning natural language requires ientifying whether ifferent mentions of a name, within an across ocuments, represent the same entity. We evelop an unsupervise learning approach that is shown to resolve accurately several aspects of the name ientity an tracing problem. At the heart of our approach is a generative moel of how ocuments are generate an how names are sprinkle into them; in particular, we moel appearance similarity between names representing the same entity, contextual correlation among entities, an co-occurrence probabilities of entities within a ocument. We show how to estimate the moel an o inference with it an how this resolves several aspects of the problem from the perspective of applications such as questions answering. 1 Introuction Robust Reaing of Ambiguous Writing Xin Li Paul Morie Dan Roth Department of Computer Science University of Illinois, Urbana, IL {xli1,morie,anr}@uiuc.eu Reaing an unerstaning text is a task that requires the ability to isambiguate at several levels, abstracting away etails an using backgroun knowlege in a variety of ways. One of the ifficulties that humans resolve instantaneously an unconsciously is that of reaing names. Most names of people, locations, organizations an others, have multiple writings that are being use freely within an across ocuments. The variability in writing a given concept, along with the fact that ifferent concepts may have very similar writings, poses a significant challenge to progress in natural language relate tasks. Consier, for example, an open omain question answering system (Voorhees, 2002) that attempts, given a question like When was Presient Kenney born? to search a large collection of articles in orer to pinpoint the concise answer: on May 29, The sentence, an even the ocument that contains the answer, may not contain the name Presient Kenney ; it may refer to this entity as Kenney, JFK or John Fitzgeral Kenney. Other ocuments may state that John F. Kenney, Jr. was born on November 25, 1960, but this fact refers to our target entity s son. Other mentions as Senator Kenney or Mrs. Kenney are even closer to the writing of the target entity, but clearly refer to ifferent entities. Even the statement John Kenney, born turns out to refer to a ifferent entity, as one can tell observing that this ocument iscusses Kenney s batting statistics. A similar problem exists for other entity types, such as locations, organization etc. A hoc solutions to this problem, as we show, fail to provie a reliable an accurate solution to this problem. This paper presents the first attempt to apply a unifie approach to all major aspects of this problem, presente here from the perspective of the question answering task: (1) Entity Ientity - o mentions A an B (typically, occurring in ifferent ocuments, or in a question an a ocument, etc.) refer to the same entity? This problem requires both ientifying when ifferent writings refer to the same entity, an when very similar or ientical writings refer to ifferent entities. (2) Name Expansion - given a writing of a name (say, in a question), fin other likely writings of the same name. (3) Prominence - given question What is Bush s foreign policy?, an given that any large collection of ocuments may contain several Bush s, there is a nee to ientify the most prominent, or relevant Bush, perhaps taking into about also some contextual information. At the heart of our approach is a global probabilistic view on how ocuments are generate an how names (of ifferent entity types) are sprinkle into them. In its most general form, our moel assumes: (1) a joint istribution over entities, so that a ocument that mentions Presient Kenney is more likely to mention Oswal or White House than Roger Clemens ; (2) an author moel, that makes sure that at least one mention of a name in a ocument is easily ientifiable, an then generates other mentions via (3) an appearance moel, governing how mentions are transforme from the representative mention. Our goal is to learn the moel from a large corpus an use it to support robust reaing - enabling on the fly ientification an tracing of entities. This work presents the first stuy of our propose moel an several relaxations of it. Given a collection of ocuments we learn the moels in an unsupervise way; that is, the system is not tol uring training whether two mentions represent the same entity. We only assume the 1

2 ability to recognize names, using a name entity recognizer run as a preprocessor. We efine several inferences that correspon to the solutions we seek, an evaluate the moels by performing these inferences against a large corpus we annotate (the corpus will be available on the web). Our experimental results suggest that the problem can be solve accurately, giving accuracies (F 1 ) close to 90%, epening on the specific task, as oppose to 80% given by state of the art a-hoc approaches. Previous work in the context of question answering has not aresse the problem stuie here. Several works in NLP an Databases, though, have aresse some aspects of the problem. From the natural language perspective, there has been a lot of work on the relate problem of coreference resolution (Soon et al., 2001; Ng an Carie, 2003; Kehler, 2002) - which aims at linking occurrences of noun phrases an pronouns within a given ocument base on their appearance an local context. In the context of atabases, several works have looke at the problem of recor linkage - recognizing uplicate recors in a atabase (Cohen an Richman, 2002; Hernanez an Stolfo, 1995; Bilenko an Mooney, 2003). Specifically, (Pasula et al., 2002) consiers the problem of ientity uncertainty in the context of citation matching an suggests a probabilistic moel for that. Perhaps the work that is most relate to ours in terms of the problem efinition an in the sense that it works with text ata an across ocuments, is (Mann an Yarowsky, 2003), which consiers the problem of istinguishing occurrences of ientical names in ifferent ocuments. Like ours, this problem is global, but they consier only one aspect of the ientity problem, an only for ientical names of people (e.g., o occurrences of Jim Clark in ifferent ocuments refer to the same person or not). The rest of this paper is organize as follows: We formalize the robust reaing problem in Sec. 2. Sec. 3 escribes a generative view of ocuments creation an three practical probabilistic moels esigne base on it, an iscusses inference in these moels. Sec. 4 illustrates how to learn these moels in an unsupervise setting, an Sec. 5 escribes the experimental stuy. Sec. 6 conclues. 2 Robust Reaing We consier reaing a large number of ocuments D = { 1, 2,..., m }, each of which may contain mentions (i.e. real occurrences) of T types of entities. In the current evaluation we consier T = {P erson, Location, Organization}. An entity refers to the real concept behin the mention an can be viewe as a unique ientifier to an object in the real worl. Examples might be the person John F. Kenney who became a presient, White House the resience of the US presients, etc. E enotes 2 the collection of all possible entities in the worl an E = {e i } l 1 is the set of entities mentione in ocument. M enotes the collection of all possible mentions an M = {m i } n 1 is the set of mentions in ocument. M i (1 i l ) is the set of mentions that refer to entity e i E. For example, for entity John F. Kenney, the corresponing set of mentions in a ocument may contain Kenney, J. F. Kenney an Presient Kenney. Among all mentions of an entity e i in ocument we istinguish the one occurring first, r i M i,astherepresentative of e i. In practice, the representative is usually the longest mention of an entity in the ocument as well, an other mentions are variations of it. Representatives can be viewe as a typical representation of an entity mentione in a specific time an place. For example, Presient J.F.Kenney an Congressman John Kenney may be representatives of John F. Kenney in ifferent ocuments. R enotes the collection of all possible representatives an R = {r i } l 1 M is the set of representatives in ocument. This way, each ocument is represente as the collection of its entities, representatives an mentions = {E,R,M }. Elements in the name space W = E R M each have an ientifying writing (enote as wrt(n) for n W ) 1 an an orere list of attributes, A = {a 1,...,a p }, which epens on the entity type. Attributes use in the current evaluation inclue both internal attributes, such as, for People, {title, firstname, milename, lastname, gener} as well as contextual attributes such as {time, location, proper-names}. Proper-names refer to a list of proper names that occur aroun the mention in the ocument. All attributes are of string value an can be empty when the values are missing or unknown 2. The funamental problem we aress in robust reaing is to ecie what entities are mentione in a given ocument (given the observe set M ) an what the most likely assignment of entity to each mention is. 3 A Moel of Document Generation We efine a probability istribution over ocuments = {E,R,M }, by escribing how ocuments are being generate. In its most general form the moel has the following three components: (1) A joint probability istribution P (E ) that governs how entities (of ifferent types) are istribute into a ocument an reflects their co-occurrence epenencies. (2) The number of entities in a ocument, size(e ), an the number of mentions of each entity in E, size(m i ), nee to be ecie. The current evaluation 1 The observe writing of a mention is its ientifying writing, i.e., Presient Kenney. For entities, it is a stanar representation of them, i.e. the full name of a person. 2 Contextual attributes are not part of the current evaluation, an will be evaluate in the next step of this work.

3 John Fitzgeral Kenney N i e John Fitzgeral Kenney e i Presient John F. Kenney r i {Presient Kenney, Kenney, JFK} E House of Representatives E House of Representatives R House of Representatives M {House of Representatives, The House} Figure 1: Generating a ocument makes the simplifying assumption that these numbers are etermine uniformly over a small plausible range. (3) The appearance probability of a name generate (transforme) from its representative is moelle as a prouct istribution over relational transformations of attribute values. This moel captures the similarity between appearances of two names. In the current evaluation the same appearance moel is use to calculate both the probability P (r e) that generates a representative r given an entity e an the probability P (m r) that generates a mention m given a representative r. Attribute transformations are relational, in the sense that the istribution is over transformation types an inepenent of the specific names. Given these, a ocument is assume to be generate as follows (see Fig. 1): A set of size(e ) entities E E is selecte to appear in a ocument, accoring to P (E ). For each entity e i E, a representative r i R is chosen accoring to P (r i e i ), generating R. Then mentions M i of an entity are generate from each representative r i R each mention m j M i is inepenently transforme from r i accoring to the appearance probability P (m j r i ), after size(m i ) is etermine. Assuming conitional inepenency between M an E given R, the probability istribution over ocuments is therefore P () =P (E,R,M )=P(E )P (R E )P (M R ), an the probability of the ocument collection D is: P (D) = D P (). 3.1 Relaxations of the Moel In orer to simplify moel estimation an to evaluate some assumptions, several relaxations are mae to form three simpler probabilistic moels. Moel I: (the simplest moel) The key relaxation here is in losing the notion of an author rather than first choosing a representative for each ocument, mentions are generate inepenently an irectly given an entity. That is, an entity e i is selecte from E accoring to the prior probability P (e i ); then its actual mention m i is selecte accoring to P (m i e i ). Also, an entity is selecte into a ocument inepenently of other entities. In this way, the probability of the whole ocument set can be written in a simpler way: n P (D) =P ({(e i,m i)} n i=1) = P (e i)p (m i e i), i=1 an the inference problem for the most likely entity given m is: e = argmax e E P (e m, θ) (3) = argmax e E P (e)p (m e) (4) Moel II: (more expressive) The major relaxation mae here is in assuming a simple moel of choosing entities to appear in ocuments. Thus, in orer to generate a ocument, after we ecie size(e ) an {size(m 1,size(M 2 ),...} accoring to uniform istributions, each entity e i is selecte into inepenently of others accoring to P (e i ). Next, the representative r i for each entity e i is selecte accoring to P (r i e i ) an for each representative the actual mentions are selecte inepenently accoring to P (m j r j ). Here, we have iniviual ocuments along with representatives, an the istribution over ocuments is: P () = P (E,R,M )=P (E )P (R E )P (M R ) E =l = [P (size(e )) P (e i )] i=1 E =l [P (size(m 1 ),size(m 2 ),...) P (r i e i )] i=1 P (m j r j ) (r j,m j ) E =l [P (e i )P (r i e i )] P (m j r j ). i=1 (r j,m j ) Given a mention m in a ocument (M is the set of observe mentions in ), the key inference problem is to etermine the most likely entity e m that correspons to it. This is one by computing: E = argmax E EP (E,R M,θ) (1) = argmax E EP (E,R,M θ), (2) where θ is the learne moel s parameters. This gives the assignment of the most likely entity e m for m. 3 after we ignore the size components. The inference problem here is the same as in Equ. (2). Moel III: This moel performs the least relaxation. After eciing size(e ) accoring to a uniform istribution, instea of assuming inepenency among entities which oes not hol in reality (For example, Gore an George. W. Bush occur together frequently, but Gore an Steve. Bush o not), we select entities using a graph base algorithm: entities in E are viewe

4 as noes in a weighte irecte graph with eges (i, j) labelle P (e j e i ) representing the probability that entity e j is chosen into a ocument that contains entity e i.we istribute entities to E via a ranom walk on this graph starting from e 1 with a prior probability P (e i ). Representatives an mentions are generate in the same way as in Moel II. Therefore, a more general moel for the istribution over ocuments is: P () E =l P (e 1 )P (r 1 e 1 ) [P (e i e i 1 )P (r i e i )] i=2 P (m j r j ). (r j,m j ) e1= George Bush e2= George W. Bush e3= Steve Bush 1 m1,r1=presient Bush m2=bush m3=j. Quayle Entities E 2 m4,r2=steve Bush m5=bush Figure 2: An conceptual example. The arrows represent The inference problem is the same as in Equ. (2). correct assignment of entities to mentions. r 1,r 2 are representatives. 3.2 Inference The funamental problem in robust reaing can be solve of entities given those representatives in their appearing as inference with the moels: given a mention m, seek orer using the Viterbi algorithm. The total time complexity is O( M 2 + E 2 R ). the most probable entity e E for m accoring to Equ. (4) for Moel I or Equ. (2) for Moel II an III. The 3.3 Discussion inference algorithm for Moel I (with time complexity Besies ifferent assumptions of the moels, there are O( E )) is simple an irect: just compute P (e, m) for some funamental ifferences in inference with the moels as well. In Moel I, the entity of a mention is e- each caniate entity e E an then choose the one with the highest value. Due to exponential number of termine completely inepenently of other mentions, possible assignments of E,R to M in Moel II an III, while in Moel II the way of figuring out the entity relies on local similarity among mentions in the same oc- precise inference is infeasible. Approximate algorithms are therefore esigne: ument. In Moel III, it is not only relate to other mentions but to a global epenency over entities. The fol- In Moel II, we aopt a two-step algorithm: First, we seek the representatives R for the mentions M in ocument by sequentially clustering the mentions accoring lowing conceptual example illustrates those ifferences as in Fig. 2. to the appearance moel. The first mention in each group is treate as the representative. Specifically, when consiering a mention m M, P (m r) is compute for 1, 2 an 5 mentions in them, an Example 3.1 Given E = {George Bush, George W. Bush, Steve Bush}, ocuments suppose the prior probability of entity George W. Bush is each representative r that have alreay been create an higher than those of the other two entities, the probable assignment of entities to mentions in the three moels coul be a fixe threshol is then use to ecie whether to create a new group for m or to a it to one of the existing group as follows: with the highest P (m r) value. In the secon step, each For Moel I, mentions(e 1) = φ, mentions(e 2) = {m representative r i R is assigne to its most likely entity accoring to e = argmax e E P (e) P (r e) 3 cause by the fact that a mention tens to be assigne to the 1,m 2,m 5} an mentions(e 3) = {m 4}. The result is. This entity with higher prior probability when the appearance similarity algorithm has a total time complexity of O( M 2 + E R ). is not istinctive. For Moel II, mentions(e 1) = φ, mentions(e 2) = Moel III has a similar two-step algorithm as Moel II. {m 1,m 2} an mentions(e 3) = {m 4,m 5}. Local epenency (appearance similarity) among mentions insie each ocument enforces constraints that they shoul refer to the same The only ifference is that we nee to consier the global epenency between entities. Thus in the secon step, entity, like Steve Bush an Bush in 2. instea of seeking an entity e for each representative r For Moel III, mentions(e 1)={m 1,m 2}, mentions(e 2) separately, we etermine a set of entities E for R in a = φ, mentions(e 3) = {m 4,m 5}. With the help of global Hien Markov Moel with entities in E as hien states epenency among entities, for example, George Bush an J. Quayle, an entity can be istinguishe from another entity an R as observations. The prior probabilities., the transitive probabilities an the observation probabilities for with a similar writing. this HMM are given by P (e), P (e j e i ) an P (r e) respectively. 3.4 Other Tasks In this step we seek the most likely sequence E is known after learning the moel in a close ocument collection that belongs to. The three basic problems relate to Robust Reaing can be solve base on the solutions to the key inference problem above. 4

5 Entity Ientity: Given two mentions m 1 1,m 2 2, etermine whether they correspon to the same entity (m 1 m 2 ) by: 1. In the initial (I-) step, an initial E 0 an R 0 is assigne to each ocument using an initialization algorithm. After this step, we can assume that we have labelle ocuments D 0 = {(E 0,R 0,M )}. m 1 m 2 iff argmax e EP (e, m 1)=argmax e EP (e, m 2) for Moel I an m 1 m 2 iff argmax e EP (E 1,R 1,M 1 )= argmax e EP (E 2,R 2,M 2 ). for Moel II an III. Name Expansion: Given a mention m q in a query q, ecie whether mention m in the ocument collection D is a legal expansion of m q : m q m iff e m q = argmax e EP (E q,r q,m q) & m mentions(e ). We assume here that we alreay know the possible mentions of e after learning the moels in D. Prominence: Given a name n W, the most prominent entity for n is given by: e = argmax e E P (e)p (n e). P (e) is given by the prior istribution P E an P (n e) is given by the appearance moel. 4 Learning the Moels Confine by the labor of annotating ata, we learn the probabilistic moels in an unsupervise way given a collection of ocuments; that is, the system is not tol uring training whether two mentions represent the same entity. A greey search algorithm moifie after the stanar EM algorithm (We call it Truncate EM algorithm) is aopte here to avoi complex computation. Given a set of ocuments D to be stuie an the observe mentions M in each ocument, this algorithm iteratively upates the moel parameter θ (several unerlying probabilistic istributions escribe before) an the structure (that is, E an R ) of each ocument. Different from the stanar EM algorithm, in the E-step, it seeks the most likely E an R for each ocument rather than the expecte assignment. 4.1 Truncate EM Algorithm The basic framework of the Truncate EM algorithm to learn Moel II an III is as follows: 5 2. In the M-step, we seek the moel parameter θ t+1 that maximizes P (D t θ). Given the labels supplie by the moel in the previous I- or E-step, this amounts to the maximum likelihoo estimation as escribe in Sec In the E-step, we seek (E t+1,r t+1 ) for each ocument that maximizes P (D t+1 θ t+1 ) where D t+1 = {(E t+1,r t+1,m )}. It s the same inference problem in Sec Stoping Criterion: If no increase is achieve over P (D t θ t ), the algorithm exits. Otherwise the algorithm will iterate over the M-step an E-step. The algorithm for Moel I is similar to the above algorithm but much simpler in the sense that it oes not have the notions of ocuments an representatives. So in the E-step we only nee to seek the most possible entity e for each mention m D an this simplifies the parameter estimation in the M-step accoringly. It usually takes 3 10 iterations before the algorithm stops for all the moels in our experiments. 4.2 Initialization The purpose of the initial step is to acquire an initial guess of ocument structures an to seek the set of entities E in a close collection of ocuments D. The hope is to fin all entities without loss even if repeate entities might be create. For all the moels, we use the same algorithm: First, a local clustering is performe to group all mentions insie each ocument. A set of simple heuristics of matching attributes is applie to calculating the similarity between mentions an pairs of mentions with similarity above a threshol are clustere together. The first mention in each group is chosen as the representative (only in Moel II an III) an an entity having the same writing with the representative is create for each cluster 4. For all the moels, the set of entities create in ifferent ocuments become the global entity set E in the following M- an E-steps. 4.3 Estimating the Moel Parameters In the learning process, assuming we have obtaine labelle ocuments D = {(e, r, m)} n 1 from previous I- or E-step, several probability istributions unerlying the relaxe moels are estimate accoring to maximum likelihoo estimation in each M-step. The moel parameters inclue a prior istribution over entities P E, a tran- 4 Note that the performance of the initialization algorithm is 97.3% precision an 10.1% recall, measures efine in our later experimental stuy in Sec. 5.

6 sitive probability istribution over pairs of entities P E E (only in Moel III) an the appearance probability P W W of a name in the name space W being transforme from another name. The prior istribution P E is moelle as a multinomial istribution. Given a set of labelle entitymention pairs {(e i,m i )} n 1, P (e) = freq(e) n where freq(e) enotes the number of pairs containing entity e. Given all the entities appearing in D, The transitive probability between entities P (e e) is estimate by P (e 2 e 1) P (wrt(e 2) wrt(e 1)) = oc# (wrt(e 2 ),wrt(e 1 )). oc # (wrt(e 1 )) Here, the conitional probability between two real entities P (e 2 e 1 ) is backe off to the conitional probability between the ientifying writings of the two entities P (wrt(e 2 ) wrt(e 1 )) in the ocument set D to avoi sparsity problem. Given D = { 1, 2,..., m }. An oc # (w 1,w 2,...) enotes the number of ocuments having the co-occurrence of writings w 1,w 2,... Appearance Probability, the probability of one name being transforme from another, enote as P (n 2 n 1 ) (n 1,n 2 W ), is moelle as a prouct of the transformation probabilities over attribute values. The transformation probability for each attribute in A is further moelle as a multi-nomial istribution over a set of preetermine typical transformation types that epen on the entity types: TT = {copy,missing,typical,non typical} 5. Suppose n 1 =(a 1 = v 1,a 2 = v 2,..., a p = v p ) an n 2 =(a 1 = v 1,a 2 = v 2,..., a p = v p) are two names belonging to the same entity type, the transformation probabilities P M R, P R E an P M E, are all moelle as a prouct istribution (naive Bayes) over attributes: P (n 2 n 1)=Π p k=1 P (v k v k ). We manually collecte typical an non-typical transformations for attributes such as titles, first names, last names, organizations an locations from multiple sources such as U.S. government census an online ictionaries. For other attributes like gener, only copy transformation is allowe. Assuming multi-nomial istribution for each attribute, the maximum likelihoo estimation of the transformation probability P (t, k) (t TT,a k A) from labelle representative-mention pairs {(r, m)} n 1 is: 5 copy enotes v k is exactly the same as v k ; missing enotes missing value for v k; typical enotes v k is a typical variation of v k, for example, Prof. for Professor, Any for Anrew ; non-typical enotes a non-typical transformation. 6 P (t, k) = freq(r, m) :vr k t v m k n vk r t vk m enotes the transformation from attribute a k of r to that of m is of type t. Simple smoothing is performe here for unseen transformations. 5 Experimental Stuy Our experimental stuy focuses on (1) evaluating our three moels on the name ientity task using three entity types (People, Locations, Organization); (2) comparing our inuce similarity measure between names with other similarity measures; (3) evaluating the contribution of the global nature of our moel, an (4) evaluating our moels on name expansion an prominence ranking. 5.1 Methoology We collecte 300 ocuments from ranomly sample New York Times articles in the TREC corpus (Voorhees, 2002). The ocuments were annotate by a name entity tagger for People, Locations an Organizations. The annotation was then correcte an each name mention was labelle with its corresponing entity by two annotators. In total, about 8, 000 mentions of name entities which correspon to about 2, 000 entities were labelle. The training process gets to see only the 300 ocuments an extracts attribute values for each mention. No supervision is supplie. These recors are use to learn the probabilistic moels. In testing, 130, 000 pairs of mentions that correspon to the same entity are generate, an are use to evaluate the moels performance. Since the probabilistic moels are learne in an unsupervise setting, testing can be viewe simply as the evaluation of the learne moel, an is thus one on the same ata. The same setting was use for all moels an all comparison performe (see below). To evaluate the performance, we pair two mentions iff the learne moel etermine that they correspon to the same entity. The list of pairs is then compare with the annotate list of pairs. We measure Precision (P ) Percentage of correctly preicte pairs, Recall (R) Percentage of correct pairs that were preicte, an F 1 = 2PR P +R. Comparisons: Our global moel inuces a similarity measure between names the appearance moel. In orer to unerstan whether the behavior of our moel is ominate by the quality of the inuce pairwise similarity or by the global aspects of the moel we (1) replace this measure by two other local similarity measures an (2) stuy the performance on entity ientity at three levels local ecision, straightforwar clustering over local similarity, an our global moel. The first similarity measure we use is a simple baseline algorithm accoring to which two names are similar iff (5)

7 All(P/L/O) Ientity SoftTFIDF Appearance Pairwise 70.7 (64.7/64.1/83.7) 82.1 (79.9/77.3/89.5) 81.5 (83.6/70.9/90.7) Clustering 70.7 (64.7/64.1/83.7) 79.8 (70.6/76.7/91.0) 79.6 (70.9/76.1/91.0) Moel II 70.7 (64.7/64.1/83.7) 82.5 (79.8/77.4/90.2) 89.0 (92.7/81.9/92.9) Table 1: Comparison of ifferent ecision levels an similarity measures. Three similarity measures are evaluate (rows) across three ecision levels (columns). Performance is evaluate by the F 1 values over the whole test set. The first number averages all entity types; numbers in parentheses represent People, Location an Organization respectively. they have ientical writings. The secon is a state-ofart similarity measure for entity names (SoftTFIDF with Jaro-Winkler istance an θ =0.9); it was ranke the best measure in a recent stuy (Cohen et al., 2003). Local ecision (Pairwise) is one by pairing two mentions iff the similarity between them is above a fixe threshol. For Clustering, a graph-base clustering algorithm is use, where two mentions are paire iff they belong to the same connecte component. Finally, we use the baseline an the SoftTFIDF in the context of Moel II, where the appearance moel is replace by the similarity measure Results The bottom line result is given in Tab. 1. All local similarity measures are compare in the context of the three levels of processing local ecision, clustering an our probabilistic moel II. The behavior across rows inicates that our unsupervise learning base appearance moel is about the same as the state-of-the-art SoftTFIDF similarity. The behavior across columns, though, shows the contribution of our global moel, an that the local appearance moel behaves better with it than a fixe similarity measure oes. A seconary observation is that our appearance moel for Location is not as goo as the one for People an Organization, probably ue to the attribute transformation types chosen. Tab. 2 presents a more etaile evaluation of the ifferent approaches on the entity ientity task. All the three probabilistic moels outperform the iscriminatory approaches in this experiment, an inication of the effectiveness of the generative moel. We note that although Moel III is more expressive an reasonable than moel II, it oes not always perform better. Inee, the global epenency among entities in Moel III achieves two-fole outcomes: it achieves better precision but, may egrae the recall. The following 6 Note that both the appearance moel s(n 1,n 2) = P (n 1 n 2) an the SoftTFIDF similarity measure are not symmetric. Also, we foun that the SoftTFIDF similarity measure behaves very baly in the context of the probabilistic moel, an improve it by converting it to P (n 1 n 2)= ec s(n 1,n 2 ) 1.c e c 1 was set to 10 in the experiments. 7 Entity Mo InDoc InterDoc All Type F 1 (%) F 1 (%) R(%) P(%) F 1 (%) All B D I II III P B D I II III L B D I II III O B D I II III Table 2: Performance of ifferent approaches over all test examples. B, D, I, II an III enote the baseline moel, the SoftTFIDF similarity moel with clustering, an the three probabilistic moels. All,P,L,O enote all entities, People, Locations an Organizations respectively. We istinguish between pairs of mentions that are insie the same ocument (InDoc, 15% of the pairs) or not (InterDoc). example, taken from the corpus, illustrates the avantage of this moel. Example 5.1 Sherman Williams is mentione along with the baseball team Dallas Cowboys in eight out of 300 ocuments, while Jeff Williams is mentione along with LA Dogers in two ocuments. In all the moels except Moel III, Jeff Williams is juge to correspon to the same entity as Sherman Williams since they are quite similar an the prior probability of the latter is higher than the former. Only in Moel III, ue to the epenency between Jeff Williams an Dogers, the system ientifies it as corresponing to a ifferent entity than Sherman Williams. While this exhibits the better precision achieve by Moel III, the recall may go own. The reason is that the global epenency among entities in Moel III enforces restrictions over possible grouping of similar mentions; in aition, with a limite ocument set, estimating this global epenency cannot be one accurately, especially in the setting that entities themselves nee to be foun when learning the moel. We expect that Moel III will ominate Moel II when we have enough ata to estimate a more accurate global epenencies. Har Cases: To analyze the experimental results further, we evaluate separately two types of harer cases of the entity ientity task: (1) mentions with ifferent writings that refer to the same entity; an (2) mentions with similar writings that refer to ifferent entities. Moel II an III outperform other moels in those two cases as well. Tab. 3 presents F 1 performance of ifferent approaches in the first case. The best F 1 value is only 73.1%, inicating that appearance similarity an global epenency are not sufficient to solve this problem when the writings are very ifferent. Tab. 4 shows the performance of ifferent approaches for isambiguating similar writings that

8 Moel B D I II III Peop Loc Org All Table 3: Ientifying ifferent writings of the same entity (F 1). We filter out ientical writings an report only on cases of ifferent writings of the same entity. The test set contains 46, 376 matching pairs (but in ifferent writings) in the whole ata set. Moel B D I II III Peop Loc Org All Table 4: Ientifying similar writings of ifferent entities. (F 1) The test set contains 39, 837 pairs of mentions that associate with ifferent entities in the 300 ocuments an have at least one token in common. correspon to ifferent entities. Both these cases exhibit the ifficulty of the problem, an that our approach provies a significant improvement over the state of the art similarity measure column D vs. column II in Tab. 4. It also shows that it is necessary to use contextual attributes of the names, which are not yet inclue in this evaluation. 5.3 Other Tasks In the following experiments, we evaluate our generative moel on other tasks relate to robust reaing. We present results only for Moel II. Name Expansion: Given a mention m (for example, in a IR query q), we fin the most likely entity e E for m using our inference algorithm. All unique mentions of the entity in the ocuments are output as the expansions of m. The accuracy of Name Expansion for a given mention is efine as the number of correct expansions over the total number of names output. The average accuracy of Name Expansion of Moel II is shown in Tab. 5. Here is an example of a query: Query: Who is Gore? Expansions: Vice Presient Al Gore, Al Gore, Gore. Prominence Information: We refer to Example 3.1 an use it to exemplify quantitatively how our system supports prominence ranking. The following examples show the ranking of entities with regar to the value of P (e) P (m e) (shown in the brackets) using Moel II, given a query name m. Input: George Bush 1. George Bush(2.49E-4) 2. George W. Bush(6.64E-7) Input: Bush 1. George W. Bush(5.13E-7) 2. George Bush(1.42E-7) 3. Steve Bush(5.69E-10) 6 Conclusion an Future Work This paper presents an unsupervise learning approach to several aspects of the robust reaing problem crossocument resolution of ambiguous writings of names. We evelope a moel that escribes the natural generation process of a ocument an the process of how names are sprinkle into them, taking into account epenencies between entities across types an an author moel. Several relaxations of this moel were evelope an stuie experimentally, an compare to a state-ofthe-art moel that oes not take a global view. The experiments exhibit goo results an show the avantage of several aspects of our moel. This work is a preliminary exploration of the robust reaing problem. There are several critical issues that our moel can support, but were not inclue in this preliminary evaluation. Some of the issues that will be inclue in future steps are: (1) integration with more contextual information (like time an place) relate to the target entities, both to support a better moel an to allow temporal tracing of entities; (2) stuying an incremental approach learning the moel; that is, when a new ocument is observe, coming, how can we upate our moel parameters an the corresponing knowlege base? (3) integration of this work with other aspect of coreference resolution (e.g., other terms like pronouns that refer to an entity) an name entity recognition (which we now take as a given); an (4) scalability issues in applying the system to very large corpora. References M. Bilenko an R. Mooney Aaptive uplicate etection using learnable string similarity measures. In KDD. W. Cohen an J. Richman Learning to match an cluster large high-imensional ata sets for ata integration. In KDD. W. Cohen, P. Ravikumar, an S. Fienberg A comparison of string metrics for name-matching tasks. In IIWeb Workshop M. Hernanez an S. Stolfo The merge/purge problem for large atabases. In SIGMOD. A. Kehler Coherence, Reference, an the Theory of Grammar. CSLI Publications. G. Mann an D. Yarowsky Unsupervise personal name isambiguation. In CoNLL. V. Ng an C. Carie Improving machine learning approaches to coreference resolution. In ACL. H. Pasula, B. Marthi, B. Milch, S. Russell, an I. Shpitser Ientity uncertainty an citation matching. In NIPS. Entity Type People Location Organization Accuracy(%) Table 5: Accuracy of Name Expansion. Accuracy is average over 30 ranomly chosen queries for each entity type. 8 W. Soon, H. Ng, an D. Lim A machine learning approach to coreference resolution of noun phrases. Computational Linguistics (Special Issue on Computational Anaphora Resolution), 27:

9 E. Voorhees Overview of the TREC-2002 question answering track. In Proceeings of TREC, pages