(12) United States Patent (10) Patent N0.: US 7,555,428 B1 Franz et al. (45) Date of Patent: Jun. 30, 2009

Transcription

1 US B1 (12) United States Patent (10) Patent N0.: US 7,555,428 B1 Franz et a. (45) Date of Patent: Jun. 30, 2009 (54) SYSTEM AND METHOD FOR IDENTIFYING OTHER PUBLICATIONS COMPOUNDS THROUGH ITERATIVE ANALYSIS Su. K., Wu, M,. and Chang, J A Corpus-based approach to automatic compound extraction. In Proceedings of the 32nd Annua (75) Inventors; Aexander Franz pao Ato C A (Us); Meeting on Association For Computationa Linguistics (Las Cruces, Brian Much Berkeey, CA (Us) New Mexico, Jun , 1994). Annua Meeting ofthe ACL. Asso ciation for Computationa Linguistics, MorristoWn, NJ, * (73) Assignee: Googe Inc., Mountain View, CA (U S) Venkataraman, A A statistica mode for Word discovery in transcribed speech. Comput. Linguist. 27, 3 (Sep. 2001), * ( * ) Notice: Subject to any discaimer, the term ofthis G30, J, Goodman, J, Li, M, and Lee, K, 2002, Toward a uni?ed patent is exiended 01 adjusted under 35 approach to statistica anguage modeing for chinese. ACM Trans U.S.C. 154(b) by 574 days. actions on Asian Language Information Processing (TALIP), 1 (Mar. 2002), DOI: (21) App. N0.: 10/647, (22) Fied: Aug. 21, 2003 (Continued) (51) Int- C Primary ExamineriParas' Shah ' ' G06F 1 7/21 (200601) (74) Attorney, Agent, or FzrmiFsh & R1chards0n P.C. G06F 17/27 G06F 17/28 ( ) ( ) (57) ABSTRACT (52) US. C /10; 704/7; 704/9 (58) Fied of Cassi?cation Search /7, A System and method for identifying Compounds through 7041/10: 9 iterative anaysis of measure of association is discosed. A See appication?e for Compete Search history imit on a number of tokens per compound is speci?ed. Com (56) References Cited pounds Within a text corpus are iterativey evauated. A num ber of occurrences of one or more n-grams Within the text U.S. PATENT DOCUMENTS corpus is determined. Each n-gram incudes up to a maximum 5,842,217 A * 11/1998 Light /101 number Oftokens, which are each Provided in a Vocabuary 5,867,812 A * 2/1999 Sassano 704/10 for the text corpus. At east one n-gram incuding a number of 6,173,298 B1 * 1/2001 Smadja /209 tokens equa to the imit based on the number OfOCCurrenCe/S 6,285,999 B1 9/2001 Page is identi?ed. A measure of association between the tokens in 6,349,282 B1 * 2/2002 Van Aeten et a /257 the identi?ed n-gram is determined. Each identi?ed n-gram 6,754,617 B1 * 6/2004 Ejerhed ~~ 704/9 With a su?icient measure of association is added to the 2007/ A* 3/2007 Kaku et a /10 JP FOREIGN PATENT DOCUMENTS * 6/1996 vocabuary as a compound token and the imit is adjusted. 15 Caims, 6 Drawing Sheets 5Q Token Compound Engne Vocabuary \ _1 > Corpus preparation I * Text Corpus N-gram,1_ 51/ V 36 Counts List 54 \de \ N-gram List k 58 M, N-gram Counter : 61 J Likeuhood E113. \ Compound Fnder :/ 53 Compounds R t' L' t 8'08 '8 Likeihood Eva \ /\55 L' t 'S 60 " q Iterator \/\ 56 W 37

2 US 7,555,428 B1 Page 2 OTHER PUBLICATIONS Jurafsky, D., et a. (2000). Backoff. Speech and Language Process ing.: An Introduction to Natura Language Processing, Computa tiona Linguistics, and Speech Recognition. Pearson Ha Jerse, pp * Smadja, F Retrieving coocations from text: Xtract. Comput. Linguist. 9, 1(Mar. 1993), * FrantZi, K. T. and Ananiadou, S Extracting nested cooca tions. In Proceedings of the 16th Conference on Computationa Linguisticsivo. (Copenhagen, Denmark, Aug. 5-9, 1996). Inter nationa Conference On Computationa Linguistics. Association for Computationa Linguistics, Morristown, NJ DOI: doi.org/0.35/9926.* Seretan V., Neriman, L. and Wehri, E Extraction of Muti Word Coocations Using Syntactic Bigram Composition. In Pro ceedings of the Internationa Conference on Recent Advances in NLP (RANLP-ZOOB), Borovets, Bugaria, pp * C.D. Manning and H. SchutZe, Foundations Of Statistica Natura Languages Processing, Ch. 5, MIT Press (1999). T. Dunning, Accurate Methods For The Statistics Of Surprise And Coincidence, Comp. Ling, vo. 19, No. I, pp (1993). * cited by examiner

3

4 US. Patent Jun. 30, 2009 Sheet 2 of6 US 7,555,428 B1 Figure 2. Q 381 Storage Compound Sewer 32 Av Memory Text Corpus 34 /_\ Compound '/ Engine AH w "o Processor Compounds List ; \/ J dz ) Figure 3. 3g %, Token Compound Engine Vocabuary i I \ i_ y Corpus preparation < Text Corpus N-gram 51 "*V 36 Counts List (I: S4/"\JII k N-gram List,L 58 \- UJ N-gram Counter 61 JIK \/\ Likeihood compound Finder 7/ 53 Compounds R t' L t a '08 '8 Likeihood Eva \i/\ 55 L'St 6O Iterator \/\ 56 w 37

5 US. Patent Jun. 30, 2009 Figure 4. Sheet 3 0f 6 US 7,555,428 B1 Prepare corpus w 71 Find compounds w 72 Figure 5. a Assembe text into corpus W81 Parse corpus into tokens W82 Construct initia token $83

6 US. Patent Jun. 30, 2009 Sheet 4 0f 6 US 7,555,428 B1 Figure 6. 1% Identify ength of compounds (LEN) Identify upper imit on compounds desired (LIM) Count n-grams Seect one or more n grams of ength LEN from n-gram ist or eac 0 one or more seected n-grams, do Determine ikeihood of seected n-gram 105 Next /* n-gram */ 107 Seect up to LIM most ikey n-grams of ength LEN Add each of most ikey n-grams as compound tokens to token vocabuary a

7 US. Patent Jun. 30, 2009 Sheet 5 0f 6 US 7,555,428 B1 Figure 6 Reconstruct token VA 1 10 vocabuary Adjust compounds ength LEN S 112 Y es 111 N0

8 US. Patent Jun. 30, 2009 Sheet 6 of6 US 7,555,428 B1 Figure Identify maximum n-gram ength (MAX) 1, J 121 Seect unique n-grams $122 For i = 1 to MAX, do 123 Count number of occurrences of unique n-grams of ength fusing token vocabuary VA 124 \ Next /*i*/ /Z/ 125

9 1 SYSTEM AND METHOD FOR IDENTIFYING COMPOUNDS THROUGH ITERATIVE ANALYSIS FIELD OF THE INVENTION The present invention reates in genera to text anaysis and, in particuar, to a system and method for identifying compounds through iterative anaysis. BACKGROUND OF THE INVENTION Athough the origins of the Internet trace back to the ate 1960s, the more recenty-deveoped WordWide Web ( Web ), together With the ong-estabished Usenet, have revoutionized accessibiity to untod voumes of information in stored eectronic form to a WordWide audience, incuding Written, spoken (audio) and visua (imagery and video) infor mation, both in archived and rea-time formats. The Web provides information via interconnected Web pages that can be navigated through embedded hyperinks. The Usenet pro vides information in a non-interactive buetin board format consisting of static news messages posted and retrievabe by readers. In short, the Web and Usenet provide desktop access to a virtuay unimited ibrary of information in amost every anguage WordWide. Information exchange on the Web and Usenet both operate under a cient-server mode. For the Web, individua cients typicay execute Web browsers to retrieve and dispay Web pages in a graphica user environment. For the Usenet, indi vidua cients generay execute news readers to retrieve, post and dispay news messages, usuay in a textua user environ ment. Both Web browsers and news readers interface to cen traized content servers, Which function as data dissemina tion, storage and retrieva repositories. NeWs messages avaiabe via the Usenet are cataoged into speci?c news groups and?nding reevant content invoves a straightforward searching of news groups and message ists. Web content, however, is not organized in any structured manner and search engines have evoved to enabe users to?nd and retrieve reevant Web content, as We as news mes sages and other types of content. As the amount and variety of Web content have increased, the sophistication and accuracy of search engines have ikewise improved. Existing methods used by search engines are based on matching search query terms to terms indexed from Web pages. More advanced methods determine the importance of retrieved Web content using, for exampe, a hyperink structure-based anaysis, such as described in S. Brin and L. Page, The Anatomy of a Large-Scae Hypertextua Search Engine, (1998) andin US. Pat. No. 6,285,999, issued Sep. 4, 2001 to Page, the disco sures of Which are incorporated by reference. Compounds frequenty occur in Web content, news mes sages, and other types of content. A compound, sometimes aso referred to as a coocation, is de?ned as any sequence of Words that co-occur more often than by mere chance. Com pounds occur in text and speech as a natura anguage con struct and can incude proper nouns, such as San Francisco, compound nouns, such as hot dog, and other semantic and syntactic anguage constructs, Which resut in the co-occur rence of two or more Words. Compounds occur With reguar ity in a range of appications, incuding speech recognition, text cassi?cation, and search resut scoring. Recognizing compounds is dif?cut, especiay When occurring in speech or ive text. Moreover, most anguages ack reguar syntactic or semantic cues to enabe easy iden ti?cation of compounds. In German, for instance, the?rst US 7,555,428 B etter of each noun is capitaized, Which compicates the identi?cation of proper nouns. Simiary, the types of poten tia compounds can depend on the subject matter. For instance, a scienti?c paper coud incude compounds Whoy unique from those found in a sports coumn. Conventiona approaches to?nding compounds in a text corpora typicay rey on n-gram anaysis, such as described in C. D. Manning and H. SchiitZe, Foundations of Statistica Natura Languages Processing, Ch. 5, MIT Press (1999), the discosure Which is incorporated by reference. An n-gram is a muti-word occurrence. N-gram-based approaches therefore count the frequencies of individua Words or tokens and the frequencies of Word sequences of varying engths. N-gram based approaches suffer from three principa di?icuties. First, n-gram-based approaches are storage inef?cient. As the number of Words occurring in each n-gram increases, the number of unique n-grams in a corpus approaches the number of Words in a corpus. Storing the counts for ong sequences of n-grams can require a prohibitivey arge amount of memory. Second, With compounds of varying engths, the ikeihood of spurious shorter compounds being incuded as substrings increases. Spurious substrings of onger compounds can occur, skewing compound identi?cation resuts. For exampe, New York City is a three-word compound, Where the Words New, York, and City are highy correated. As a side effect, York City is aso highy correated, but generay does not represent a meaningfu compound. York and City are ony correated in the context of the arger com pound, New York City. Simiary, With compounds consisting of three or more Words, the ikeihood that a onger compound Wi contain two-word or three-word compounds as substrings increases. Spurious ong compounds that contain shorter, but signi? cant, compounds as substrings can occur. For exampe, San Francisco as a two-word compound, but San Francisco has is not a three-word compound. Nevertheess, n-gram based approaches, Which assume a Words are independent, Woud erroneousy identify San Francisco has as a three Word compound. Therefore, there is a need for an approach to ef?cienty identifying compounds in a text corpus based on a measure of association, such as a ikeihood of co-occurrence between the Words Which constitute each compound. There is a further need for an approach to forming a ist of compounds though an anaysis of a text corpus With minima overapping substrings, minima overapping compounds, and e?icient memory utiization. SUMMARY OF THE INVENTION The present invention provides a system and method for discovering and identifying compounds Within a text corpus through iterative anaysis of measures of association between constituent tokens. A text corpus is evauated into a set of unique n-grams and counts of the frequencies of occurrence of each unique n-gram are taied. Those n-grams having a speci?ed ength are seected and the ikeihood of each seected n-gram being a compound, that is, the ikeihood of coocation, is determined. In the described embodiment, the ikeihood of coocation is evauated using the ikeihood ratio method, athough other methodoogies and approaches coud be used, as Woud be recognized by one skied in the art. Those n-grams most ikey constituting compounds are added to a token vocabuary, preferaby up to an upper imit number of n-grams. The token vocabuary is reconstructed to add the new compounds and remove constituent tokens, Which occur in the new compounds. The speci?ed ength is

10 US 7,555,428 B1 3 adjusted and evauation continues using the revised token vocabuary. In the preceding exampe, the n-gram San Fran cisco has Woud receive a reativey ow score because a ikeihood under the assumption that the n-gram San Fran cisco has is a compound Woud be ony sighty higher than 5 a ikeihood under the assumption that San Francisco is a compound but the entire n-gram is not a compound. An embodiment provides a system and method for?nding compounds in a text corpus. A vocabuary incuding tokens extracted from a text corpus is buit. Compounds having a puraity of engths Within the text corpus are iterativey iden ti?ed. Each compound incudes a puraity of tokens. A fre quency of occurrence for one or more n-grams in the text corpus is evauated. Each n-gram incudes tokens seected from the vocabuary. A ikeihood of coocation for one or more of the n-grams having a same ength is determined. The n-grams having a highest ikeihood are added as compounds to the vocabuary. The vocabuary is rebuit based on the added compounds. 20 A further embodiment provides a system and method for identifying compounds through iterative anaysis of measure of association. A imit on a number of tokens per compound is speci?ed. Compounds Within a text corpus are iterativey 25 evauated. A number of occurrences of one or more n-grams Within the text corpus is determined. Each n-gram incudes up to a maximum number of tokens, Which are each provided in a vocabuary for the text corpus. At east one n-gram incud ing a number of tokens equa to the imit based on the number 30 of occurrences is identi?ed. A measure of association between the tokens in the identi?ed n-gram is determined. Each identi?ed n-gram With a suf?cient measure of associa tion is added to the vocabuary as a compound token. The vocabuary is rebuit based on the added compound tokens 35 and adjusting the imit. The imit is adjusted. Sti other embodiments of the present invention Wi become readiy apparent to those skied in the art from the foowing detaied description, Wherein are described 40 embodiments of the invention by Way of iustrating the best mode contempated for carrying out the invention. As Wi be reaized, the invention is capabe of other and different embodiments and its severa detais are capabe of modi?ca tions in various obvious respects, a Without departing from 45 the spirit and the scope of the present invention. Accordingy, the drawings and detaied description are to be regarded as iustrative in nature and not as restrictive. BRIEF DESCRIPTION OF THE DRAWINGS FIG. 1 is a bock diagram showing a system for identifying compounds through iterative anaysis, in accordance With the present invention. FIG. 2 is a functiona bock diagram showing a compound server, in accordance With the present invention. FIG. 3 is a functiona bock diagram showing the software modues used by the compound engine of FIG. 2. FIG. 4 is a How diagram showing a method for identifying compounds through iterative anaysis, in accordance With the present invention. FIG. 5 is a How diagram showing the routine for preparing a corpus for use in the method of FIG. 4. FIG. 6 is a How diagram showing the routine for?nding compounds for use in the method of FIG FIG. 7 is a How diagram showing the routine for counting n-grams for use in the routine of FIG. 6. DETAILED DESCRIPTION System OvervieW FIG. 1 is a bock diagram showing a system 9 for identi fying compounds through iterative anaysis, in accordance With the present invention. A puraity of individua cients 12 are communicativey interfaced to a server 11 via an intemet Work 10, such as the Internet, or other form of communica tions network, as Woud be recognized by one skied in the art. The individua cients 12 are operated by users 19 Who transact requests for Web content, news messages, other types of content, and other operations through their respective ci ent 12. In genera, each cient 12 can be any form of computing patform connectabe to a network, such as the internetwork 10, and capabe of interacting With appication programs. Exempary exampes of individua cients incude, Without imitation, persona computers, digita assistances, smart ceuar teephones and pagers, ightweight cients, Worksta tions, dumb terminas interfaced to an appication server, and various arrangements and con?gurations thereof, as Woud be recognized by one skied in the art. The intemet Work 10 incudes various topoogies, con?gurations, and arrangements of network interconnectivity components arranged to interoperativey coupe With enterprise, Wide area and oca area networks and incude, Without imitation, con ventionay Wired, Wireess, sateite, optica, and equivaent network technoogies, as Woud be recognized by one skied in the art. For Web content retrieva and news message posting and retrieva, each cient 12 executes a Web browser and news reader appication 18 ( BroWser/ Reader ). Web content 25 is requested via a Web server 20 executing on the server 11. Simiary, news messages ( News Msgs ) 26 are posted and retrieved via a news server 21 aso executing on the server 11. In addition, speech, as communicated from a user 19 via a cient 12, can be recognized through a speech recognizer 23. Search resuts and other types of text can be cassi?ed by a text cassi?er 24. Other types of server functionaity can be provided, as Woud be recognized by one skied in the art. Note the Web browsing, news reading, speech recognition, and text cassi?cation functions coud be aso be impe mented separatey as stand aone appications, as are known in the art. The server 11 maintains an attached storage device 15 in Which the Web content 25, news messages 26, and other content 27 are stored. The Web content 25, news messages 26, and other content 27 coud aso be maintained remotey on other Web and news servers (not shown) interconnected either directy or indirecty via the intemetwork 10 and Which are preferaby accessibe by each cient 12. A compound server (not shown) identi?es compounds from a training cor pus and creates a ist of compounds, as further described beow With reference to FIG. 2. The compounds ist can be used by the search engine 22, speech recognizer 23, text cassi?er 24, and other components (not shown) on the server 11, one or more of the cients 12, or on other functiona components, as Woud be recognized by one skied in the art. In a further embodiment, a search engine 22 executes on the server 11 for processing queries for Web content 25, news messages 26, and other content 27. Each query describes or identi?es information, Which is potentiay retrievabe via either the Web server 20 or news server 21. Preferaby, each

11 5 query provides characteristics, typicay expressed as terms, incuding individua Words and compounds. A search engine 22, aso executing on the server 11, receives each query, identi?es matching Web content 25, news messages 26, and other content 27, and send back resuts conforming to the query preferences. Other styes, forms or de?nitions of que ries, query characteristics, and reated metadata are feasibe, as Woud be recognized by one skied in the art. The search engine 22 preferaby identi?es the Web content 25, news messages 26, and other content 27 best matching the search query terms to provide high quaity search resuts, such as described in S. Erin and L. Page, The Anatomy of a Large-Scae Hypertextua Search Engine (1998) and in US. Pat. No. 6,285,999, issued Sep. 4, 2001 to Page, the disco sures of Which are incorporated by reference. In identifying matching Web content 25, news messages 26, and other con tent 27, the search engine 22 operates on information charac teristics describing potentiay retrievabe content. Note the functionaity provided by the server 20, incuding the Web server 20, news server 21, search engine 22, speech recog nizer 23, and text cassi?er 24 coud be provided by a oosey or tighty-couped distributed or paraeized computing con?guration, in addition to a uniprocessing environment. The individua computer systems, incuding server 11 and cients 12, incude genera purpose, programmed digita com puting devices consisting of a centra processing unit (pro cessors 13 and 16, respectivey), random access memory (memories 14 and 17, respectivey), non-voatie secondary storage 15 and 28, such as a hard drive or CD ROM drive, network or Wireess interfaces, and periphera devices, incuding user interfacing means, such as a keyboard and dispay. Program code, incuding software programs, and data is oaded into the RAM for execution and processing by the CPU and resuts are generated for dispay, output, trans mitta, or storage. Compound Server FIG. 2 is a functiona bock diagram 30 showing a com pound server 31, in accordance With the present invention. The compound server 31 discovers and identi?es compounds based on tokens extracted from a text corpus 36 and stores the compounds in a compounds ist 37. The compounds server 31 incudes a compound engine 34, Which identi?es compounds through iterative anaysis, as further described beow With reference to FIG. 3. The compound server 31 maintains an attached storage device 35 in Which the text corpus 36 and compounds ist 37 are stored. The text corpus 36 consists of documents that incude Web content, news messages, and other content, incuding the Web content 25, news messages 26, and other content 27 stored by the server 11 (shown in FIG. 1), as We as documents from other sources, as is known in the art. The individua computer system, incuding the compound server 31, incude genera purpose, programmed digita com puting devices consisting of a centra processing unit (pro cessor 33), random access memory (memory 32), non-voa tie secondary storage 35, such as a hard drive or CD ROM drive, network or Wireess interfaces, and periphera devices, incuding user interfacing means, such as a keyboard and dispay. Program code, incuding software programs, and data is oaded into the RAM for execution and processing by the CPU and resuts are generated for dispay, output, trans mitta, or storage. Compound Server Components FIG. 3 is a functiona bock diagram 50 showing the soft Ware modues used by the compound engine 34 of FIG. 2. The US 7,555,428 B compound engine 34 consists of a corpus preparation com ponent 51, n-gram counter 52, and compound?nder 53. The corpus preparation component 51 and n-gram counter 52 both operate on the text corpus 36, Which consists of a set of documents ( Docs ) 61 that contain raw text provided as Web content, news messages, and other content. The corpus preparation component 51 evauates the text corpus 36 to construct an initia token vocabuary 57, as further described beow With reference to FIG. 5. The corpus preparation com ponent 51 incudes a parser 54 that tokenizes each document 61 in the text corpus 36. TokeniZing removes White space, punctuations, and formatting artifacts to form a cean ist of individua Words, Which each becomes a one-word token. The n-gram counter 52 is used by the compound?nder 53 to determine the frequencies of occurrences of unique n-grams Within the text corpus 36, as further described beow With reference to FIG. 7. The n-gram counter 52 generates a ist of the counts of occurrences of each n-gram 58 and a ist of unique n-grams 59. The compound?nder 53 uses the n-gram counts ist 58 to determine, for each n-gram of a desired ength, the ikeihood that the n-gram is a compound, as further described beow With reference to FIG. 6. The compound?nder 53 retrieves each unique n-gram from the unique n-gram ist 59. A ikei hood ratio component 55 determines the ikeihood of an n-gram being a compound and stores the computed ikeihood ratio in a ikeihood ratio ist 60. The compound?nder 53 identi?es those unique n-grams having a highest ikeihood of being compounds and generates a compounds ist 37. An iterator 56 repetitivey executes the operations of the n-gram counter 52 and compound?nder 53 to progressivey identify compounds of varying engths in the text corpus 36. Method OvervieW FIG. 4 is a How diagram showing a method 70 for identi fying compounds through iterative anaysis, in accordance With the present invention. The method 70 is described as a sequence of process operations or steps, Which can be executed, for instance, by the compound engine 34 of FIG. 2, or equivaent components. The method 70 performs two functions: preparing the text corpus 36 and generating the ist of compounds 37. Accord ingy, the text corpus 36 is prepared (bock 71), as further described beow With reference to FIG. 5. Next, compounds are found (bock 72), as further described beow With refer ence to FIG. 6. Routine then terminates. Preparing Corpus FIG. 5 is a How diagram showing the routine 80 for pre paring a corpus for use in the method 70 of FIG. 4. The purpose of this routine is to convert the documents 61 in the text corpus 36 into a raw set of individua Words, Which are stored as one-word tokens in a token vocabuary 57 (shown in FIG. 3). The routine begins by assembing the text in the individua documents 61 into the text corpus 36 (bock 81). If required, the text corpus 36 is parsed into individua tokens (bock 82) by removing White space, punctuations, and formatting arti facts, such as HTML tags and reated extraneous data, as is known in the art. Finay, an initia token vocabuary 57 is constructed from Words extracted from the text corpus 36 (bock 83). The routine then returns. Finding Compounds FIG. 6 is a How diagram showing the routine 100 for?nding compounds for use in the method 70 of FIG. 4. The purpose of this routine is to discover and identify compounds

12 7 Within the text corpus 36 based on evauation of measures of association for each potentia compound. The routine anayzes the text corpus 36 in an iterative manner. During each iteration (bocks ), a set of n-grams of a speci?ed ength having the highest ikeihood of co-occurrence, that is, of being compounds, are identi?ed. In subsequent iterations (bock 111), the ength of the com pounds is adjusted to identify further compounds. The use of measures of association, such as ikeihood hypotheses, aows spurious substrings and mutipe-token compounds to be avoided, as We as to e?icienty store ony those n-grams With stronger ikeihoods of co-occurrence. The routine begins by initiay identifying an initia ength for compounds and, in one embodiment, an upper imit on the number of compounds desired (bocks 101 and 102, respec tivey). The ength of the compounds changes during subse quent iterations (bock 111). The upper imit functions as a quaity?ter that imits the potentia compounds to a?xed number of candidates preferaby having the highest ikei hood of being compounds. Next, the n-grams occurring in the text corpus 36 are identi?ed and counted (bock 103), as further described beow With reference to FIG. 7. The count ing of n-grams generates the ist of n-gram counts 58 and the ist of unique n-grams 59. One or more of the n-grams of the speci?ed ength of compounds are seected from the ist of unique n-grams 59 (bock 104). Note that the entire set of n-grams of the speci?ed ength need not be seected. For exampe, n-grams appearing in a ist of known compounds, such as city names, coud be skipped for e?iciency. For each of the one or more seected n-grams (bock 105) for Which a compound determination is desired, the ikeihood of the seected n-gram being a compound is determined (bock 106). Likeihood determination may then continue With each of the remaining seected n-grams (bock 107). In the described embodiment, the ikeihood of a seected n-gram being a compound is determined using a measure of association, known as the ikeihood ratio method, as described in T. Dunning, Accurate Methods for the Statistics ofsurprise and Coincidence, Comp. Ling, Vo. 19, No. 1, pp (1993), the discosure of Which is incorporated by reference. Each seected n-gram is assigned a score equa to the ike ihood of the observed text corpus under the assumption that the n-gram is a compound, divided by the ikeihood of the observed text corpus under the assumption that the n-gram is not a compound, as expressed as equation (1): Where L(H-) is the ikeihood of observing the data HZ- under an independence hypothesis and L(Hc) is the ikeihood of observing the data HG under a coocation hypothesis. Assum ing a binomia distribution appies, the independence hypoth esis can be expressed as equation (2): Where t1 and t2 are a pair of tokens in the seected n-gram. Simiary, the coocation hypothesis can be expressed as equation (3): P(21)>P(2Z) (3) Where t1 and t2 are a pair of tokens in the seected n-gram. US 7,555,428 B Under the coocation hypothesis, for each sequence S: W1,...,Wn ofn tokens in the text corpus 36, MS) Wi be the greatest ikeihood ratio found by considering a possibe Ways to spit the n-token sequence into two contiguous parts. The n-token sequences S are sorted by 7»(S) and designated the K sequences With the owest MS) vaues as coocations. The coocation hypothesis L(HC) can be computed, as expressed as equation (4): L(1, [2 form compound) (4) mi 5:) L(n grarn does not form compound) ar Finay, the score 7» is cacuated by soving for L(H.) and L(Hc). Other equations, methods and processes for determin ing measures of association are visibe, as Woud be recog nized by one skied in the art. Next, up to the upper imit most ikey n-grams of the speci?ed ength are seected (bock 108) and added as com pound tokens to the token vocabuary 57 (bock 109). The token vocabuary 57 is reconstructed (bock 110) using the newy-added tokens in the token vocabuary 57. During reconstruction, constituent tokens, Which occur in each newy-added token are removed from the token vocabuary 57 and the newy-added tokens are subsequenty treated as one unit in the text corpus 36. For exampe, each occurrence of the Words San and Francisco Wi subsequenty be treated as a singe compound token San Francisco. If fur ther iterations are required (bock 111), the ength of the compounds is adjusted (bock 112). In the described embodi ment, ong compounds are identi?ed during the?rst iteration and progressivey shorter compounds are identi?ed in subse quent iterations. Each subsequent iteration begins With the identi?cation and recounting of the n-grams occurring in the text corpus 36 (bock 103). The n-grams must be recounted to account for those compound tokens newy-added and con stituent tokens newy-removed from the token vocabuary 57 during the previous iteration. Upon competion of the itera tions, the routine returns. Counting N-Grams FIG. 7 is a How diagram showing the routine 120 for counting n-grams for use in the routine 100 of FIG. 6. The purpose of this routine is to generate the ist of unique n-grams 59 and ist of n-gram counts 58. The maximum n-gram ength is identi?ed (bock 121). The maximum n-gram ength Wi equa the current compounds ength used in the routine to?nd compounds 100. Next, the unique n-grams having a number of tokens equa to the maxi mum n-gram ength are seected (bock 122). For each n-gram having a ength of up to the maximum n-gram ength (bock 123), the number of occurrences of each unique n-gram of that ength is counted against the token vocabuary 57 (bock 124). Counting continues for each subsequent n-gram ength (bock 125), after Which the routine returns. Whie the invention has been particuary shown and described as referenced to the embodiments thereof, those skied in the art Wi understand that the foregoing and other changes in form and detai may be made therein Without departing from the spirit and scope of the invention. What is caimed is: 1. A computer-impemented method for identifying com pounds in text, comprising: extracting a vocabuary of tokens from text; iterating from n>2 down to n:2 Where n decreases by one each iteration and in each iteration performing the actions of: identifying a puraity of unique n-grams in the

13 9 text, each n-gram being an occurrence in the text of n sequen tia tokens, each token being found in the vocabuary; divid ing each n-gram into n- pairs of two adjacent segments, Where each segment consists of at east one token; for each n-gram, cacuating a ikeihood of coocation for each pair of the n- pairs of two adjacent segments of the n-gram and determining a score for the n-gram based on a owest cacu ated ikeihood of coocation for the each of the n- pairs; identifying a set of n-grams having scores above a threshod; and adding the identi?ed set of n-grams as compound tokens to the vocabuary and removing constituent tokens that occur in the added compound tokens from the vocabuary, Wherein the iterating is performed by one or more processors. 2. The method of caim 1 Where cacuating a ikeihood of coocation for each pair of segments of the n- gram comprises determining a ikeihood ratio 7» for each pair of segments that is computed in accordance With the formua: A : mm mm Where L(H.) is a ikeihood of observing HZ. under an indepen dence hypothesis, L(Hc) is a ikeihood of observing Hc under a coocation hypothesis, and H is a pair of segments. 3. The method of caim 2 Where the L(HC) is computed for each pair of segments, t1, t2, in each n-gram in accordance With the formua: L(1, [2 form compound) argrnax UH.) [(n gram does not form compound) 4. The method of caim 2 Where, for each pair of segments, t1, t2, in each n-gram, the independence hypothesis comprises P(t2 t):p(t2?) and the coocation hypothesis comprises P(r2 r)>p(r2. 5. The method of caim 1 Where identifying a puraity of unique n-grams in the text comprises skipping n-grams appearing in a ist of known compounds. 6. A computer readabe storage medium on Which program code is stored, Which program code, When executed by a processor, causes the processor to perform operations com prising: extracting a vocabuary of tokens from text; iterating from n>2 down to n:2 Where n decreases by one each itera tion and in each iteration performing the actions of: identify ing a puraity of unique n-grams in the text, each n-gram being an occurrence in the text of n sequentia tokens, each token being found in the vocabuary; dividing each n-gram into n- pairs of two adjacent segments, Where each segment consists of at east one token; for each n-gram, cacuating a ikeihood of coocation for each of the n- pairs of two adjacent segments of the n-gram and determining a score for the n-gram based on a owest cacuated ikeihood of coo cation for the each of the n- pairs; identifying a set of n-grams having scores above a threshod; and adding the identi?ed set of n-grams as compound tokens to the vocabu ary and removing constituent tokens that occur in the added compound tokens from the vocabuary. 7. The computer-readabe storage medium of caim 6 Where cacuating a ikeihood of coocation for each pair of segments of the n-gram comprises determining a ikeihood ratio 7» for each pair of segments that is computed in accor dance With the formua: US 7,555,428 B where L(H-) is a ikeihood of observing HZ- under an indepen dence hypothesis, L(Hc) is a ikeihood of observing Hc under a coocation hypothesis, and H is a pair of segments. 8. The computer-readabe storage medium of caim 7 Where the L(Hc) is computed for each pair of segments, t1, t2, in each n-gram in accordance With the formua: L(t1, [2 form compound) argrnax UH [(n gram does not form compound) 9. The computer-readabe storage medium of caim 7 Where, for each pair of segments, t1, t2, in each n-gram, the independence hypothesis comprises P(t2 t):p(t2 ) and the coocation hypothesis comprises P(t2 t)>p(t2?). 10. The computer-readabe storage medium of caim 6 Where identifying a puraity of unique n-grams in the text comprises skipping n-grams appearing in a ist of known compounds. 11. A system comprising: a computer readabe storage medium on Which a program product is stored; and one or more processors con?gured to execute the program product and perform operations comprising: extracting a vocabuary of tokens from text; iterating from n>2 down to n:2 Where n decreases by one each iteration and in each iteration perform ing the actions of: identifying a puraity of unique n-grams in the text, each n-gram being an occurrence in the text of n sequentia tokens, each token being found in the vocabuary; dividing each n-gram into n- pairs of two adjacent seg ments, Where each segment consists of at east one token; for each n-gram, cacuating a ikeihood of coocation for each of the n- pairs of two adjacent segments of the n-gram and determining a score for the n-gram based on a owest cacu ated ikeihood of coocation for the each of the n- pairs; identifying a set of n-grams having scores above a threshod; and adding the identi?ed set of n-grams as compound tokens to the vocabuary and removing constituent tokens that occur in the added compound tokens from the vocabuary. 12. The system of caim 11 Where cacuating a ikeihood of coocation for each pair of segments of the n-gram com prises determining a ikeihood ratio 7» for each pair of seg ments that is computed in accordance With the formua: where L(H-) is a ikeihood of observing HZ- under an indepen dence hypothesis, L(HC) is a ikeihood of observing Hc under a coocation hypothesis, and H is a pair of segments. 13. The system of caim 12 Where the L(Hc) is computed for each pair of segments, t1, t2, in each n-gram in accordance With the formua: L(t1, [2 form compound) argrnax UH.) [(n gram does not form compound)

14 The system of caim 12 Where, for each pair of seg ments, t1, t2, in each n-gram, the independence hypothesis comprises P(t2 t):p(t2?) and the coocation hypothesis comprises P(t2 t)>p(t2. US 7,555,428 B The system of caim 11 Where identifying a puraity of unique n-grams in the text comprises skipping n-grams appearing in a ist of known compounds. * * * * *

15 UNITED STATES PATENT AND TRADEMARK OFFICE CERTIFICATE OF CORRECTION PATENT NO. : 7,555,428 B Page 1 of 1 APPLICATION NO. : 10/ DATED : June 30, 2009 INVENTOR(S) : Aexander Mark Franz and Brian Mich It is certified that error appears in the above-identi?ed patent and that said Letters Patent is hereby corrected as shown beow: Coumn 9; Lines at Caim 3; repace the current formua with: arg max we] L01,t2jorm compound) [.(n gram does not form compound) Coumn 10; Lines at Caim 8; repace the current formua with: arg max L011) L(:],tzjorm compound) L(n gram does not form compound) Coumn 10; Line 21 at Caim 9; repace the current formua with: P(t2[tr):P(t2t1) Coumn 10; Lines at Caim 13; repace the current formua With: L(t,t2 form compound) L(n gram does not form compound) Signed and Seaed this Twenty-?fth Day of August, 2009 David J. Kappos Director ofthe United States Patent and Trademark O?ice

16 UNITED STATES PATENT AND TRADEMARK office CERTIFICATE OF CORRECTION PATENT No. : 7,555,428 B1 Page 1 of 1 APPLICATION NO. : 10/ DATED : June 30, 2009 INVENTOR(S) : Franz et a1. It is certified that error appears in the above-identi?ed patent and that said Letters Patent is hereby corrected as shown beow: On the Tite Page: The first or soe Notice shoud read - Subject to any discaimer, the term of this patent is extended or adjusted under 35 U.S.C. 154(b) by 914 days. Signed and Seaed this Twenty-eighth Day of December, 2010 David J. Kappos Director ofthe United States Patent and Trademark O?ice