Factoring Synchronous Grammars By Sorting

Transcription

1 Factoring Syncronous Grammars By Sorting Daniel Gildea Computer Science Dept. Uniersity of Rocester Rocester, NY Giorgio Satta Dept. of Information Eng g Uniersity of Padua I- Padua, Italy Hao Zang Computer Science Dept. Uniersity of Rocester Rocester, NY Abstract Syncronous Context-Free Grammars (SCFGs) ae been successfully exploited as translation models in macine translation applications. Wen parsing wit an SCFG, computational complexity grows exponentially wit te lengt of te rules, in te worst case. In tis paper we examine te problem of factorizing eac rule of an input SCFG to a generatiely equialent set of rules, eac aing te smallest possible lengt. Our algoritm works in time O(n log n), for eac rule of lengt n. Tis improes upon preious results and soles an open problem about recognizing permutations tat can be factored. Introduction Syncronous Context-Free Grammars (SCFGs) are a generalization of te Context-Free Grammar (CFG) formalism to simultaneously produce strings in two languages. SCFGs ae a wide range of applications, including macine translation, word and prase alignments, and automatic dictionary construction. Variations of SCFGs go back to Ao and Ullman (9) s Syntax-Directed Translation Scemata, but also include te Inersion Transduction Grammars in Wu (99), wic restrict grammar rules to be binary, te syncronous grammars in Ciang (00), wic use only a single nonterminal symbol, and te Multitext Grammars in Melamed (00), wic allow independent rewriting, as well as oter tree-based models suc as Yamada and Knigt (00) and Galley et al. (00). Wen iewed as a rewriting system, an SCFG generates a set of string pairs, representing some translation relation. We are concerned ere wit te time complexity of parsing suc a pair, according to te grammar. Assume ten a pair wit eac string aing a maximum lengt of N, and consider an SCFG G of size G, wit a bound of n nonterminals in te rigt-and side of eac rule in a single dimension, wic we call below te rank of G. As an upper bound, parsing can be carried out in time O( G N n+ ) by a dynamic programming algoritm maintaining continuous spans in one dimension. As a lower bound, parsing strategies wit discontinuous spans in bot dimensions can take time Ω( G N c n ) for unfriendly permutations (Satta and Peserico, 00). A natural question to ask ten is: Wat if we could reduce te rank of G, presering te generated translation? As in te case of CFGs, one way of doing tis would be to factorize eac single rule into seeral rules of rank strictly smaller tan n. It is not difficult to see tat tis would result in a new grammar of size at most G. In te time complexities reported aboe, we see tat suc a size increase would be more tan compensated by te reduction in te degree of te polynomial in N. We tus conclude tat a reduction in te rank of an SCFG would result in more efficient parsing algoritms, for most common parsing strategies. In te general case, normal forms wit bounded rank are not admitted by SCFGs, as sown in (Ao and Ullman, 9). Noneteless, an SCFG wit a rank of n may not necessarily meet te worst case of Ao and Ullman (9). It is ten reasonable to ask if our SCFG G can be factorized, and wat is te smallest rank k < n tat can be obtained in tis way. Tis paper answers tese two questions, by proiding an algoritm tat factorizes te rules of an input SCFG, resulting in a new, generatiely equialent, SCFG wit rank k as low as possible. Te algoritm works in time O(n log n) for eac rule, regardless of te rank k of te factorized rules. As discussed aboe, in tis way we aciee an improement of te parsing time for SCFGs, obtaining an upper bound of O( G N k+ ) by using a parsing strategy tat maintains continuous

2 ,,,,,,,,,,,,,, Figure : Two permutation trees. Te permutations associated wit te leaes can be produced by composing te permutations at te internal nodes. spans in one dimension. Preious work on tis problem as been presented in Zang et al. (00), were a metod is proided for casting an SCFG to a form wit rank k =. If generalized to any alue of k, tat algoritm would run in time O(n ). We tus improe existing factorization metods by almost a factor of n. We also sole an open problem mentioned by Albert et al. (00), wo pose te question of weter irreducible, or simple, permutations can be recognized in time less tan Θ(n ). Syncronous CFGs and permutation trees We begin by describing te syncronous CFG formalism, wic is more rigorously defined by Ao and Ullman (9) and Satta and Peserico (00). Let us consider strings defined oer some set of nonterminal and terminal symbols, as defined for CFGs. We say tat two suc strings are syncronous if some bijectie relation is gien between te occurrences of te nonterminals in te two strings. A syncronous context-free grammar (SCFG) is defined as a CFG, wit te difference tat it uses syncronous rules of te form [A α, A α ], wit A, A nonterminals and α, α syncronous strings. We can use production [A α, A α ] to rewrite any syncronous strings [γ A γ, γ A γ ] into te syncronous strings [γ α γ, γ α γ ], under te condition tat te indicated occurrences of A and A be related by te bijection associated wit te source syncronous strings. Furtermore, te bijectie relation associated wit te target syncronous strings is obtained by composing te relation associated wit te source syncronous strings and te relation associated wit syncronous pair [α, α ], in te most obious way. As in standard constructions tat reduce te rank of a CFG, in tis paper we focus on eac single syncronous rule and factorize it into syncronous rules of lower rank. If we iew te bijectie relation associated wit a syncronous rule as a permutation, we can furter reduce our factorization problem to te problem of factorizing a permutation of arity n into te composition of seeral permutations of arity k < n. Suc factorization can be represented as a tree of composed permutations, called in wat follows a permutation tree. A permutation tree can be conerted into a set of k-ary SCFG rules equialent to te input rule. For example, te input rule: [ X A () B () C () D () E () F () G () H (), X B () A () C () D () G () E () H () F () ] yields te permutation tree of Figure (left). Introducing a new grammar nonterminal X i for eac internal node of te tree yields an equialent set of smaller rules: [ X X () X(), X X() X() ] [ X X () X(), X X () X() ] [ X A () B (), X B () A () ] [ X C () D (), X C () D () ] [ X E () F () G () H (), X G () E () H () F () ] In te case of stocastic grammars, te rule corresponding to te root of te permutation tree is assigned te original rule s probability, wile all oter rules, associated wit new grammar nonterminals, are assigned probability. We process eac rule of an input SCFG independently, producing an equialent grammar wit te smallest possible arity. Factorization Algoritm In tis section we specify and discuss our factorization algoritm. Te algoritm takes as input a permutation defined on te set {,, n}, representing a rule of some SCFG, and proides a permutation tree of arity k n for tat permutation, wit k as small as possible. Permutation trees coering a gien input permutation are unambiguous wit te exception of sequences of binary rules of te same type (eiter inerted or straigt) (Albert et al., 00). Tus, wen factorizing a permutation into a permutation

3 tree, it is safe to greedily reduce a subsequence into a new subtree as soon as a subsequence is found wic represents a continuous span in bot dimensions of te permutation matrix associated wit te input permutation. For space reasons, we omit te proof, but empasize tat any greedy reduction turns out to be eiter necessary, or equialent to te oter alternaties. Any sequences of binary rules can be rearranged into a normalized form (e.g. always leftbrancing) as a postprocessing step, if desired. Te top-leel structure of te algoritm exploits a diide-and-conquer approac, and is te same as tat of te well-known mergesort algoritm (Cormen et al., 990). We work on subsequences of te original permutation, and merge neigboring subsequences into successiely longer subsequences, combining two subsequences of lengt i into a subsequence of lengt i+ until we ae built one subsequence spanning te entire permutation. If eac combination of subsequences can be performed in linear time, ten te entire permutation can be processed in time O(n log n). As in te case of mergesort, tis is an application of te so-called master teorem (Cormen et al., 990). As te algoritm operates, we will maintain te inariant tat we must ae built all subtrees of te target permutation tree tat are entirely witin a gien subsequence tat as been processed. Tis is analogous to te inariant in mergesort tat all processed subsequences are in sorted order. Wen we combine two subsequences, we need only build nodes in te tree tat coer parts of bot subsequences, but are entirely witin te combined subsequence. Tus, we are looking for subtrees tat span te midpoint of te combined subsequence, but ae left and rigt boundaries witin te boundaries of te combined subsequence. In wat follows, tis midpoint is called te split point. From tis inariant, we will be guaranteed to ae a complete, correct permutation tree at te end of last subsequence combination. An example of te operation of te general algoritm is sown in Figure. Te top-leel structure of te algoritm is presented in function KARIZE of Figure. Tere may be more tan one reduction necessary spanning a gien split point wen combining two subsequences. Function MERGE in Fig- A permutation matrix is a way of representing a permutation, and is obtained by rearranging te row (or te columns) of an identity matrix, according to te permutation itself.,,,,,,,,,,,,,,, Figure : Recursie combination of permutation trees. Top row, te input permutation. Second row, after combination into sequences of lengt two, binary nodes ae been built were possible. Tird row, after combination into sequences of lengt four; bottom row, te entire output tree. ure initializes certain data structures described below, and ten cecks for reductions repeatedly until no furter reduction is possible. It looks first for te smallest reduction crossing te split point of te subsequences being combined. If SCAN, described below, finds a alid reduction, it is committed by calling REDUCE. If a reduction is found, we look for furter reductions crossing eiter te left or rigt boundary of te new reduction, repeating until no furter reductions are possible. Because we only need to find reductions spanning te original split point at a gien combination step, tis process is guaranteed to find all reductions needed. We now turn to te problem of identifying a specific reduction to be made across a split point, wic inoles identifying te reduction s left and rigt boundaries. Gien a subsequence and candidate left and rigt boundaries for tat subsequence, te alidity of making a reduction oer tis span can be tested by erifying weter te span constitutes a permuted sequence, tat is, a permutation of a contiguous sequence of integers. Since te starting permutation is defined on a set {,,, n}, we ae no repeated integers in our subsequences, and te aboe condi-

4 function KARIZE() initialize wit identity mapping min max (0.. ); mergesort core for size ; size ; size size * do for min 0; min < -size+; min min + * size do di = min + size - ; max min(, min + *size - ); MERGE(min, di, max); function MERGE(min, di, max) initialize sort [min..max] according to [i]; sort min[min..max] according to [i]; sort max[min..max] according to [i]; merging sorted list takes linear time initialize for i min; i max; i i + do [ [i] ] i; ceck if start of new reduced block if i = min or min[i] min[i-] ten min i; min[ [i] ] min; for i max; i min; i i - do ceck if start of new reduced block if i = max or max[i] max[i+] ten max i ; max[ [i] ] max; look for reductions if SCAN(di) ten REDUCE(scanned reduction); wile SCAN(left) or SCAN(rigt) do REDUCE(smaller reduction); function REDUCE(left, rigt, bot, top) for i bot..top do min[i] left; max[i] rigt; for i left..rigt do min[i] bot; max[i] top; print reduce: left..rigt ; Figure : KARIZE: Top leel of algoritm, identical to tat of mergesort. MERGE: combines two subsequences of size i into new subsequence of size i+. REDUCE: commits reduction by updating min and max arrays. tion can be tested by scanning te span in question, finding te minimum and maximum integers in te span, and cecking weter teir difference is equal to te lengt of te span minus one. Below we call tis condition te reduction test. As an example of te reduction test, consider te subsequence (,,, ), and take te last tree elements, (,, ), as a candidate span. We see tat and are te minimum and maximum integers in te corresponding span, respectiely. We ten compute =, wile te lengt of te span minus one is, implying tat no reduction is possible. Howeer, examining te entire subsequence, te minimum is and te maximum is, and =, wic is te lengt of te span minus one. We terefore conclude tat we can reduce tat span by means of some permutation, tat is, parse te span by means of a node in te permutation tree. Tis reduction constitutes te -ary node in te permutation tree of Figure. A triial implementation of te reduction test would be to tests all combinations of left and rigt boundaries for te new reduction. Unfortunately, tis would take time Ω(n ) for a single subsequence combination step, wereas to aciee te oerall O(n log n) complexity we need linear time for eac combination step. It turns out tat te boundaries of te next reduction, coering a gien split point, can be computed in linear time wit te tecnique sown in function SCAN of Figure. We start wit left and rigt candidate boundaries at te two points immediately to te left and rigt of te split point, and ten repeatedly ceck weter te current left and rigt boundaries identify a permuted sequence by applying te reduction test, and moe te left and rigt boundaries outward as necessary, as soon as missing integers are identified outside te current boundaries, as explained below. We will sow tat, as we moe outward, te number of possible configurations acieed for te positions of te left and te rigt boundaries is linearly bounded in te lengt of te combined subsequence (as opposed to quadratically bounded). In order to efficiently implement te aboe idea, we will in fact maintain four boundaries for te candidate reduction, wic can be isualized as te left, rigt, top and bottom boundaries in te permutation matrix. No explicit representation of te permutation matrix itself is constructed, as tat would require quadratic time. Rater, we

5 Figure : Permutation matrix for input permutation (left) and witin-subsequence permutation (rigt) for subsequences of size four. maintain two arrays:, wic maps from ertical to orizontal positions witin te current subsequence, and wic maps from orizontal to ertical positions. Tese arrays represent te witinsubsequence permutation obtained by sorting te elements of eac subsequence according to te input permutation, wile keeping eac element witin its block, as sown in Figure. Witin eac subsequence, we alternate between scanning orizontally from left to rigt, possibly extending te top and bottom boundaries (Figure lines 9 to ), and scanning ertically from bottom to top, possibly extending te left and rigt boundaries (lines 0 to ). Eac extension is forced wen, looking at te witin-subsequence permutation, we find tat some element is witin te current boundaries in one dimension but outside te boundaries in te oter. If te distance between ertical boundaries is larger in te input permutation tan in te subsequence permutation, necessary elements are missing from te current subsequence and no reduction is possible at tis step (line ). Wen all necessary elements are present in te current subsequence and no furter extensions are necessary to te boundaries (line 0), we ae satisfied te reduction test on te input permutation, and make a reduction. Te trick used to keep te iteratie scanning linear is tat we skip te subsequence scanned on te preious iteration on eac scan, in bot te orizontal and ertical directions. Lines and of Figure perform tis skip by adancing te x and y counters past preiously scanned regions. Considering te orizontal scan of lines 9 to, in a gien iteration of te wile loop, we scan only te items between newleft and left and between rigt and newrigt. On te next iteration of te wile loop, te newleft boundary as moed furter to te left, : function SCAN (di) : left ; : rigt ; : newleft di; : newrigt di + ; : newtop ; : newbot ; : wile do orizontal scan 9: for x newleft; x newrigt ; do 0: newtop max(newtop, max[x]); : newbot min(newbot, min[x]); skip to end of reduced block : x max[min[x]] + ; skip section scanned on last iter : if x = left ten : x rigt + ; : rigt newrigt; : left newleft; te reduction test : if newtop - newbot < : [[newtop]] - [[newbot]] ten 9: return (0); ertical scan 0: for y newbot; y newtop ; do : newrigt : max(newrigt, max[y]); : newleft min(newleft, min[y]); skip to end of reduced block : y max[min[y]] + ; skip section scanned on last iter : if y = bot ten : y top + ; : top newtop; : bot newbot; if no cange to boundaries, reduce 9: if newrigt = rigt 0: and newleft = left ten : return (, left, rigt, bot, top); Figure : Linear time function to ceck for a single reduction at split point di.

6 wile te ariable left takes te preious alue of newleft, ensuring tat te items scanned on tis iteration are distinct from tose already processed. Similarly, on te rigt edge we scan new items, between rigt and newrigt. Te same analysis applies to te ertical scan. Because eac item in te permutation is scanned only once in te ertical direction and once in te orizontal direction, te entire call to SCAN takes linear time, regardless of te number of iterations of te wile loop tat are required. We must furter sow tat eac call to MERGE takes only linear time, despite tat fact tat it may inole many calls to SCAN. We accomplis tis by introducing a second type of skipping in te scans, wic adances past any preiously reduced block in a single step. In order to skip past preious reductions, we maintain (in function REDUCE) auxiliary arrays wit te minimum and maximum positions of te largest block eac point as been reduced to, in bot te orizontal and ertical dimensions. We use tese data structures (min, max, min, max) wen adancing to te next position of te scan in lines and of Figure. Because eac call to SCAN skips items scanned by preious calls, eac item is scanned at most twice across an entire call to MERGE, once wen scanning across a new reduction s left boundary and once wen scanning across te rigt boundary, guaranteeing tat MERGE completes in linear time. An Example In tis section we examine te operation of te algoritm on a permutation of lengt eigt, resulting in te permutation tree of Figure (rigt). We will build up our analysis of te permutation by starting wit indiidual items of te input permutation and building up subsequences of lengt,, and finally. In our example permutation, (,,,,,,, ), no reductions can be made until te final combination step, in wic one permutation of size is used, and one of size. We begin wit te input permutation along te bottom of Figure a. We represent te intermediate data structures, min, and max along te ertical axis of te figure; tese tree arrays are all initialized to be te sequence (,,, ). Figure b sows te combination of indiidual items into subsequences of lengt two. Eac new subsequence of te array is sorted according to a) b) c) max min min max max min min max max min min max Figure : Steps in an example computation, wit input permutation on left and witinsubsequence permutation described by array on rigt. Panel (a) sows initial blocks of unit size, (b) sows combination of unit blocks into blocks of size two, and (c) size two into size four. No reductions are possible in tese stages; example continued in next figure.

7 a) min max min max b) min max min max Left and rigt boundaries are initialized to be adjacent to orizontal split point. Vertical scan sows left and rigt boundaries must be extended. Permutation of size four is reduced. c) min max min max d) min max min max Searc for next reduction: left and rigt boundaries initialized to be adjacent to left edge of preious reduction. Vertical scan sows rigt boundary must be extended. e) min max min max f) min max min max Horizontal scan sows top boundary must be extended. Vertical scan sows left boundary must be extended. Permutation of size fie is reduced. Figure : Steps in scanning for final combination of subsequences, were =. Area witin current left, rigt, top and bottom boundaries is saded; darker sading indicates a reduction. In eac scan, te span scanned in te preious panel is skipped oer.

8 te ertical position of te dots in te corresponding columns. Tus, because [] = > [] =, we swap and in te array. Te algoritm cecks weter any reductions can be made at tis step by computing te difference between te integers on eac side of eac split point. Because none of te pairs of integers in are consecutie, no reductions are made at tis step. Figure c sows te combination te pairs into subsequences of lengt four. Te two split points to be examined are between te second and tird position, and te sixt and seent position. Again, no reductions are possible. Finally we combine te two subsequences of lengt four to complete te analysis of te entire permutation. Te split point is between te fourt and fift positions of te input permutation, and in te first orizontal scan of tese two positions, we see tat [] = and [] =, meaning our top boundary will be and our bottom boundary, sown in Figure a. Scanning ertically from position to, we see orizontal positions,,, and, giing te minimum,, as te new left boundary and te maximum,, as te new rigt boundary, sown in Figure b. We now perform anoter orizontal scan starting at position, but ten jumping directly to position, as orizontal positions and were scanned preiously. After tis scan, te minimum ertical position seen remains, and te maximum ertical position is still. At tis point, because we ae te same boundaries as on te preious scan, we can stop and erify weter te region determined by our current boundaries as te same lengt in te ertical and orizontal dimensions. Bot dimensions ae lengt four, meaning tat we ae found a subsequence tat is continuous in bot dimensions and can safely be reduced, as sown in Figure d. After making tis reduction, we update te min array to ae all s for te newly reduced span, and update max to ae all sixes. We ten ceck weter furter reductions are possible coering tis split point. We repeat te process of scanning orizontally and ertically in Figure c-f, tis time skipping te span just reduced. One furter reduction is possible, coering te entire input permutation, as sown in Figure f. Conclusion Te algoritm aboe not only identifies weter a permutation can be factored into a composition of permutations, but also returns te factorization tat minimizes te largest rule size, in time O(n log n). Te factored SCFG wit rules of size at most k can be used to syncronously parse in time O(N k+ ) by dynamic programming wit continuous spans in one dimension. As mentioned in te introduction, te optimal parsing strategy for SCFG rules wit a gien permutation may inole dynamic programming states wit discontinuous spans in bot dimensions. Weter tese optimal parsing strategies can be found efficiently remains an interesting open problem. Acknowledgments Tis work was partially supported by NSF ITR IIS-09 and NSF ITR IIS-000. References Albert V. Ao and Jeffery D. Ullman. 9. Te Teory of Parsing, Translation, and Compiling, olume. Prentice-Hall, Englewood Cliffs, NJ. M. H. Albert, M. D. Atkinson, and M. Klazar. 00. Te enumeration of simple permutations. Journal of Integer Sequences, (0..): pages. Daid Ciang. 00. A ierarcical prase-based model for statistical macine translation. In Proceedings of ACL-0, pages 0. Tomas H. Cormen, Carles E. Leiserson, and Ronald L. Riest Introduction to algoritms. MIT Press, Cambridge, MA. Micel Galley, Mark Hopkins, Kein Knigt, and Daniel Marcu. 00. Wat s in a translation rule? In Proceedings of HLT/NAACL. I. Dan Melamed. 00. Multitext grammars and syncronous parsers. In Proceedings of HLT/NAACL. Giorgio Satta and Enoc Peserico. 00. Some computational complexity results for syncronous context-free grammars. In Proceedings of HLT/EMNLP, pages 0 0. Dekai Wu. 99. Stocastic inersion transduction grammars and bilingual parsing of parallel corpora. Computational Linguistics, (): 0. Kenji Yamada and Kein Knigt. 00. A syntaxbased statistical translation model. In Proceedings of ACL-0. Hao Zang, Liang Huang, Daniel Gildea, and Kein Knigt. 00. Syncronous binarization for macine translation. In Proceedings of HLT/NAACL.