Efficiently building a parse tree from a regular expression?

Transcription

1 ACTA Informatica ManuscritNr. (will be inserted by hand later) Efficiently building a arse tree from a regular exression? Danny Dubé and Marc Feeley?? Université de Montréal C.P. 6128, succ. centreville, Montréal Canada H3C 3J7 Received: / Acceted: Abstract. We show in this aer that arsing with regular exressions instead of contextfree grammars, when it is ossible, is desirable. We resent efficient algorithms for erforming different tasks that concern arsing: roducing the external reresentation and the internal reresentation of arse trees; roducing all ossible arse trees or a single one. Each of our algorithms to roduce a arse tree from an inut string has an otimal time comlexity, linear with the length of the string. Moreover, ambiguous regular exressions can be used. Key words: regular exression arse tree arsing contextfree grammar ambiguity time comlexity 1. Introduction In language theory, regular exressions and contextfree grammars are among the most imortant tools used to recognize languages. These are simle models and efficient algorithms exist to make them ractical. Finite deterministic and nondeterministic automata, ushdown automata, LL(k) and LR(k) arsers are all art of this technology. In many alication fields, such as comiling, the interest is not only in recognizing languages, but also in arsing them. That is, given a grammar and a string that it generates, constructing a arse tree for the string which indicates how the string can be derived from the grammar. The task of arsing is traditionally done using contextfree grammars. Still there are cases where a kind of arsing with regular exressions is desired. For examle, it is the case when one wants to recover a floating oint number from a string like 1.234e5. The string as a whole matches the regular exression describing floating oint numbers, but arts of it (sign, integer and fraction, exonent)? Research suorted in art by the Natural Sciences and Engineering Research Council of Canada.?? fdube,feeleyg@iro.umontreal.ca.

2 2 D. Dubé and M. Feeley must be extracted in order to obtain useful information. For single regular exressions, such extraction is usually done in an ad hoc way. Parsing is often done using algorithms with linear comlexity based on LR(k) or LL(k) grammars. Unfortunately, the set of regular exressions does not fall in the class of LR(k) grammars 1 ; ambiguous regular exressions are not LR(k). To be able to arse with any grammar, even an ambiguous one, we must use general tools such as Earley s arser ([E70]) or the dynamic rogramming method (see [HU79], CYK algorithm). These algorithms have cubic comlexity in the worst case. There are some tools which allow to arse secifically with regular exressions. In the Regex ackage (see [RE87]) and in the regular exression tools included in the Emacs text editor (see [Emacs]), one can mark a subexression of a regular exression to indicate that a substring matching the marked subexression is desired. The result is either the substring itself or a range (air of integers) indicating where the characters can be found in the original string. If the subexression is matched many times, there is a substring for each match (or for only one of them). This facility is not comletely satisfactory. It can ose roblems when more than one subexression is marked (using \( and \) ), like in: (a+\(c\)?b+\(c\)?)* It is not ossible to unambiguously determine where the c s aear in the reetitions, even if all the occurrences of c are noted. For examle, the strings aacbcab and abcacbb both cause the same informations to be saved: one c string er mark. So, extraction by marks (and by ad hoc techniues) is not owerful enough to rovide a clean and general solution. What we show in this aer is that comlete arsing with general regular exressions can be done in linear time. We resent a set of techniues that rovide comlete arse trees and that do not imose restrictions on the kind of regular exressions one can use. The techniues are efficient in the sense that they have a linear time comlexity in the length of the string to arse. Our aroach is interesting since there are many cases where regular exressions are the most natural descrition one has. Indeed, although the set of regular languages is smaller than the set of LR(k) languages, many useful languages are regular. Configuration files for many systems can be described by regular exressions. In some cases, assembly source files can be regular. If the assembler does not suort advanced features like macros, sources files basically contain lines with an otional label, an oerator and its oerands, lus some assembler directives and sectioning commands. All this gives a regular language. Finally, in the rocessing of natural languages, regular exressions sometimes are the most aroriate tools. They can be used in various lexiconrelated tasks, in robust arsing, and in automatic language identification. They can be used as an aroximate definition of the language to recognize, allowing fast rocessing due to their regularity. In some situations, like in the rocessing of Finnish, where the words don t have a secific order, they are a good solution to an otherwise difficult roblem (see [LJN85]). One can find more comlete descritions of the use of regular exressions in natural language rocessing in [MN97] and [GM89]. We resent our techniues in the following order. Section 2 resents definitions and conventions that we use. Among which is the descrition of the arse trees themselves. 1 Strictly seaking, the regular exressions are not contextfree grammars. But we can give to a regular exression a natural euivalent grammar. For examle, r = r 0 jr 00 can be transformed into the substitution rules T r! T r 0 j T r 00, and r = r 0, into T r! j T r 0 T r.

3 Regular exressions and arse trees 3 We give them an external and an internal reresentation. Section 3 resents a nondeterministic automaton used to generate the external reresentation of a arse tree and Section 4 resents another one for the internal reresentation. The second is the most useful but the first is simler. Section 5 resents an imrovement of the receding techniues using deterministic automata. Finally, Section 6 resents a techniue to obtain an imlicit reresentation of the set of all arse trees. This techniue is needed when a regular exression is ambiguous and there are many arse trees corresonding to different arses of the same string. We resent many observations and roerties throughout the text. For brevity, we do not include their roof. The roofs are usually simle but long and tedious. We believe the reader should easily be able to convince himself of the truth of the statements. 2. Definitions, conventions and tools 2.1. Notation Let be the set of symbols used to write the strings that we arse. We denote the emty string as. The length of a string w is jwj. Strings are concatenated by writing them one after the other. The techniues described here are based rimarily on automata. The automata are simly reresented by grahs. The start state is indicated by a > sign. The acceting states are deicted as double circles. Every edge has a string attached to it, indicating what refix of the inut string should be consumed when a transition is made through the edge. A wtransition is a transition that consumes w. Most of the automata that we introduce are nondeterministic. Their nodes are usually denoted using the letters, or s. For a deterministic automaton, we use a hat on the node labels (as in ˆ). L(r) is the language of the regular exression r. L(G) is the language of the contextfree grammar G. L(A(r)) is the language acceted by the automaton A(r). Paths in a grah are denoted by writing the seuence of node labels. We searate labels by dashes if there can be confusion between them. Concatenated aths are written one after the other and searated by dots (). In a concatenation, the last node of a ath must be the same as the first node of the following ath. In general, we try to use distinct letters to denote different things; such as v and w for strings and r for regular exressions Regular exressions We define the set R of regular exressions as the smallest set such that the following hold: R B fg [ [ f(r) j r 2 R E g R F R B [ fr j r 2 R B g R T R F [ fr 0 r 1 : : : r n?1 j n 2 ^ r i 2 R F ; 8 0 i < ng R E R T [ fr 0 jr 1 j : : : jr n?1 j n 2 ^ r i 2 R T ; 8 0 i < ng R R E

4 4 D. Dubé and M. Feeley We refer to define the set of regular exressions with set ineualities rather than with a contextfree grammar because it simlifies the remaining of the resentation. The set of regular exressions that we consider is almost the standard one. We omit the ositive closure (r + ) and the otional oerator (r? ) which are simle modifications of the Kleene closure (r ). Note also that we did not introduce the emty regular exression (;), which corresonds to the emty language. First, the roblem of finding a arse tree for a string w matching ; never occurs. Second, a comlex exression containing the exression ; can easily be reduced to the exression ; itself or to an exression that does not contain ; Parse trees We first describe the kind of arse trees that our automata should create from a string. An imortant issue is that a regular exression may reresent an ambiguous contextfree grammar. The exression (ajbjab) is ambiguous because some strings such as aab have more than one ossible arse tree. Let T denote the set of all ossible trees and T : R! 2 T the function giving the set of valid arse trees from a regular exression and a string. That is, T (r; w) is the set of all valid arse trees coming from the decomosition of w according to the grammar reresented by the regular exression r. For our urose, arse trees are built with symbols of, with lists, and with selectors. More formally, we can define T as the smallest set such that: c 2 T ; 8c 2 #i : t 2 T ; 8i 2 N; 8t 2 T [t 0 ; t 1 ; : : : ; t n?1 ] 2 T ; 8n 0; 8t i 2 T ; 0 i < n Let us describe formally the arse trees in T (r; w). Note that T (r; w) 6= ; if and only if w 2 L(r). T (; w) = T (c; w) = T ((r 0 ); w) = T (r 0 ; w) T (r 0 ; w) = T (r 0 : : : r n?1 ; w) = f[ ]g; if w = ;; otherwise fcg; if w = c (where c 2 ) ;; otherwise 8 >< >: [t 0; : : : ; t n?1 ] 8 < : [t 0; : : : ; t n?1 ] n 0 ^ 80 i < n; 9w i 2, s.t. w = w 0 : : : w n?1 ^ t i 2 T (r 0 ; w i ); 80 i < n 80 i < n; 9w i 2, s.t. w = w 0 : : : w n?1 ^ t i 2 T (r i ; w i ); 80 i < n T (r 0 j : : : jr n?1 ; w) = f#i : t i j 0 i < n ^ t i 2 T (r i ; w)g 9 >= >; 9 = ; This is the meaning of each case: Case r =. The corresonding arse tree is the emty list: [ ].

5 Regular exressions and arse trees 5 Case r = c 2. The arse tree returned in this case is the symbol c itself. Case r = (r 0 ). The arentheses are used only to override the riority of the oerators. They do not affect the shae of the arse trees. Case r = r 0. The arse tree returned in this case is a list containing subtrees. That is, if r matches w, then w is a concatenation of substrings w i all matching r 0, and there is one subtree er submatch. Case r = r 0 : : : r n?1. A arse tree is a list of length n. Each subtree of that list is a arse tree of a substring w i of w according to r i. Case r = r 0 j : : : jr n?1. Each arse tree is a selector indicating which subexression has ermitted a match and in which way. It is obvious that a arse tree gives a comlete descrition of a articular match. Given the regular exression used in the match, we can interret the arse tree and find a derivation for the string. Note that with our arse tree reresentation two different regular exressions both matching the same string can have the same set of arse trees. For examle, T (a ; aaa) = T (aaa; aaa) = f[a; a; a]g; in the first case, the list comes from a reetition; in the second case, the list comes from a concatenation. On the other hand, arse trees coming from two different strings must be different since a arse tree contains all the symbols of the string in order. Here are a few examles. The last two show ambiguous regular exressions. The last one shows a regular exression for which all the strings in its language have an infinite number of arse trees. T (ajbjc; b) = f#1 : bg T (a b c ; aabbbcc) = f[[a; a]; [b; b; b]; [c; c]]g T ((ajaa) ; aa) = f[#0 : a; #0 : a]; [#1 : [a; a]]g T ((a ) ; ) = f[ ]; [[ ]]; [[ ]; [ ]]; [[ ]; [ ]; [ ]]; : : :g 2.4. External reresentation of arse trees Note that the arse trees in T are only mathematically defined and we have not given them an actual reresentation. They first have an external reresentation. That reresentation consists in writing them down as strings. We choose that their textual reresentation is identical to the one we used in the text before. The alhabet that we use to write them is [ f [ ; ; ; ] ; # ; : g [ D, where D is the set of digits in a certain base. For examle, D = f0; 1; 2; 3g in base 4. Lists are written as a finite seuence of elements between suare brackets ([ and ]) and searated by commas, such as [a; b; c]. Selectors are written as a natural number and a arse tree, each of these being receded by # and :, resectively, as in #3 : [c; a] Internal reresentation of arse trees The arse trees in T also have an internal reresentation. That reresentation consists in building trees using data structures. This section describes those data structures. The data structures needed are: the emty list, airs, symbols and selectorairs. Pairs are simly the traditional twofield data structures used to build lists. However, we will use a nontraditional reresentation of lists: lasttofirst lists. That is, the last element

6 6 D. Dubé and M. Feeley of a list is immediately accessible while an access to the first reuires the traversal of all the airs of the list. This reresentation is unusual but it is as easy to imlement as the traditional one. We use it for technical reasons exlained in Section 4.2. In our reresentation of lists, the first field of a air references an element of the list (the last element) and the second field references the rest of the list (the first elements). When we need to build a air, we use the following function: make air(t 1 ; t 2 ), where t 1 is an element added at the end of the list t 2. A selectorair has two fields. The first contains an integer which reresents the number of the alternative chosen in a match. To build a selectorair, we use: make selector(i; t). To evaluate the comlexity of our algorithms, we assume that the basic oerations on these data structures can be done in constant time. 3. External reresentation nondeterministic automaton This section describes how to build a nondeterministic automaton and how to use it to build the external reresentation of the arse trees, and exlains some of its roerties. The automaton is very similar to the finite nondeterministic automaton that we use in language theory to recognize a regular language. The main difference is the addition of outut strings to some edges of the automaton. When the automaton makes a transition through such an edge, it oututs the string that is indicated. We start by exlaining the rocess of roducing a arse tree using the automaton. Then we resent the construction rules and an examle. Next, we mention some roerties related to the automaton. Finally, we discuss erformance issues Obtaining a arse tree Let w be generated by r and let A E (r) be the external reresentation automaton corresonding to r. In order to describe recisely the comutations haening with our automaton, we use configurations. A configuration is a tule (; w; v) that indicates that the automaton is currently in state, that there is still the string w to be read from the inut, and that v is the string that has been outut until now. The initial configuration of the automaton is ( 0 ; w; ), where 0 is the start state. As with a finite state automaton, a transition can be made using an edge if: the edge starts from the current state; and, the (inut) string associated with the edge is a refix of the remaining inut. So, let us assume that our automaton is in configuration (; w ; v ) and there exists an edge e from to that consumes w 0 and oututs v 0 (if there is no outut string, is the default). If w 0 is a refix of w, that is, if there exist w such that w = w 0 w, then the edge can be taken. By doing so, the automaton goes into the configuration (; w ; v ), where v = v v 0. The automaton has successfully outut a arse tree if we can make transitions u to a configuration ( f ; ; v), where f is the acceting state. In such a case, v is the desired arse tree. In cases where r is ambiguous and w has many derivations, v is only one of the ossible arse trees Construction Figure 1 resents the construction rules for the external reresentation automaton A E (r) corresonding to a regular exression r. The construction rules are described in a

7 Regular exressions and arse trees 7 A E (): A E (c): A E ((r 0 )): A E (r 0 ): > [ A E (r 0 : : : r m) (where m = n? 1 and n 2): A E (r 0 j : : : jr m): (where m = n? 1 and n 2) > > [ ] > c c > A E (r 0 ) [ 0 A 0 E (r 0 ) 6 ; ; ] 0 A E (r 0 m A E (r m) m 0 A E (r 0 ) 0 m A E (r m) #0 : #m : Fig. 1. Construction rules for the external reresentation automaton. recursive manner. These rules are similar to those resented in [ASU86] and [HU79] with the excetion that we add outut strings on the edges. Note that the start and acceting states of an automaton included in a bigger one lose their secial meaning Examle Let us see an examle to hel understand the way the automaton works. We will consider the regular exression r = (ajbjab) and the string w = ab. Clearly, r is ambiguous and w has more than one derivation. In fact, there are two different arse trees for w according to r: T ((ajbjab) ; ab) = f[#2 : [a; b]]; [#0 : a; #1 : b]g The first corresonds to the derivation where the Kleene closure causes one reetition of the internal exression and ab is chosen among the three choices. The second corresonds to the derivation where two reetitions occur and first a, then b are chosen. Figure 2 shows the automaton A E (r). Each ath from state 0 to state 1 that consumes w will cause a arse tree to be outut. One can see that there are exactly two such aths. Those are and

8 8 D. Dubé and M. Feeley???? > 0 i i i 2?? a 4 a 5 i #0 b i6 i 7 3 i : b 6 : 8 i 10 i 11 i i ḇ [ ; i 9 i?? ] a a ; b 1 im ] Fig. 2. The external nondeterministic automaton associated to (ajbjab). First ath Second ath (0; ab; ) (0; ab; ) (2; ab; [ ) (2; ab; [ ) (8; ab; [#2 : ) (4; ab; [#0 : ) (10; ab; [#2 : [ ) (5; b; [#0 : a ) (11; b; [#2 : [a ) (3; b; [#0 : a ) (12; b; [#2 : [a; ) (2; b; [#0 : a; ) (13; ; [#2 : [a; b ) (6; b; [#0 : a; #1 : ) (9; ; [#2 : [a; b] ) (7; ; [#0 : a; #1 : b ) (3; ; [#2 : [a; b] ) (3; ; [#0 : a; #1 : b ) (1; ; [#2 : [a; b]] ) (1; ; [#0 : a; #1 : b] ) Result: [#2 : [a; b]] Result: [#0 : a; #1 : b] Fig. 3. Two traces of A E ((ajbjab) ) on ab. Figure 3 shows the seuence of configurations that the automaton goes through for each of these aths. It is imortant to note in each seuence the first configuration (0; w; ) and the last (1; ; v) Observations We can make many observations about the automaton A E (r) created from r: Correct language: The automaton accets the same language as the one generated by the regular exression, that is: L(A E (r)) = L(r). Indeed, our method is similar to that resented in many aers (see [ASU86, HU79]). Linear size: The size of the automaton (the number of states and edges) grows linearly with the size of the regular exression. We measure the size of r in elementary symbols:, c (c 2 ), (, ), and j. Soundness: The arse trees for a string outut using the automaton are correct arse trees for the string. More formally, if is a ath from the start state to the acceting state of A E (r) and consumes w, then causes a valid arse tree t 2 T (r; w) to be outut. Exhaustiveness: The automaton can outut any of the valid arse trees. Let t 2 T (r; w), then there exists a ath going from the start state to the acceting state of A E (r) and consuming w that causes t to be outut.

9 Regular exressions and arse trees 9 Uniueness: There is only one way to roduce a articular tree. Let 1 6= 2, two different aths traversing A E (r). Then the trees t 1 and t 2 that are outut during the two traversals are different. Note that the last three roerties concern A E (r) for r in general. That is, they are true even if r is a subexression of another exression. It imlies that a ath traversing A E (r) may cause the consumtion of just a art of the inut string that is fed into the global automaton. Similarly, it may cause the roduction of just a subtree of the whole tree. This fact is imortant if one wants to rove these roerties Comlexity We have not yet given a totally comlete method to roduce a arse tree. The nondeterministic automaton creates a valid arse tree as long as we have found a ath that fulfills all the conditions. We describe a method to find such a ath in the next aragrahs and another one in Section 5. In order to find an aroriate ath, we simly forget momentarily that there are outut strings on the edges and use an algorithm simulating a nondeterministic automaton such as the one described in [HU79]. Then, by following the ath and oututting the strings, we get our arse tree. We do not reeat the descrition of the simulation techniue here. What is imortant to us is its comlexity. The techniue finds a ath in time O(jwj n), where w is the string that we want to arse and n is the number of states of the automaton. Since we know that the number of states of the automaton grows linearly with the size of the corresonding regular exression r, the techniue takes a time in O(jwj jrj). When we have found an aroriate ath, we simly have to follow it through A E (r) and outut the strings indicated on the edges. It takes O(jwj jrj) to follow the ath and to outut the arse tree. So, the whole rocess of writing a arse tree by simulation takes O(jwj jrj) in time. In situations where r is known, we can consider that the techniue takes O(jwj) in time. Still, the hidden constant heavily deends on r. Even if the stes of finding a ath and following it both have a O(jwjjrj) comlexity, it is reasonable to think that the search for a ath has a greater hidden constant in the average case. It could be rofitable to get a faster algorithm for finding a valid ath. Section 5 resents a method to search for a ath in time O(jwj), where the hidden constant does not deend on r, although r must be known a riori. 4. Internal reresentation nondeterministic automaton Section 3 describes an automaton that we can use to outut the external reresentation of a arse tree from a string generated by a regular exression. What we want to do here is to build the internal reresentation of a arse tree using data structures. So, instead of adding outut strings to some edges of the automaton, we add construction commands. When a ath traverses the automaton and consumes the whole inut string, the seuence of commands that is emitted forms a recie to build a arse tree. We start by describing the instrumentation that is reuired to build internal arse trees. Then we resent the construction rules and an examle. We omit to list the roerties of this automaton because they are very similar to those of the revious automaton.

10 10 D. Dubé and M. Feeley 4.1. Instrumentation In order to build the internal reresentation of a arse tree, we augment the automaton with a stack and some stack oerations The stack The function of the stack is to contain the different ieces of the arse tree under construction. This stack does not turn the automaton into a ushdown automaton. The automaton can only send commands (stack oerations) to it and cannot read from it. On most edges of our automaton there are stack oerations to erform. So, when a transition is made through an edge, a art of the inut string may be consumed and a stack oeration may be erformed on the stack. The automaton is built in such a way that, if a ath is followed from the start state to the acceting state and if that ath has caused the consumtion of the inut string then the resulting effect on the stack is that a valid arse tree has been ushed on it. Now, we define the stack functions. The three basic stack functions are ush, o and to. ush(t; s) returns the stack s with the new tree t added on to. o(s) returns s without its to element. The argument stack s is not altered by this oeration. to(s) returns the to element of the stack s The stack oerations The oerations that can be found on the edges of the automata are ush, snoc, and sel. Push adds a tree on to of the stack. The ush oerator always has a constant tree as first argument. For examle, ush [ ] is an oerator that takes a stack and ushes the emty list on to of it. Snoc 2 takes the to two elements of the stack, grous them in a air and laces the result back on to of the stack. One can see that we can build arbitrarily long lists on the stack with successive ush and snoc oerators. Sel takes the to element of the stack and encasulates it in a selector. The sel oerator always has a constant integer as first argument. (ush t)(s) = ush(t; s) snoc(s) = ush(make air(to(s); to(o(s))); o(o(s))) (sel i)(s) = ush(make selector(i; to(s)); o(s)) 4.2. Construction Figure 4 resents the construction of the internal reresentation nondeterministic automaton. The function A I returns the internal automaton associated to a regular exression. 2 This name is the reverse of cons, which is a common name for the function that adds an element in front of a list. We use snoc in reference to our reversed imlementation of lists.

11 Regular exressions and arse trees 11 A I (): A I (c): A I ((r 0 )): A I (r 0 ): > > ush [ ] > c ush c > A I (r 0 ) ush [ 0 A 0 I (r 0 ) ush 6 snoc snoc A I (r 0 : : : r m) (where m = n? 1 and n 2): > 0 A I (r 0 ) 0 m A I (r m) m ush [ ] snoc snoc A I (r 0 j : : : jr m): (where m = n? 1 >?? 0 A I (r 0 ) 0 and m A I (r m) m sel m Fig. 4. Construction rules for the internal reresentation automaton. The rules for r = r 0 and r = r 0 : : : r n?1 are those involved in the creation of lists. It is because of these rules and the fact that A I (r) reads its inut from left to right (i.e. normally) that we reuire a secial imlementation of lists (see Section 2.5). It would be ossible to use only traditional lists if we changed the two rules mentioned above and made the automaton read its inut backwards (from right to left) Examle We illustrate the oeration of the automaton with the same regular exression and string as in the examle for the external reresentation automaton: r = (ajbjab) and w = ab. Figure 5 shows A I (r). There are two aths that consume w: and Figure 6 shows the seuence of configurations that the automaton goes through along each ath. The first ath is articularly illustrative of the imortance of using the reversed imlementation of lists.

12 12 D. Dubé and M. Feeley ush [ ] i i i a 4 i ush a 2?? i b ush[ ] sel > 0 i 7 ush sel i8 10 a 11 i ḇ i i 3 snoc 6 9?? sel 2 i i ush[ ] ush a snoc snoc i ush b snoc Fig. 5. The internal nondeterministic automaton associated to (ajbjab). im 1 First ath Second ath (0; ab; [ ]) (0; ab; [ ]) (2; ab; [[ ]]) (2; ab; [[ ]]) (8; ab; [[ ]]) (4; ab; [[ ]]) (10; ab; [[ ]; [ ]]) (5; b; [a; [ ]]) (11; b; [a; [ ]; [ ]]) (3; b; [#0 : a; [ ]]) (12; b; [[a]; [ ]]) (2; b; [[#0 : a]]) (13; ; [b; [a]; [ ]]) (6; b; [[#0 : a]]) (9; ; [[a; b]; [ ]]) (7; ; [b; [#0 : a]]) (3; ; [#2 : [a; b]; [ ]]) (3; ; [#1 : b; [#0 : a]]) (1; ; [[#2 : [a; b]]]) (1; ; [[#0 : a; #1 : b]]) Result: [#2 : [a; b]] Result: [#0 : a; #1 : b] Fig. 6. Two traces of A I ((ajbjab) ) on ab. 5. Deterministic automaton As exlained in revious sections, we exect the search for a valid ath in a nondeterministic automaton to be uite exensive. So we develo here a deterministic euivalent to the nondeterministic automaton generated for a regular exression r. We arbitrarily choose to give the exlanations using the internal automaton, but these also aly to the external automaton. The algorithm using the deterministic automaton can find a valid ath in time O(jwj). Note the ath itself has generally a length in O(jrj jwj). The reason why it takes less time to find the ath than it would take to outut it is because the algorithm only creates a skeleton of the real ath. It only returns the identity of numerous subaths. The concatenation of those subaths would give the desired ath Construction Basically, the construction of the deterministic automaton is almost identical to the usual construction used in language theory ([ASU86, HU79]), so we will not rewrite it here. The difference resides in the fact that we collect some information about the relation between the deterministic automaton and the nondeterministic one. We exlain what is the nature of this information, but we do not formally describe the way it should be comuted. We assume it is sufficiently straightforward.

13 Regular exressions and arse trees 13 We must kee two matrices of information, which we denote as functions here: f( ˆ; c; ) where ˆ is a deterministic state (corresonding to a set of nondeterministic states), c 2 and a nondeterministic state. The function f returns a ath from some state in ˆ to the state that goes through a ctransition first, and then, through zero or more transitions. It is easy to identify because it is the first state in. g() where is a nondeterministic state. The function g returns a ath from the nondeterministic start state to that goes through transitions only. In order to describe the usage of the automaton, we first introduce some variables. We have a string w = c 0 : : : c n?1, where c i 2 for 0 i n?1. start and acc are the start state and the acceting state of the nondeterministic automaton A I (r), resectively. The deterministic start state is ˆ start. The deterministic states are distinguished by the hat they have on their name. Naturally, ˆ start is the closure of start. The first ste in using the deterministic automaton consists in rocessing the inut string w with it. One has to note each state in which the automaton goes into. Let us call ˆ 0, : : :, ˆ n those states. Note that start 2 ˆ start = ˆ 0 and acc 2 ˆ n. The second ste consists in recovering the comlete ath traversing A I (r) and consuming w. We do this by finding n + 1 subaths, 0 to n, in reverse and concatenating them together. i consumes c i?1, 1 i n. 0 starts at start and n ends at acc. Here is the techniue: Initialize the tail of the ath. n = acc Find the subath i that consumes c i?1. i = f( ˆ i?1 ; c i?1 ; i ) i?1 = the first state in i i = n; n? 1; : : : ; 1 Find the subath going to 0. 0 = g( 0 ) Recover the whole ath. = 0 : : : n By taking care of just keeing a reference to each subath and not coying them entirely, we then have a skeleton of a valid ath through the nondeterministic automaton in time O(n) (where the hidden constant does not deend on r). Subseuently, the skeleton allows one to make a traversal of the real ath in time O(n jrj). One might worry, with reason, about the size of the matrix for f. In the worst case, the argument ˆ can take 2 O(jrj) values, the argument c, jj values and, O(jrj) values. Each answer can be of length O(jrj). In ractice, though, ˆ takes much less than 2 O(jrj) values. Nevertheless, it is ossible to reexress f in terms of three smaller matrices. 3 It is also ossible to use table comression on f as it tends to be sarse and as many of its defined entries are similar. Such aroaches are beyond the scoe of this aer. 3 Smaller in the sense of being only one or twodimensional, instead of threedimensional as f.

14 14 D. Dubé and M. Feeley 5.2. Examle The following examle illustrates the way the deterministic automaton works. Let r = (ab) and w = abab. Then A I (r) is: ush[ ] 5 i i 2 i 4 i i ḇ 6 i ush[ ] ush[ ] ush a snoc 7 3 ush b snoc 6 snoc > The deterministic automaton  I (r) we can build from A I (r) is: i snoc im 1 > ˆ0 ˆ1 ˆ2 a where ˆ0 = f0; 1; 2; 4g ˆ1 = f5; 6g ˆ2 = f1; 2; 3; 4; 7g The functions f and g associated to the deterministic automaton are: f = ˆ0 ˆ1 ˆ2 a b a b a b 0?????? 1??? 6731?? 2??? 6732?? 3??? 673?? 4??? 67324?? 5 45??? 45? 6 456??? 456? 7??? 67?? g = ? ? 6? 7? If we feed  I (r) with the inut string w, the automaton goes through this seuence of states: We recover the ath in A I (r) this way: ˆ 0 = ˆ0; ˆ 1 = ˆ1; ˆ 2 = ˆ2; ˆ 3 = ˆ1; ˆ 4 = ˆ2 4 = f( ˆ 3 ; b; 1) = 6? 7? 3? 1 ( 3 = 6) 3 = f( ˆ 2 ; a; 3 ) = 4? 5? 6 ( 2 = 4) 2 = f( ˆ 1 ; b; 2 ) = 6? 7? 3? 2? 4 ( 1 = 6) 1 = f( ˆ 0 ; a; 1 ) = 4? 5? 6 ( 0 = 4) 0 = g( 0 ) = 0? 2? 4 = = 0? 2? 4? 5? 6? 7? 3? 2? 4? 5? 6? 7? 3? 1 It is easy to verify that this ath goes from the start state to the acceting state, consumes w and causes the ush of the valid arse tree.

15 Regular exressions and arse trees Reresentation of the set of arse trees The revious sections describe ways to obtain a arse tree reresenting the decomosition of a string w according to a regular exression r. However, only one of the ossible arse trees is built. In cases where there are more than one ossible arse tree for a string, one might be interested in obtaining the comlete set of trees. That is, obtaining T (r; w) instead of a certain t 2 T (r; w) Considerations There are some imortant considerations we must take into account if we intend to get an algorithm able to return T (r; w). First, it is unrealistic to try to return an exlicit reresentation of T (r; w). The set is, in general, too big. For examle, the cardinality of T ((a ) ; ) is infinite, that of T ((aja) ; a n ) is exonential, and that of T ((a ) k+1 ; a n ) is olynomial of degree k. So, it is clear that we must return an imlicit reresentation of T (r; w). Second, existing techniues for contextfree grammars such as dynamic rogramming and the Earley arsing (see [HU79] and [E70]) roduce an imlicit reresentation of the set of arse trees. It is a treelike reresentation: each node of this tree contains the set of all the different subtrees that could be referenced by the node. That is, there is a node for the set of arse trees of each nonterminal and each substring of the inut string. We say that the node comacts all the arse trees of a substring generated by a nonterminal (see [JM93]). Unfortunately, both algorithms have a time comlexity of O(jwj 3 ), where w is the string to arse. In articular, they still exhibit this time comlexity even if we restrict the contextfree grammars to be only translations of regular exressions. Since our interest is in efficiency and since we restrict ourselves to regular exressions, we are able to resent an algorithm roducing an imlicit reresentation of T (r; w) in linear time. It takes the form of a contextfree grammar which generates the set of arse trees. That is, the grammar G r;w roduced is such that L(G r;w ) = T (r; w). For the sake of simlicity, we resent an algorithm roducing a grammar that generates the external reresentation of the trees Construction We describe how to roduce the contextfree grammar G r;w such that L(G r;w ) = T (r; w). The idea behind the algorithm roducing it is simle. Let r 2 R, w 2 L(r) and A E (r). The grammar is created in such a way that it mimics A E (r) consuming the inut string w. That is, doing a substitution using a rule corresonds to making a transition in the automaton. Figure 7 gives the algorithm. The algorithm first roduces rules simulating transitions and ctransitions, resectively. The main nonterminal corresonds to the initial configuration. Finally, the last rule is the only one that can end a derivation, which is euivalent to recognizing the reaching of the acceting state after the consumtion of the inut string. The indices of P i; mean that i symbols of the inut string have been consumed and that A E (r) is in state. The set of strings that P i; can generate is the set of strings that A E (r) can outut if it is in state and has already consumed the first i symbols

16 16 D. Dubé and M. Feeley MakeGrammar(A E (r); w) f Suose w = c 0 : : : c n?1, where c i 2 g f Suose start and acc are the start and acceting states of A E (r) g For i = 0 to n (inclusive) For each edge e in A E (r) f e = i win i wout g If w in = Then Produce rule P i;! w outp i; Else If i < n And w in = c i Then Produce rule P i;! w outp i+1; Mark P 0;start as the main nonterminal Produce rule P n;acc! Fig. 7. Algorithm roducing a contextfree grammar that generates an imlicit reresentation for T (r; w). of w. Formally, here is the relation between the nonterminals of the grammar and the configurations in which A E (r) can be: vp i; corresonds to (; w 00 ; v) where 9w 0 s.t. w = w 0 w 00 and jw 0 j = i This relation allows us to easily obtain many roerties of the grammar by translating roerties of the external reresentation automaton. The algorithm has a time comlexity in O(jrj jwj). The jwj factor comes from the outer loo, which enumerates each osition in the inut string. The inner loo iterates on the edges of A E (r). Recall that the number of edges in A E (r) grows linearly with the size of r. This justifies the factor jrj. Since a roduction may be roduced at each iteration of the inner loo, the algorithm generates a grammar which has O(jrj jwj) roductions. 4 Our algorithm is otimal in its time comlexity in the following sense. Once the regular exression is known, the algorithm is able to generate G r;w in time O(jwj), which is the best any algorithm can do. The algorithm generally roduces a grammar that is not clean. 5 That is, there are useless and unreachable nonterminals. We could have given an adated algorithm that avoids this roblem, but it would have been more comlex and, as we oint out in Section 6.5.1, it is not a serious roblem Examle We illustrate our algorithm with a simle examle. Let r = w = abc. Figure 8 shows A E (r). Figure 9 shows the grammar G = MakeGrammar(A E (r); w). In order to make things clear, we have searated the roductions corresonding to the transitions and the c i transitions, the final roduction and the main nonterminal. 4 The fact that our algorithm creates a grammar with O(jrj jwj) roductions does not automatically imly that its time comlexity is O(jrj jwj). This is because the right hand side of the roductions is not necessarily bounded in length. In articular, when w out = #i :, it might be arbitrarily long. It takes O(log i) to denote i in, say, decimal digits. Nevertheless, this is not a big roblem because an easy modification of the algorithm can eliminate this roblem. That is, we can modify it so that it creates O(jrj jwj) roductions, each having a right hand side of at most four symbols. It is somehow technical and we won t resent it here. We consider this roblem solved for the remaining of the text. 5 Some authors refer to talk about reduced grammars.

17 Regular exressions and arse trees 17 > 0i i a 2 [ a 3 i i ; b 4 b 5 i i c ; 6 i 7 1 im c ] Fig. 8. The external reresentation automaton corresonding to the regular exression abc. P 0;0! [P 0;2 P 1;0! [P 1;2 P 2;0! [P 2;2 P 3;0! [P 3;2 P 0;3! ; P 0;4 P 1;3! ; P 1;4 P 2;3! ; P 2;4 P 3;3! ; P 3;4 P 0;5! ; P 0;6 P 1;5! ; P 1;6 P 2;5! ; P 2;6 P 3;5! ; P 3;6 P 0;7! ]P 0;1 P 1;7! ]P 1;1 P 2;7! ]P 2;1 P 3;7! ]P 3;1 P 0;2! ap 1;3 P 1;4! bp 2;5 P 2;6! cp 3;7 Main nonterminal: P 0;0 P 3;1! Fig. 9. The grammar MakeGrammar(A E (abc); abc). Note that the roductions corresonding to the transitions are resent for each osition in the inut. On the other hand, those corresonding to the dtransitions are resent only for the osition where the next inut symbol is d. Note also the resence of useless and unreachable nonterminals. For examle, P 0;3 is unreachable and P 3;6 is useless. There is only one derivation that we can make with this grammar: P 0;0! [P 0;2! [ap 1;3! [a; P 1;4! [a; bp 2;5! [a; b; P 2;6! [a; b; cp 3;7! [a; b; c]p 3;1! [a; b; c] The string that is roduced by the derivation is effectively the only arse tree in T (r; w). Note that the ath that we can extract from the derivation (0? 2? 3? 4? 5? 6? 7? 1) is a ath traversing A E (r) and consuming w that oututs [a; b; c] Observations The corresondence between the aths through an automaton and the derivations with the grammar allows us to obtain many interesting results uite easily. Adeuacy: Any string generated by G r;w is a valid arse tree of w according to r, and conversely. That is: L(G r;w ) = T (r; w). Unambiguousness: The grammars G r;w roduced by our algorithm are unambiguous. Notice that the grammars (and the languages they generate) roduced by our algorithm are regular. So it should be ossible to reresent the same languages with regular exressions or finite automata. However, we refer contextfree grammars for two reasons. First, we have no guarantee that the smallest regular exression is as short as its corresonding contextfree grammar because regular exressions do not have the sharing ability that contextfree grammars have. Second, even if it is ossible to roduce a finite automaton as comact as its corresonding contextfree grammar, we would lose in clarity due to the necessity to formally describe it.

18 18 D. Dubé and M. Feeley 6.5. Use of the grammar Useless and unreachable nonterminals The grammars generated by our algorithm have useless and unreachable nonterminals. The unreachable ones do not ose a real roblem since they simly cause a grammar to have more roductions than necessary. The useless nonterminals are a more serious roblem because they may make the generation of the set of arse trees more costly. For examle, a generation of the trees by simle recursive descent would lose much time reetitively trying to generate trees and backtracking from a useless nonterminal. Of course, one might think of an otimization consisting in noting the nonterminals that have not generated any sentence. This otimization could simly be relaced by a rerocessing hase of the grammar which consists in detecting and removing the useless nonterminals. An algorithm to remove those is described in [HU79]. The idea is to mark every nonterminal that is useful and then to remove the unmarked ones. The comlexity of this cleaning algorithm is in O(L l), where L is the number of roductions and l is the length of the longest righthand side among the roductions. We know that the number of roductions in our grammars is in O(jrj jwj) and that the length of the righthand side of our roductions is bounded by a constant. So the overall comlexity of the algorithm is O(jrj jwj). So the cleaning of a grammar allows the generation of the set of arse trees using a naive recursive descent algorithm. Under the condition that there is only a finite number of trees, naturally Infinite sets of arse trees For certain r 2 R and w 2 L(r), there may be an infinite number of arse trees. In such a case, it is obviously not ossible to generate all the arse trees. Even if one is ready to enumerate as many trees as necessary to find a articular one, some care must be taken to make sure that the enumeration eventually reaches every ossible tree. A grammar that has been cleaned before the search allows the search to roceed without loss of time because of useless substitutions. Another otion is to ay secial attention in the design of the regular exressions in order to avoid situations where jt (r; w)j = 1. This way, one can enumerate the arse trees in any order without ever risking to fall into an infinite comutation Relation between the grammar and the regular exression Unfortunately, the relation between the grammar generated by our algorithm and its regular exression is not obvious. The grammar is concetually related to the automaton and not to the regular exression. There is no relation between a nonterminal and, say, a subexression of the regular exression. This may ose a roblem in alications where a sohisticated heuristic search through the set of arse trees is used to find a good one instead of brute force. In natural language arsing, for examle, lausibility estimation can be done on the basis of the structure of the regular exression. But it would be difficult to adat it to work on the basis of the aths in the automaton.

19 Regular exressions and arse trees 19 We believe that nonterminals like the following would be more intuitive. The nonterminal Q r0 ;w would generate the trees for the substring 0 w0 of w according to the subexression r 0 of r. That kind of nonterminal is closely related to the regular exression and, conseuently, related to the intention insiring the design of the regular exression. Unfortunately, a grammar based on these nonterminals would be much more exensive to generate because the number of nonterminals grows as a uadratic function of jwj instead of as a linear one. Since we are interested in efficiency, we consider that an algorithm roducing such a grammar is beyond the scoe of this article Grammars for the internal reresentation It may not be clear what grammars for the internal reresentation could be since the internal reresentation of a arse tree is not even a string. Nevertheless, the grammars we have described may be adated to become an imlicit reresentation of the internal trees. We know that, in order to roduce the internal reresentation of a arse tree, one must aly aroriate stack oerations on a stack and take the to element. The aroriate seuences of stack oerations are those rescribed by traversals of the internal reresentation automaton. So we can adat the algorithm of Figure 7 to build grammars generating these seuences of stack oerations. From each seuence that is generated by a grammar, one can build a arse tree. As with the grammars for the external reresentation, those for the internal reresentation suffer from the same oor relationshi with the original regular exression and the arse trees. A more intuitive reresentation would be one similar to the one used by the dynamic rogramming and Earley s arsers: one node for each subexression r 0 of r and each substring w 0 of w which contains the different arses of w 0 according to r 0. Two mathematical models that could rovide such reresentations are regular tree automata and regular tree grammars (see [B67] and [H68]). However, such reresentations necessarily have a uadratic number of nodes (states or nonterminals), one for each substring, and so, they cannot be roduced efficiently. We consider that these aroaches are beyond the scoe of this article. 7. Conclusion In this aer, we have shown that arsing with regular exressions, when it is ossible, is desirable. There are efficient algorithms to do most of the tasks one might be interested in: external reresentation vs. internal reresentation; one arse tree vs. all of them. Moreover, there is no restriction on the articular regular exressions one might use. Each of our algorithms to roduce arse trees from an inut string has a time comlexity that is linear in the length of the string. So they are all otimal. Even the construction of nondeterministic automata is efficient (but not the construction of deterministic ones, though). It is not ossible to achieve such efficiency by simly considering a regular exression as a contextfree grammar and using a arsing tool based on grammars. Indeed, some regular exressions are ambiguous and the fast algorithms based on LL(k) and LR(k) grammars cannot be used. Moreover, the general algorithms such as the Earley arser and the dynamic rogramming method show their worst case cubic time

20 20 D. Dubé and M. Feeley comlexity on some ambiguous grammars; some of which come directly from regular exressions. Unfortunately, one of our results is not comletely satisfactory. The grammar that we roduce as the imlicit reresentation of the set of arse trees (Section 6.5.3) is too artificial. The link between the original regular exression and the grammar reresenting the arse trees of a string is not natural. It might comlicate the rocess of searching in the set for a arse tree corresonding to a certain criterion. This sacrifice seems unavoidable to have an algorithm with linear time comlexity. References [ASU86] Aho, A.V., Sethi, R., Ullman, J.D.: Comilers Princiles, Techniues and Tools. AddisonWesley [B67] Brainerd, W.S.: Tree generating systems and tree automata. Ph.D. Thesis. Deartment of Mathematics, Purdue University. University microfilms, inc. Ann Arbor, Michigan [E70] Earley, J.: An Efficient ContextFree Parsing Algorithm. In Communications of the ACM. February 1970 [Emacs] The Emacs text editor. Provided by Gnu: gnu.org [GM89] Gazdar, G., Mellish, C.: Natural Language Processing in PROLOG. AddisonWesley [H68] Hossley, R.: Finitetree automata and!automata. Technical reort 102, Project MAC, Massachusetts Institute of Technology. Cambridge, Massachusetts [HU79] Hocroft, J.E., Ullman, J.D.: Introduction to Automata Theory, Languages, and Comutation. AddisonWesley [JM93] Jones, E.K., Miller, L.M.: The L Parsing Algorithm. Technical reort CSTR939. Victoria University of Wellington. Deartment of Comuter Science. December 1993 [LJN85] Lehtola, A., Jäinen, H., Nelimarkka, E.: Languagebased Environment for Natural Language Parsing. In Proceedings of the 2 nd Conference of the Euroean Chater of the Association for Comutational Linguistics [MN97] Mitkov, R., Nicolov, N.: Recent Advances in Natural Language Processing. Amsterdam, Philadelhia: John Benjamins [RE87] Sencer, H.: The Regex ackage. A regular exression ackage written in C, for Unix, available on the Internet This article was rocessed by the author using the LaT E X style file ljour1 from SringerVerlag.