Principles of Programming Languages

Transcription

1 Principles of Progrmming Lnguges h"p:// 15/ Prof. Andre Corrdini Deprtment of Computer Science, Pis Lesson 6! Towrds Gener=on of Lexicl Anlyzers Finite stte utomt (FSA) From Regulr Expressions to FSA The Lex- Flex lexicl nlyzer genertor

2 We hve seen tht: Tokens re defined with regulr expressions RE à Trnsi=on digrms à code, y hnd!!! Exmple: id letter ( letter digit ) * letter or digit strt 9 letter cse 9: c = nextchr(); if (isletter(c)) stte = 10; else stte = fil(); rek; cse 10: c = nextchr(); if (isletter(c)) stte = 10; else if (isdigit(c)) stte = 10; else stte = 11; rek; 10 other 11 * return(gettoken(), instll_id()) We present more system?c nd formlized pproch 2

3 Design of Lexicl Anlyzer Genertor 1. From the RE of ech token uild n NFA (non- determinis=c finite utomton) tht ccepts the sme regulr lnguge 2. Comine the NFAs into single one 3. Either 1. Simulte directly the NFA, or 2. Determinize the NFA nd simulte the resul=ng DFA (determinis=c FA) 4. Solve conflicts Optionl regulr expressions NFA DFA 3

4 Non- determinis=c Finite Automt An NFA is 5-tuple (S, Σ, δ, s 0, F) where S is finite set of sttes Σ is finite set of symols, the lphet δ is mpping from S (Σ {}) to set of sttes δ : S (Σ {}) à 2 S s 0 S is the strt stte F S is the set of ccepting (or finl) sttes 4

5 Trnsi=on Grph An NFA cn e digrmmticlly represented y leled directed grph clled trnsition grph strt S = {0,1,2,3} Σ = {,} s 0 = 0 F = {3} 5

6 Trnsi=on Tle The mpping δ of n NFA cn e represented in trnsition tle δ(0,) = {0,1} δ(0,) = {0} δ(1,) = {2} δ(2,) = {3} Stte Input Input 0 {0, 1} {0} 1 {2} 2 {3} 6

7 The Lnguge Defined y n NFA An NFA ccepts n input string w (over Σ) if nd only if there is t lest one pth with edges leled with symols from w in sequence from the strt stte to some ccep=ng stte in the trnsi=on grph Note tht - trnsi=ons do not contriute with symols A stte trnsi=on from one stte to nother on the pth is clled move The lnguge defined y n NFA A is the set of input strings it ccepts, denoted L(A) 7

8 Exmples strt A Which NFA, if ny, ccepts???? Which re the lnguges ccepted y A 1 nd A 2? 8 A 1 strt

9 From Regulr Expression to NFA: Thompson s Construc=on Given RE, it uilds y structurl induc;on NFA tht: Accepts exctly the lnguge of the RE Hs single ccep=ng stte Hs no trnsi=ons to the ini=l stte Hs no trnsi=ons from the finl stte 9

10 Thompson s Construc=on r : RE à N(r) : NFA strt i f strt i f r 1 r 2 r 1 r 2 strt i strt i N(r 1 ) N(r 2 ) f r* strt Complexity: liner in the size of the RE i N(r 1 ) N(r 2 ) N(r) f f 10

11 An exmple: RE à Syntx Tree à NFA..! ( )* *.!! strt ! 8!

12 Comining the NFAs of Set of Regulr Expressions strt 1 2 { ction 1 } { ction 2 } *+ { ction 3 } strt strt 7 8 strt

13 Simul=ng the Comined NFA Given n input string w, we look for prefix ccepted y the NFA, i.e. tht is the lexeme of token We strt with the set of sttes rechle y strt with - trnsi=ons For ech symol we collect ll sttes to which we cn move from the current sttes Complexity: liner in (length of w) * (numer of sttes), using efficient represent=on of set of sttes Conflicts: severl prefixes of w cn e legl lexemes

14 Simul=ng the Comined NFA Exmple 1 strt ction 1 ction 3 6 ction none 7 8 ction 3 Must find the longest mtch: Continue until no further moves re possile When lst stte is ccepting: execute ction Conflict resolu?on I 14

15 Simul=ng the Comined NFA Exmple 2 strt ction 1 ction 3 6 ction none ction 2 ction 3 Conflict resolu?on II Lex: When two or more ccepting sttes re reched, the first ction given in the specifiction is executed 15

16 Design of Lexicl Anlyzer Genertor: RE to NFA to DFA Specifiction with regulr expressions NFA p 1 { ction 1 } p 2 { ction 2 } p n { ction n } strt s 0 N(p 1 ) N(p 2 ) N(p n ) ction 1 ction 2 ction n Simul=ng the DFA is more efficient, ut The size of the DFA could e DFA Suset construction exponen=l w.r.t. the NFA 16

17 Determinis=c Finite Automt A determinis=c finite utomton is specil cse of n NFA No stte hs n - trnsi=on For ech stte s nd input symol there is t most one edge leled leving s Ech entry in the trnsi=on tle is single stte At most one pth exists to ccept string Simul=on lgorithm is simple Altern=ve defini=on: For ech stte s nd input symol there is exctly one edge leled leving s Esily shown to e equivlent (sink stte ) 17

18 Exmple DFA strt A DFA tht ccepts the sme lnguge of A 2, ( )* strt A

19 Conversion of n NFA into DFA The suset construction lgorithm converts n NFA into DFA using: -closure(s) = {s} {t s t} -closure(t) = s T -closure(s) move(t, ) = {t s t nd s T} The lgorithm produces: Dsttes is the set of sttes of the new DFA consisting of sets of sttes of the NFA Dtrn is the trnsition tle of the new DFA 19

20 - closure nd move Exmples strt closure({0}) = {0,1,3,7} move({0,1,3,7},) = {2,4,7} - closure({2,4,7}) = {2,4,7} move({2,4,7},) = {7} - closure({7}) = {7} move({7},) = {8} - closure({8}) = {8} move({8},) = 0 2 none Also used to simulte NFAs (!) 20

21 Simul=ng n NFA using - closure nd move S := -closure({s 0 }) S prev := := nextchr() while S do S prev := S S := -closure(move(s,)) := nextchr() end do if S prev F then execute ction in S prev return yes else return no 21

22 The Suset Construc=on Algorithm: from NFA to n equivlent DFA Initilly, -closure(s 0 ) is the only stte in Dsttes nd it is unmrked while there is n unmrked stte T in Dsttes do mrk T for ech input symol Σ do U := -closure(move(t,)) if U is not in Dsttes then dd U s n unmrked stte to Dsttes end if Dtrn[T, ] := U end do end do 22

23 Suset Construc=on Exmple 1 strt strt A C B D E Dsttes A = {0,1,2,4,7} B = {1,2,3,4,6,7,8} C = {1,2,4,5,6,7} D = {1,2,4,5,6,7,9} E = {1,2,4,5,6,7,10} 23

24 Suset Construc=on Exmple 2 strt strt 1 3 A C D B E F Dsttes A = {0,1,3,7} B = {2,4,7} C = {8} D = {7} E = {5,8} F = {6,8} 24

25 Minimizing the Numer of Sttes of DFA Given DFA, let us show how to get DFA which ccepts the sme regulr lnguge with miniml numer of sttes strt A C B D E strt AC B D E 25

26 On the Minimiz=on Algorithm Two sttes q nd q' in DFA M = (Q, Σ, δ, q 0, F ) re equivlent (or indis-nguishle) if for ll strings w Σ*, the sttes on which w ends on when red from q nd q' re oth ccept, or oth non- ccept. An utomton is irreducile if it contins no useless (unrechle) sttes, nd no two dis=nct sttes re equivlent The Minimiz?on Algorithm cretes n irreducile utomton ccep=ng the sme lnguge Pr==on- refinement: strts with pr==on of sttes {Accep=ng, Non- ccep=ng} nd refines it =ll done 26

27 Minimiz=on Algorithm (Pr==on Refinement) Code DFA minimize(dfa (Q, Σ, d, q 0, F ) ) remove ny stte q unrechle from q 0 Pr==on P = {F, Q - F } oolen Consistent = flse while ( Consistent == flse ) Consistent = true for(every Set S P, chr Σ, Set T P ) // collect sttes of T tht rech S using Set temp = {q T d(q,) S } if (temp!= Ø && temp!= T ) Consistent = flse P = (P\{T} ) {temp,t- temp} return defineminimizor( (Q, Σ, d, q 0, F ), P ) 27

28 Minimiz=on Algorithm. (Pr==on Refinement) Code DFA defineminimizor (DFA (Q, Σ, δ, q 0, F ), Pr==on P ) Set Q' =P Stte q' 0 = the set in P which contins q 0 F' = { S P S F } for (ech S P, Σ) define δ' (S,) = the set T P which contins the sttes δ(s,) for ech s S return (Q', Σ, δ', q' 0, F' ) 28

29 Minimiz=on Algorithm: Exmple P 1 = {{A, B, C, D}, {E}} ({A,B,C,D}, ) not consistent P 2 = {{A, B, C}, {D}, {E}} ({A,B,C}, ) not consistent P 3 = {{A, C}, {B}, {D}, {E}} Consistent! strt A C B D E strt AC B D E 29

30 Is the constructed utomton miniml? The previous lgorithm gurnteed to produce n irreducile DFA. Why should tht FA e the smllest possile FA for its ccepted lnguge? THM (Myhill- Nerode): The minimiz;on lgorithm produces the smllest possile utomton for its ccepted lnguge. 30

31 Proof of Myhill- Nerode theorem Proof. Show tht ny irreducile utomton is the smllest for its ccepted lnguge L: Two strings u,v Σ* re indis-nguishle if for ll strings w, uw L ó vw L. Thus if u nd v re dis?nguishle, their pths from the strt stte must hve different endpoints. Therefore the numer of sttes in ny DFA for L must e lrger thn or equl to the numer of mutully dis=nguishle strings for L. But in n irreducile DFA every stte gives rise to nother mutully dis=nguishle string! Therefore, ny other DFA for the sme lnguge must hve t lest s mny sttes s the irreducile DFA 31

32 The Lex nd Flex Scnner Genertors Lex nd its newer cousin flex re scnner genertors Scnner genertors system=clly trnslte regulr defini=ons into C source code for efficient scnning Generted code is esy to integrte in C pplic=ons 32

33 Cre=ng Lexicl Anlyzer with Lex nd Flex lex source progrm lex.l lex (or flex) lex.yy.c lex.yy.c C compiler.out input strem.out sequence of tokens 33

34 Lex Specific=on A lex specifiction consists of three prts: regulr definitions, C declrtions in %{ %} %% trnsltion rules %% user-defined uxiliry procedures The trnsltion rules re of the form: p 1 { ction 1 } p 2 { ction 2 } p n { ction n } 34

35 Regulr Expressions in Lex x mtch the chrcter x \. mtch the chrcter. string mtch contents of string of chrcters. mtch ny chrcter except newline ^ mtch eginning of line $ mtch the end of line [xyz] mtch one chrcter x, y, or z (use \ to escpe -) [^xyz]mtch ny chrcter except x, y, nd z [-z] mtch one of to z r* closure (mtch zero or more occurrences) r+ positive closure (mtch one or more occurrences) r? optionl (mtch zero or one occurrence) r 1 r 2 mtch r 1 then r 2 (conctention) r 1 r 2 mtch r 1 or r 2 (union) ( r ) grouping r 1 \r 2 mtch r 1 when followed y r 2 {d}mtch the regulr expression defined y d 35

36 Exmple Lex Specific=on 1 Trnsltion rules %{ #include <stdio.h> %} %% [0-9]+ { printf( %s\n, yytext); }. \n { } %% min() { yylex(); } Contins the mtching lexeme Invokes the lexicl nlyzer lex spec.l gcc lex.yy.c -ll./.out < spec.l 36

37 Exmple Lex Specific=on 2 Trnsltion rules %{ #include <stdio.h> int ch = 0, wd = 0, nl = 0; %} delim [ \t]+ %% \n { ch++; wd++; nl++; } ^{delim} { ch+=yyleng; } {delim} { ch+=yyleng; wd++; }. { ch++; } %% min() { yylex(); printf("%8d%8d%8d\n", nl, wd, ch); } Regulr definition 37

38 Exmple Lex Specific=on 3 Trnsltion rules %{ #include <stdio.h> %} digit [0-9] letter [A-Z-z] Regulr definitions id {letter}({letter} {digit})* %% {digit}+ { printf( numer: %s\n, yytext); } {id} { printf( ident: %s\n, yytext); }. { printf( other: %s\n, yytext); } %% min() { yylex(); } 38

39 Exmple Lex Specific=on 4 %{ /* definitions of mnifest constnts */ #define LT (256) %} delim [ \t\n] ws {delim}+ letter [A-Z-z] digit [0-9] id {letter}({letter} {digit})* numer {digit}+(\.{digit}+)?(e[+\-]?{digit}+)? %% {ws} { } if {return IF;} then {return THEN;} else {return ELSE;} {id} {yylvl = instll_id(); return ID;} {numer} {yylvl = instll_num(); return NUMBER;} < {yylvl = LT; return RELOP;} <= {yylvl = LE; return RELOP;} = {yylvl = EQ; return RELOP;} <> {yylvl = NE; return RELOP;} > {yylvl = GT; return RELOP;} >= {yylvl = GE; return RELOP;} %% int instll_id() { } Token ttriute Return token to prser Instll yytext (of length yyleng) 39 s identifier in symol tle