Grammars: Notation to succinctly represent the structure of a language. Example:

Transcription

1 Introduction Syntx Anlysis Structure of Lnguge Grmmrs: Nottion to succinctly represent the structure of lnguge. Exmple: Stmt if Expr then Stmt else Stmt Stmt while Expr do Stmt Stmt do Stmt until Expr. Expr Expr + Expr. Compilers Lexicl Anlysis CSE 304/504 1 / 54 Introduction Syntx Anlysis Grmmrs Stmt if Expr then Stmt else Stmt Terminl symols: if, then, else Terminl symols represent group of chrcters in input lnguge: Tokens. Anlogous to words. Nonterminl symols: Stmt, Expr Nonterminl symols represent sequence of terminl symols. Anlogous to sentences. Compilers Lexicl Anlysis CSE 304/504 2 / 54

2 Introduction Syntx Anlysis Phses of Syntx Anlysis 1 Identify the words: Lexicl Anlysis. Converts strem of chrcters (input progrm) into strem of tokens. Also clled Scnning or Tokenizing. 2 Identify the sentences: Prsing. Derive the structure of sentences: construct prse trees from strem of tokens. Compilers Lexicl Anlysis CSE 304/504 3 / 54 Introduction Lexicl Anlysis Lexicl Anlysis Convert strem of chrcters into strem of tokens. Simplicity: Conventions out words re often different from conventions out sentences. Efficiency: Word identifiction prolem hs much more efficient solution thn sentence identifiction prolem. Portility: Chrcter set, specil chrcters, device fetures. Compilers Lexicl Anlysis CSE 304/504 4 / 54

3 Introduction Lexicl Anlysis Terminology Token: Nme given to fmily of words. e.g., integer constnt Lexeme: Actul sequence of chrcters representing word. e.g., Pttern: Nottion used to identify the set of lexemes represented y token. e.g., [0 9]+ Compilers Lexicl Anlysis CSE 304/504 5 / 54 Introduction Lexicl Anlysis Terminology A few more exmples: Token Smple Lexemes Pttern while while while integer constnt 32894, -1093, 0 [0-9]+ identifier uffer size [-za-z]+ Compilers Lexicl Anlysis CSE 304/504 6 / 54

4 Introduction Lexicl Anlysis Ptterns How do we compctly represent the set of ll lexemes corresponding to token? For instnce: The token integer constnt represents the set of ll integers: tht is, ll sequences of digits (0 9), preceded y n optionl sign (+ or ). Oviously, we cnnot simply enumerte ll lexemes. Use Regulr Expressions. Compilers Lexicl Anlysis CSE 304/504 7 / 54 Regulr Expressions Expressions & Mening Regulr Expressions Nottion to represent (potentilly) infinite sets of strings over lphet Σ. : stnds for the set {} tht contins single string. : stnds for the set {, } tht contins two strings nd. Anlogous to Union. : stnds for the set {} tht contins single string. Anlogous to Product. ( )( ): stnds for the set {,,, }. : stnds for the set {ɛ,,,,...} tht contins ll strings of zero or more s. Anlogous to closure of the product opertion. ɛ stnds for the empty string. Compilers Lexicl Anlysis CSE 304/504 8 / 54

5 Regulr Expressions Expressions & Mening Regulr Expressions Exmples of Regulr Expressions over {, }: ( ) : Set of strings with zero or more s nd zero or more s: {ɛ,,,,,,,,,...} ( ): Set of strings with zero or more s nd zero or more s such tht ll s occur efore ny : {ɛ,,,,,,,,,...} ( ) : Set of strings with zero or more s nd zero or more s: {ɛ,,,,,,,,,...} Compilers Lexicl Anlysis CSE 304/504 9 / 54 Regulr Expressions Expressions & Mening Lnguge of Regulr Expressions Let R e the set of ll regulr expressions over Σ. Then, Empty String: ɛ R Unit Strings: α Σ α R Conctention: r 1, r 2 R r 1 r 2 R Alterntive: r 1, r 2 R (r 1 r 2 ) R Kleene Closure: r R r R Compilers Lexicl Anlysis CSE 304/ / 54

6 Regulr Expressions Expressions & Mening Regulr Expressions Exmple: ( ) L 0 = {ɛ} L 1 = L 0 {, } = {ɛ} {, } = {, } L 2 = L 1 {, } = {, } {, } = {,,, } L 3 = L 2 {, }. L = L i = {ɛ,,,,,,,...} i=0 Compilers Lexicl Anlysis CSE 304/ / 54 Regulr Expressions Expressions & Mening Semntics of Regulr Expressions Semntic Function L : Mps regulr expressions to sets of strings. L(ɛ) = {ɛ} L(α) = {α} (α Σ) L(r 1 r 2 ) = L(r 1 ) L(r 2 ) L(r 1 r 2 ) = L(r 1 ) L(r 2 ) L(r ) = {ɛ} (L(r) L(r )) Compilers Lexicl Anlysis CSE 304/ / 54

7 Regulr Expressions Expressions & Mening Computing the Semntics L() = {} L( ) = L() L() = {} {} = {, } L() = L() L() = {} {} = {} L(( )( )) = L( ) L( ) = {, } {, } = {,,, } Compilers Lexicl Anlysis CSE 304/ / 54 Regulr Expressions Expressions & Mening Computing the Semntics of Closure Exmple: L(( ) ) = {ɛ} (L( ) L(( ) )) L 0 = {ɛ} Bse cse L 1 = {ɛ} ({, } L 0 ) = {ɛ} ({, } {ɛ}) = {ɛ,, } L 2 = {ɛ} ({, } L 1 ).. = {ɛ} ({, } {ɛ,, }) = {ɛ,,,,,, } L(( ) ) = L = {ɛ,,,,,,,...} Compilers Lexicl Anlysis CSE 304/ / 54

8 Regulr Expressions Expressions & Mening Another Exmple L(( ) ) : L( ) = {ɛ,,,...} L( ) = {ɛ,,,...} L( ) = {ɛ,,,,,, L(( ) ) = {ɛ},,,,...} {ɛ,,,,,,,,,,...} {ɛ,,,,,,,.,,,,,,,,...} = {ɛ,,,,,,,...} Compilers Lexicl Anlysis CSE 304/ / 54 Regulr Expressions Regulr Definitions Regulr Definitions Assign nmes to regulr expressions. For exmple, Shorthnds: Exmple: digit nturl digit digit + : Set of strings with one or more occurrences of.? : Set of strings with zero or one occurrences of. integer (+ )? digit + Compilers Lexicl Anlysis CSE 304/ / 54

9 Regulr Expressions Regulr Definitions Regulr Definitions: Exmples flot integer. frction integer (+ )? no leding zero no leding zero (nonzero digit digit ) 0 frction no triling zero exponent? no triling zero (digit nonzero digit) 0 exponent (E e) integer digit nonzero digit Compilers Lexicl Anlysis CSE 304/ / 54 Regulr Expressions Regulr Definitions Regulr Definitions nd Lexicl Anlysis Regulr Expressions nd Definitions specify sets of strings over n input lphet. They cn hence e used to specify the set of lexemes ssocited with token. Tht is, regulr expressions nd definitions cn e used s the pttern lnguge How do we decide whether n input string elongs to the set of strings specified y regulr expression? Compilers Lexicl Anlysis CSE 304/ / 54

10 Regulr Expressions Regulr Definitions Using Regulr Definitions for Lexicl Anlysis Q: Is in L((( ) )? A: Hm. Well. Let s see. L(( ) ) = {ɛ} {ɛ,,,,,,,,,,...} {ɛ,,,,,,,.. =???,,,,,,,,...} Compilers Lexicl Anlysis CSE 304/ / 54 Regulr Expressions Regulr Definitions Recognizers Construct utomt tht recognize strings elonging to lnguge. Finite Stte Automt Regulr Lnguges Finite Stte cnnot mintin ritrry counts. Push Down Automt Context-free Lnguges Stck is used to mintin counter, ut only one counter cn go ritrrily high. Compilers Lexicl Anlysis CSE 304/ / 54

11 Finite Stte Automt Recognizers Recognizing Finite Sets of Strings Identifying words from smll, finite, fixed voculry is strightforwrd. For instnce, consider stck mchine with push, pop, nd dd opertions with two constnts: 0 nd 1. We cn use the utomton: p 0 1 u o d integer_constnt s p d h pop dd push Compilers Lexicl Anlysis CSE 304/ / 54 Finite Stte Automt Recognizers Finite Stte Automt Represented y leled directed grph. A finite set of sttes (vertices). Trnsitions etween sttes (edges). Lels on trnsitions re drwn from Σ {ɛ}. One distinguished strt stte. One or more distinguished finl sttes. Compilers Lexicl Anlysis CSE 304/ / 54

12 Finite Stte Automt Recognizers Finite Stte Automt: An Exmple Consider the Regulr Expression ( ) ( ). L(( ) ( )) = {,,,,,,,,,,,...}. The following utomton determines whether n input string elongs to L(( ) ( ): Compilers Lexicl Anlysis CSE 304/ / 54 Finite Stte Automt Recognizers Acceptnce Criterion A finite stte utomton (NFA or DFA) ccepts n input string x... if eginning from the strt stte... we cn trce some pth through the utomton... such tht the sequence of edge lels spells x... nd end in finl stte. Compilers Lexicl Anlysis CSE 304/ / 54

13 Finite Stte Automt Recognizers Recognition with n NFA Is L(( ) ( ))? Input: Accept? Pth 1: no Pth 2: yes Pth 3: no YES Compilers Lexicl Anlysis CSE 304/ / 54 Finite Stte Automt Recognizers Regulr Expressions to NFA Thompson s Construction: For every regulr expression r, derive n NFA N(r) with unique strt nd finl sttes. ɛ α Σ α (r 1 r 2 ) N(r ) 1 N(r ) 2 Compilers Lexicl Anlysis CSE 304/ / 54

14 Finite Stte Automt Recognizers Regulr Expressions to NFA (contd.) r 1 r 2 N(r ) N(r ) 1 2 r N(r) Compilers Lexicl Anlysis CSE 304/ / 54 Finite Stte Automt Recognizers Exmple ( ) ( ): Compilers Lexicl Anlysis CSE 304/ / 54

15 Finite Stte Automt Recognizers Recognition with n NFA Is L(( ) ( ))? Input: Accept? Pth 1: Pth 2: Accept Pth 3: All Pths {1} {1, 2} {1, 3} {1, 2} {1, 3} Accept Compilers Lexicl Anlysis CSE 304/ / 54 Finite Stte Automt Recognizers Recognition with n NFA (contd.) Is L(( ) ( ))? Input: Accept? Pth 1: Pth 2: Pth 3: Accept Pth 4: Pth 5: All Pths {1} {1, 2} {1, 2, 3} {1, 2, 3} {1, 2, 3} Accept Compilers Lexicl Anlysis CSE 304/ / 54

16 Finite Stte Automt Recognizers Recognition with n NFA (contd.) Is L(( ) ( ))? Input: Accept? Pth 1: Pth 2: Pth 3: All Pths {1} {1, 2} {1, 2, 3} {1, 3} {1} REJECT Compilers Lexicl Anlysis CSE 304/ / 54 Finite Stte Automt DFA & NFA Determinism ( ) ( ): Nondeterministic: (NFA) Deterministic: (DFA) Compilers Lexicl Anlysis CSE 304/ / 54

17 Finite Stte Automt DFA & NFA Recognition with DFA Is L(( ) ( ))? Input: Accept? Pth: YES Compilers Lexicl Anlysis CSE 304/ / 54 Finite Stte Automt DFA & NFA NFA vs. DFA For every NFA, there is DFA tht ccepts the sme set of strings. NFA my hve trnsitions leled y ɛ. (Spontneous trnsitions) All trnsition lels in DFA elong to Σ. For some string x, there my e mny ccepting pths in n NFA. For ll strings x, there is one unique ccepting pth in DFA. Usully, n input string cn e recognized fster with DFA. NFAs re typiclly smller thn the corresponding DFAs. Compilers Lexicl Anlysis CSE 304/ / 54

18 Finite Stte Automt DFA & NFA NFA vs. DFA (contd.) R = Size of Regulr Expression N = Length of Input String Size of Automton Recognition time per input string NFA DFA O(R) O(2 R ) O(N R) O(N) Compilers Lexicl Anlysis CSE 304/ / 54 Finite Stte Automt DFA & NFA Converting NFA to DFA Suset construction Given set S of NFA sttes, compute S ɛ = ɛ-closure(s): S ɛ is the set of ll NFA sttes rechle y zero or more ɛ-trnsitions from S. compute S α = goto(s, α): S is the set of ll NFA sttes rechle from S y tking trnsition leled α. S α = ɛ-closure(s ). Compilers Lexicl Anlysis CSE 304/ / 54

19 Finite Stte Automt DFA & NFA Converting NFA to DFA (contd). Ech stte in DFA corresponds to set of sttes in NFA. Strt stte of DFA = ɛ-closure(strt stte of NFA). From stte s in DFA tht corresponds to set of sttes S in NFA: let S = goto(s, α) such tht S is non-empty. dd n α-trnsition to stte s tht corresponds S in NFA, S contins finl NFA stte, nd s is the corresponding DFA stte s is finl stte of DFA Compilers Lexicl Anlysis CSE 304/ / 54 Finite Stte Automt DFA & NFA NFA DFA: An Exmple ɛ-closure({1}) = {1} goto({1}, ) = {1, 2} goto({1}, ) = {1} goto({1, 2}, ) = {1, 2, 3} goto({1, 2}, ) = {1, 3} goto({1, 2, 3}, ) = {1, 2, 3} goto({1, 2, 3}, ) = {1, 3} goto({1, 3}, ) = {1, 2} goto({1, 3}, ) = {1} Compilers Lexicl Anlysis CSE 304/ / 54

20 Finite Stte Automt DFA & NFA NFA DFA: An Exmple (contd.) ɛ-closure({1}) = {1} goto({1}, ) = {1, 2} goto({1}, ) = {1} goto({1, 2}, ) = {1, 2, 3} goto({1, 2}, ) = {1, 3} goto({1, 2, 3}, ) = {1, 2, 3} goto({1, 2, 3}, ) = {1, 3} goto({1, 3}, ) = {1, 2} goto({1, 3}, ) = {1} {1} {1,2} {1,2,3} {1,3} Compilers Lexicl Anlysis CSE 304/ / 54 Generting Lexicl Anlyzers Construction of Lexicl Anlyzer Regulr Expressions nd Definitions re used to specify the set of strings (lexemes) corresponding to token. An utomton (DFA/NFA) is uilt from the ove specifictions. Ech finl stte is ssocited with n ction: emit the corresponding token. Compilers Lexicl Anlysis CSE 304/ / 54

21 Generting Lexicl Anlyzers Specifying Lexicl Anlysis Consider recognizer for integers (sequence of digits) nd flots (sequence of digits seprted y deciml point). [0-9]+ { emit(integer_constant); } [0-9]+"."[0-9]+ { emit(float_constant); } INTEGER_CONSTANT "." 0 9 FLOAT_CONSTANT Compilers Lexicl Anlysis CSE 304/ / 54 Generting Lexicl Anlyzers Lex Tool for uilding lexicl nlyzers. Input: lexicl specifictions (.l file) Output: C function (yylex) tht returns token on ech invoction. Exmple: %% [0-9]+ { return(integer_constant); } [0-9]+"."[0-9]+ { return(float_constant); } Tokens re simply integers (#define s). Compilers Lexicl Anlysis CSE 304/ / 54

22 Lex Specifictions Generting Lexicl Anlyzers %{ C heder sttements for inclusion %} Regulr Definitions e.g.: digit [0-9] %% Token Specifictions e.g.: {digit}+ { return(integer_constant); } %% Support functions in C Compilers Lexicl Anlysis CSE 304/ / 54 Generting Lexicl Anlyzers Regulr Expressions in Lex Adds syntctic sugr to regulr expressions: Rnge: [0-7]: Integers from 0 through 7 (inclusive) [-nx-za-q]: Letters thru n, x thru z nd A thru Q. Exception: [^/]: Any chrcter other thn /. Definition: {digit}: Use the previously specified regulr definition digit. Specil chrcters: fetures. e.g.: * ^ Connectives of regulr expression, convenience Compilers Lexicl Anlysis CSE 304/ / 54

23 Generting Lexicl Anlyzers Specil Chrcters in Lex * +? ( ) Sme s in regulr expressions [ ] Enclose rnges nd exceptions { } Enclose nmes of regulr definitions ^ Used to negte specified rnge (in Exception). Mtch ny single chrcter except newline \ Escpe the next chrcter \n, \t Newline nd T For literl mtching, enclose specil chrcters in doule quotes (") e.g.: "*" Or use \ to escpe. e.g.: \* Compilers Lexicl Anlysis CSE 304/ / 54 Exmples Generting Lexicl Anlyzers for Sequence of f, o, r " " C-style OR opertor (two vert. rs).* Sequence of non-newline chrcters [^*/]+ Sequence of chrcters except * nd / \"[^"]*\" Sequence of non-quote chrcters eginning nd ending with quote ({letter} " ")({letter} {digit} " ")* C-style identifiers Compilers Lexicl Anlysis CSE 304/ / 54

24 A Complete Exmple Generting Lexicl Anlyzers %{ #include <stdio.h> #include "tokens.h" %} digit [0-9] hexdigit [0-9-f] %% "+" { return(plus); } "-" { return(minus); } {digit}+ { return(integer_constant); } {digit}+"."{digit}+ { return(float_constant); }. { return(syntax_error); } %% Compilers Lexicl Anlysis CSE 304/ / 54 Actions Generting Lexicl Anlyzers Actions re ttched to finl sttes. Distinguish the different finl sttes. Used to return tokens. Cn e used to set ttriute vlues. Frgment of C code (locks enclosed y { nd } ). Compilers Lexicl Anlysis CSE 304/ / 54

25 Generting Lexicl Anlyzers Attriutes Additionl informtion out token s lexeme. Stored in vrile yylvl Type of ttriutes (usully union) specified y YYSTYPE Additionl vriles: yytext: Lexeme (Actul text string) yyleng: length of string in yytext yylineno: Current line numer (numer of \n seen thus fr) enled y %option yylineno Compilers Lexicl Anlysis CSE 304/ / 54 Priority of mtching Generting Lexicl Anlyzers Wht if n input string mtches more thn one pttern? "if" { return(token_if); } {letter}+ { return(token_id); } "while" { return(token_while); } A pttern tht mtches the longest string is chosen. Exmple: if1 is mtched with n identifier, not the keyword if. Of ptterns tht mtch strings of sme length, the first (from the top of file) is chosen. Exmple: while is mtched s n identifier, not the keyword while. Compilers Lexicl Anlysis CSE 304/ / 54

26 Generting Lexicl Anlyzers Constructing Scnners using (f)lex Scnner specifictions: specifictions.l (f)lex specifictions.l lex.yy.c Generted scnner in lex.yy.c (g)cc lex.yy.c executle yywrp(): hook for signlling end of file. Use -lfl (flex) or -ll (lex) flgs t link time to include defult function yywrp() tht lwys returns 1. Compilers Lexicl Anlysis CSE 304/ / 54 Generting Lexicl Anlyzers Implementing Scnner trnsition : stte Σ stte lgorithm scnner() { current stte = strt stte; while (1) { c = getc(); /* on end of file,... */ if defined(trnsition(current stte, c)) current stte = trnsition(current stte, c); else return s; } } Compilers Lexicl Anlysis CSE 304/ / 54

27 Generting Lexicl Anlyzers Implementing Scnner (contd.) Implementing the trnsition function: Simplest: 2-D rry. Spce inefficient. Trditionlly compressed using row/colum equivlence. (defult on (f)lex) Good spce-time trdeoff. Further tle compression using vrious techniques: Exmple: RDM (Row Displcement Method): Store rows in overlpping mnner using 2 1-D rrys. Smller tles, ut longer ccess times. Compilers Lexicl Anlysis CSE 304/ / 54 Generting Lexicl Anlyzers Lexicl Anlysis: Summry Convert strem of chrcters into strem of tokens. Mke rest of compiler independent of chrcter set Strip off comments Recognize line numers Ignore white spce chrcters Process mcros (definitions nd uses) Interfce with symol tle (lso clled nme tle ). Compilers Lexicl Anlysis CSE 304/ / 54