Lexical and Syntax Analysis

Transcription

1 Lexical and Syntax Analysis (of Programming Languages) Lexical Analysis

2 Lexical and Syntax Analysis (of Programming Languages) Lexical Analysis

3 What is Parsing? Parser String of characters Data structure Easy for humans to write Easy for programs to process A parser also checks that the input string is well-formed, and if not, rejects it.

4 What is Parsing? Parser String of characters Data structure Easy for humans to write Easy for programs to process A parser also checks that the input string is well-formed, and if not, rejects it.

5 PARSING = LEXICAL ANALYSIS + SYNTAX ANALYSIS

6 PARSING = LEXICAL ANALYSIS + SYNTAX ANALYSIS

7 Lexical Analysis (Also known as scanning ) Identifies the lexemes in a sentence. Lexeme: a minimal meaningful unit of a language. Converts each lexeme to a token. Throws away ignorable text such as spaces, new-lines, and comments.

8 Lexical Analysis (Also known as scanning ) Identifies the lexemes in a sentence. Lexeme: a minimal meaningful unit of a language. Converts each lexeme to a token. Throws away ignorable text such as spaces, new-lines, and comments.

9 What is a token? Every token has an identifier, used to denote the kind of lexeme that it represents, e.g. Token identifier PLUS ASSIGN VAR NUM denotes a + operator a := operator a variable a number Some tokens have a component value, conventionally written in parenthesis after the identifier, e.g. VAR(foo), NUM(12).

10 What is a token? Every token has an identifier, used to denote the kind of lexeme that it represents, e.g. Token identifier PLUS ASSIGN VAR NUM denotes a + operator a := operator a variable a number Some tokens have a component value, conventionally written in parenthesis after the identifier, e.g. VAR(foo), NUM(12).

11 Lexical Analysis Stream of characters Stream of tokens Example input: foo := 20 + bar Example output: VAR(foo), ASSIGN, NUM(20), PLUS, VAR(bar)

12 Lexical Analysis Stream of characters Stream of tokens Example input: foo := 20 + bar Example output: VAR(foo), ASSIGN, NUM(20), PLUS, VAR(bar)

13 Lexical Analysis Lexemes are specified by regular expressions. For example: number = digit digit * variable = letter (letter digit) * digit = letter = a... z Example numbers: Example variables: x foo foo2 x1y20

14 Lexical Analysis Lexemes are specified by regular expressions. For example: number = digit digit * variable = letter (letter digit) * digit = letter = a... z Example numbers: Example variables: x foo foo2 x1y20

15 REGULAR EXPRESSIONS What exactly is a regular expression?

16 REGULAR EXPRESSIONS What exactly is a regular expression?

17 Notation Alphabet is the set of all characters that can appear in an input string. If a string s matches a regular expressions r, we write s r. Language L(r) = { s s r }, i.e. the set of all strings matching regular expression r. We write s 1 s 2 to denote the concatenation of strings s 1 and s 2.

18 Notation Alphabet is the set of all characters that can appear in an input string. If a string s matches a regular expressions r, we write s r. Language L(r) = { s s r }, i.e. the set of all strings matching regular expression r. We write s 1 s 2 to denote the concatenation of strings s 1 and s 2.

19 Syntax The syntax of regular expressions is defined by the following grammar, where x ranges over symbols in. r ε r x r r r r r r r r * r ( r )

20 Syntax The syntax of regular expressions is defined by the following grammar, where x ranges over symbols in. r ε r x r r r r r r r r * r ( r )

21 Intuitive definition Regular expression Matching strings are ε The empty string ε x The singleton string x if x r 1 r 2 Any string matching r 1 or r 2. r 1 r 2 r * Any string that can be split into substrings s 1 and s 2 such that s 1 matches r 1 and s 2 matches r 2 The empty string or any string that can be split into substrings s 1...s n such that s i matches r for all i in 1...n

22 Intuitive definition Regular expression Matching strings are ε The empty string ε x The singleton string x if x r 1 r 2 Any string matching r 1 or r 2. r 1 r 2 r * Any string that can be split into substrings s 1 and s 2 such that s 1 matches r 1 and s 2 matches r 2 The empty string or any string that can be split into substrings s 1...s n such that s i matches r for all i in 1...n

23 Formal definition: Base cases L(ε ) = { ε } L(x) = { x } where x

24 Formal definition: Base cases L(ε ) = { ε } L(x) = { x } where x

25 Formal definition: Choice and Sequence L(r 1 r 2 ) = L(r 1 ) L(r 2 ) L(r 1 r 2 ) = { s 1 s 2 s 1 L(r 1 ), s 2 L(r 2 ) }

26 Formal definition: Choice and Sequence L(r 1 r 2 ) = L(r 1 ) L(r 2 ) L(r 1 r 2 ) = { s 1 s 2 s 1 L(r 1 ), s 2 L(r 2 ) }

27 Formal definition: Kleene closure L(r n ) = { ε }, if n = 0 L(r r n-1 ), if n > 0 L(r * ) = { L(r n ) n { 0 } }

28 Formal definition: Kleene closure L(r n ) = { ε }, if n = 0 L(r r n-1 ), if n > 0 L(r * ) = { L(r n ) n { 0 } }

29 Example 1 Suppose = { a, b }. r L(r) a b { a, b } (a b) (a b) { aa, ab, ba, bb } a * { ε, a, aa, aaa,... } (a b) * { ε, ab, abab,... } (a b) * { ε, a, b, ab, ba,... }

30 Example 1 Suppose = { a, b }. r L(r) a b { a, b } (a b) (a b) { aa, ab, ba, bb } a * { ε, a, aa, aaa,... } (a b) * { ε, ab, abab,... } (a b) * { ε, a, b, ab, ba,... }

31 Example 2 Example of a language that cannot be defined by a regular expression: { a n b n n N } The set of strings containing n consecutive a symbols followed by n consecutive b symbols, for all n.

32 Example 2 Example of a language that cannot be defined by a regular expression: { a n b n n N } The set of strings containing n consecutive a symbols followed by n consecutive b symbols, for all n.

33 Exercise 1 Characterise the languages defined by the following regular expressions a (a b) * a a * b a * b a * b a * ((ε a) b * ) *

34 Exercise 1 Characterise the languages defined by the following regular expressions a (a b) * a a * b a * b a * b a * ((ε a) b * ) *

35 Proof rules: Base cases The empty string ε matches ε. ε ε [Empty] If x then x matches x. x x x [Single]

36 Proof rules: Base cases The empty string ε matches ε. ε ε [Empty] If x then x matches x. x x x [Single]

37 Proof rules: Sequence If s 1 matches r 1 and s 2 matches r 2 then s 1 s 2 matches r 1 r 2. s 1 r 1 s 2 r 2 [Seq] s 1 s 2 r 1 r 2

38 Proof rules: Sequence If s 1 matches r 1 and s 2 matches r 2 then s 1 s 2 matches r 1 r 2. s 1 r 1 s 2 r 2 [Seq] s 1 s 2 r 1 r 2

39 Proof rules: Choice If s matches r 1 then s matches r 1 r 2 s r 1 [Or 1 ] s r 1 r 2 If s matches r 2 then s matches r 1 r 2 s r 2 [Or 2 ] s r 1 r 2

40 Proof rules: Choice If s matches r 1 then s matches r 1 r 2 s r 1 [Or 1 ] s r 1 r 2 If s matches r 2 then s matches r 1 r 2 s r 2 [Or 2 ] s r 1 r 2

41 Proof rules: Kleene closure If s matches ε then s matches r *. s ε s r * [Kleene 1 ] If s matches r r * then s matches r *. s r r * [Kleene s r * 2 ]

42 Proof rules: Kleene closure If s matches ε then s matches r *. s ε s r * [Kleene 1 ] If s matches r r * then s matches r *. s r r * [Kleene s r * 2 ]

43 Exercise 2 Proove that the string cab matches the regular expression ((a b) c) *

44 Exercise 2 Proove that the string cab matches the regular expression ((a b) c) *

45 Exercise 2 cab ((a b) c) * { Kleene 2 } cab ((a b) c) ((a b) c) * { Seq } c (a b) c, ab ((a b) c) * { Or 2 } c c, ab ((a b) c) * { Single } ab ((a b) c) * Continued...

46 Exercise 2 cab ((a b) c) * { Kleene 2 } cab ((a b) c) ((a b) c) * { Seq } c (a b) c, ab ((a b) c) * { Or 2 } c c, ab ((a b) c) * { Single } ab ((a b) c) * Continued...

47 Exercise 2 ab ((a b) c) * { Kleene 2 } ab ((a b) c) ((a b) c) * { Seq } ab (a b) c, ε ((a b) c) * { Or 1, Kleene 1 } ab (a b) { Seq } a a, b b { Single, Single } true

48 Exercise 2 ab ((a b) c) * { Kleene 2 } ab ((a b) c) ((a b) c) * { Seq } ab (a b) c, ε ((a b) c) * { Or 1, Kleene 1 } ab (a b) { Seq } a a, b b { Single, Single } true

49 Sound and Complete The proof rules are: Sound: if we can prove s r using the rules then s L(r). Complete: if s L(r) then we can prove s r using the rules.

50 Sound and Complete The proof rules are: Sound: if we can prove s r using the rules then s L(r). Complete: if s L(r) then we can prove s r using the rules.

51 Prolog If we define our proof rules in Prolog then we get a regular expression implementation. [] []. [X] X. S R1!R2 :- S R1. S R1!R2 :- S R2. S R1.R2 :- append(s1, S2, S), S1 R1, S2 R2. [] R*. S R* :- append(s1, S2, S), S1=[X Xs], S1 R, S2 R*. NOTES: Operator! used to represent vertical bar. Read :- as if. Strings represented by lists of symbols. Termination ensured by requiring S1 to be non-empty in final clause.

52 Prolog If we define our proof rules in Prolog then we get a regular expression implementation. [] []. [X] X. S R1!R2 :- S R1. S R1!R2 :- S R2. S R1.R2 :- append(s1, S2, S), S1 R1, S2 R2. [] R*. S R* :- append(s1, S2, S), S1=[X Xs], S1 R, S2 R*. NOTES: Operator! used to represent vertical bar. Read :- as if. Strings represented by lists of symbols. Termination ensured by requiring S1 to be non-empty in final clause.

53 Prolog Sadly the Prolog implementation is not very efficient: when applying the proof rules by hand we used human intuition to know where to split the string; Prolog does not have this intuition; instead, Prolog guesses, trying all possible ways to split a string, and backtracks on failure.

54 Prolog Sadly the Prolog implementation is not very efficient: when applying the proof rules by hand we used human intuition to know where to split the string; Prolog does not have this intuition; instead, Prolog guesses, trying all possible ways to split a string, and backtracks on failure.

55 Common extensions Regular expression is the same as r + r r * r? ε r [c 1 c 2 c 3 -c n ] c 1 c 2 c 3... c n [^c 1 c 2 ] { x x, x {c 1, c 2 } }

56 Common extensions Regular expression is the same as r + r r * r? ε r [c 1 c 2 c 3 -c n ] c 1 c 2 c 3... c n [^c 1 c 2 ] { x x, x {c 1, c 2 } }

57 Escaping What if contains regular expression symbols such as * ( + [? We can escape such symbols by prefixing with a backslash: \ \ * \ \( \[ \? And if we want \ then write \\. Example: \[ * \] * means zero or left brackets followed by zero or more right brackets.

58 Escaping What if contains regular expression symbols such as * ( + [? We can escape such symbols by prefixing with a backslash: \ \ * \ \( \[ \? And if we want \ then write \\. Example: \[ * \] * means zero or left brackets followed by zero or more right brackets.

59 Regular definitions For convenience, we may wish to name a regular expression so that we can refer to it many times: name = r We write name but sometimes the notation {name} is used (e.g. in Flex). Example: number = digit digit * digit =

60 Regular definitions For convenience, we may wish to name a regular expression so that we can refer to it many times: name = r We write name but sometimes the notation {name} is used (e.g. in Flex). Example: number = digit digit * digit =

61 Implementing regular expressions How do we convert a regular expression r into an efficient program that prints YES when applied to any string in L(r) and NO in all other cases? Two options: By hand (LSA Lab 1) Automatically (Chapters 3 & 4 of lecture notes, and LSA Lab 2)

62 Implementing regular expressions How do we convert a regular expression r into an efficient program that prints YES when applied to any string in L(r) and NO in all other cases? Two options: By hand (LSA Lab 1) Automatically (Chapters 3 & 4 of lecture notes, and LSA Lab 2)

63 Outline Automatic conversion of regular expressions to efficient stringmatching functions: Step 1: RE NFA Step 2: NFA DFA Step 3: DFA C Function Acronym RE NFA DFA Meaning Regular Expression Non-deterministic Finite Automaton Deterministic Finite Automaton

64 Outline Automatic conversion of regular expressions to efficient stringmatching functions: Step 1: RE NFA Step 2: NFA DFA Step 3: DFA C Function Acronym RE NFA DFA Meaning Regular Expression Non-deterministic Finite Automaton Deterministic Finite Automaton

65 STEP 1: RE NFA Thompson s construction

66 STEP 1: RE NFA Thompson s construction

67 What is an NFA? A directed graph with nodes denoting states s A state s s The start state s s An accepting state s s The start state s that is also an accepting state and edges labelled with a symbol x {ε} denoting transitions x s 1 s 2

68 What is an NFA? A directed graph with nodes denoting states s A state s s The start state s s An accepting state s s The start state s that is also an accepting state and edges labelled with a symbol x {ε} denoting transitions x s 1 s 2

69 Meaning of an NFA A string x 1 x 2...x n is accepted by an NFA if there is a path labelled x 1,x 2,...,x n (including any number of ε transitions) from the start state to an accepting state.

70 Meaning of an NFA A string x 1 x 2...x n is accepted by an NFA if there is a path labelled x 1,x 2,...,x n (including any number of ε transitions) from the start state to an accepting state.

71 Example of an NFA The following NFA accepts exactly the strings that match the regular expression a a * b b *. ε 1 a 2 0 a ε 3 b 4 b

72 Example of an NFA The following NFA accepts exactly the strings that match the regular expression a a * b b *. ε 1 a 2 0 a ε 3 b 4 b

73 Thompson s construction: Notation Let N(r) be the NFA accepting exactly the set of strings in L(r). We abstractly represent an NFA N(r) with start state s 0 and final state s a by the diagram: s 0 N(r) s a

74 Thompson s construction: Notation Let N(r) be the NFA accepting exactly the set of strings in L(r). We abstractly represent an NFA N(r) with start state s 0 and final state s a by the diagram: s 0 N(r) s a

75 Thompson s construction: Base cases = s 0 N(ε) s a s 0 ε s a = s 0 N(x) s a s 0 x s a where x

76 Thompson s construction: Base cases = s 0 N(ε) s a s 0 ε s a = s 0 N(x) s a s 0 x s a where x

77 Thompson s construction: Choice s 0 N(r t) s a = s 0 ε ε N(r) N(t) ε ε s a

78 Thompson s construction: Choice s 0 N(r t) s a = s 0 ε ε N(r) N(t) ε ε s a

79 Thompson s construction: Sequence s 0 N(r t) s a = s 0 N(r) N(t) s a

80 Thompson s construction: Sequence s 0 N(r t) s a = s 0 N(r) N(t) s a

81 Thompson s construction: Kleene closure s 0 N(r* ) s a = ε s 0 ε N(r) ε s a ε

82 Thompson s construction: Kleene closure s 0 N(r* ) s a = ε s 0 ε N(r) ε s a ε

83 Exercise 3 Apply Thompson s construction to the following regular expression. ((a b) c) *

84 Exercise 3 Apply Thompson s construction to the following regular expression. ((a b) c) *

85 Problem with NFAs It is not straightforward to turn an NFA into a deterministic program because: There may be many possible next-states for a given input. Which one do we choose? Try them all? Idea: convert an NFA into a DFA: a DFA can be easily converted into an efficient executable program.

86 Problem with NFAs It is not straightforward to turn an NFA into a deterministic program because: There may be many possible next-states for a given input. Which one do we choose? Try them all? Idea: convert an NFA into a DFA: a DFA can be easily converted into an efficient executable program.

87 STEP 2: NFA DFA The subset construction.

88 STEP 2: NFA DFA The subset construction.

89 What is a DFA? A deterministic finite automaton (DFA) is an NFA in which there are no ε transitions, and for each state s and input symbol a there is at most one transition out of s labelled a.

90 What is a DFA? A deterministic finite automaton (DFA) is an NFA in which there are no ε transitions, and for each state s and input symbol a there is at most one transition out of s labelled a.

91 Example of a DFA The following DFA accepts exactly the strings that match the regular expression a a * b b *. 0 a 1 a b 2 b

92 Example of a DFA The following DFA accepts exactly the strings that match the regular expression a a * b b *. 0 a 1 a b 2 b

93 NFA DFA: key observation After consuming an input string, an NFA can be in be in one of a set of states. Example 3: b 0 1 a a a Input States aa 0, 1, 2 aba aab 2 b aaba 3 ε

94 NFA DFA: key observation After consuming an input string, an NFA can be in be in one of a set of states. Example 3: b 0 1 a a a Input States aa 0, 1, 2 aba aab 2 b aaba 3 ε

95 NFA DFA: key idea Idea: construct a DFA in which each state corresponds to a set of NFA states. After consuming a 1 a n the DFA is in a state which corresponds to the set of states that the NFA can reach on input a 1 a n.

96 NFA DFA: key idea Idea: construct a DFA in which each state corresponds to a set of NFA states. After consuming a 1 a n the DFA is in a state which corresponds to the set of states that the NFA can reach on input a 1 a n.

97 Example 3, revisited Create a DFA state corresponding to each set of NFA states. Input NFA States DFA State aa 0, 1, 2 A aba 0,1 B aab 0,3 C aaba 0,1 B ε 0 D Question: which states would be initial and final DFA states?

98 Example 3, revisited Create a DFA state corresponding to each set of NFA states. Input NFA States DFA State aa 0, 1, 2 A aba 0,1 B aab 0,3 C aaba 0,1 B ε 0 D Question: which states would be initial and final DFA states?

99 Notation Operation ε-closure(s) Description Set of NFA states reachable from NFA state s on zero or more ε-transitions. ε-closure(t) move(t, a) ε-closure(s) s T Set of NFA states to which there is a transition on symbol a from some state s in T.

100 Notation Operation ε-closure(s) Description Set of NFA states reachable from NFA state s on zero or more ε-transitions. ε-closure(t) move(t, a) ε-closure(s) s T Set of NFA states to which there is a transition on symbol a from some state s in T.

101 Exercise 4 Consider the following NFA. a a c ε ε 3 ε a Compute: ε-closure(0) ε-closure({1, 2}) move({0,3}, a) ε-closure(move({0,3}, a))

102 Exercise 4 Consider the following NFA. a a c ε ε 3 ε a Compute: ε-closure(0) ε-closure({1, 2}) move({0,3}, a) ε-closure(move({0,3}, a))

103 Subset construction: input and output Input: an NFA N. Output: a DFA D accepting the same language as N. Specifically, the set of states of D, termed D states, and its transition function D tran that maps any state-symbol pair to a next state.

104 Subset construction: input and output Input: an NFA N. Output: a DFA D accepting the same language as N. Specifically, the set of states of D, termed D states, and its transition function D tran that maps any state-symbol pair to a next state.

105 Subset construction: input and output Each state in D is denoted by a subset of N s states. To ensure termination, every state is either marked or unmarked. Initially, D states contains a single unmarked start state ε-closure(s 0 ) where s 0 is the start state of N. The accepting states of D are the states that contain at least one accepting state of N.

106 Subset construction: input and output Each state in D is denoted by a subset of N s states. To ensure termination, every state is either marked or unmarked. Initially, D states contains a single unmarked start state ε-closure(s 0 ) where s 0 is the start state of N. The accepting states of D are the states that contain at least one accepting state of N.

107 Subset construction: algorithm while (there is an unmarked state T in D states ) { mark T; for (each input symbol a) { U = ε-closure(move(t, a)); D tran [T, a] = U if (U is not in D states ) add U as unmarked state to D states ; } }

108 Subset construction: algorithm while (there is an unmarked state T in D states ) { mark T; for (each input symbol a) { U = ε-closure(move(t, a)); D tran [T, a] = U if (U is not in D states ) add U as unmarked state to D states ; } }

109 Exercise 5 Convert the following NFA into a DFA by applying the subset construction algorithm. ε 1 a b ε 7 ε ε 8 ε 10 ε 5 6 c ε ε

110 Exercise 5 Convert the following NFA into a DFA by applying the subset construction algorithm. ε 1 a b ε 7 ε ε 8 ε 10 ε 5 6 c ε ε

111 Exercise 6 It is not obvious how to simulate an NFA in linear time with respect to the length of the input string. But it may be converted to a DFA that can be simulated easily in linear time. What s the catch? Can you think of any problems with the DFA produced by subset construction?

112 Exercise 6 It is not obvious how to simulate an NFA in linear time with respect to the length of the input string. But it may be converted to a DFA that can be simulated easily in linear time. What s the catch? Can you think of any problems with the DFA produced by subset construction?

113 Caveats Number of DFA states could be exponential in number of NFA states! DFA produced is not minimal in number of states. (Can apply a minimisation algorithm.) Often no problem in practice.

114 Caveats Number of DFA states could be exponential in number of NFA states! DFA produced is not minimal in number of states. (Can apply a minimisation algorithm.) Often no problem in practice.

115 Homework Exercise Convert the following NFA into a DFA by applying the subset construction algorithm. b a a b a

116 Homework Exercise Convert the following NFA into a DFA by applying the subset construction algorithm. b a a b a

117 STEP 3: DFA C CODE

118 STEP 3: DFA C CODE

119 Exercise 7 Implement the DFA a B b a A a D c C c c as a C function int match(char *next) { } returning 1 if the string pointed to by next is accepted by the DFA and 0 otherwise.

120 Exercise 7 Implement the DFA a B b a A a D c C c c as a C function int match(char *next) { } returning 1 if the string pointed to by next is accepted by the DFA and 0 otherwise.

121 int match(char* next) { goto A; /* start state */ A: if (*next == '\0') return 1; if (*next == 'a') { next++; goto B; } if (*next == 'c') { next++; goto C; } return 0; B: if (*next == '\0') return 0; if (*next == 'b') { next++; goto D; } return 0; C: if (*next == '\0') return 1; if (*next == 'a') { next++; goto B; } if (*next == 'c') { next++; goto D; } return 0; } D: if (*next == '\0') return 1; if (*next == 'a') { next++; goto B; } if (*next == 'c') { next++; goto D; } return 0;

122 int match(char* next) { goto A; /* start state */ A: if (*next == '\0') return 1; if (*next == 'a') { next++; goto B; } if (*next == 'c') { next++; goto C; } return 0; B: if (*next == '\0') return 0; if (*next == 'b') { next++; goto D; } return 0; C: if (*next == '\0') return 1; if (*next == 'a') { next++; goto B; } if (*next == 'c') { next++; goto D; } return 0; } D: if (*next == '\0') return 1; if (*next == 'a') { next++; goto B; } if (*next == 'c') { next++; goto D; } return 0;

123 SUMMARY

124 SUMMARY

125 Summary In lexical analysis, the lexemes of the language are identified and converted into tokens. Lexemes are typically specified by regular expressions. Matching of regular expressions formalised by proof rules. Defining proof rules in Prolog gives a simple but inefficient implementation.

126 Summary In lexical analysis, the lexemes of the language are identified and converted into tokens. Lexemes are typically specified by regular expressions. Matching of regular expressions formalised by proof rules. Defining proof rules in Prolog gives a simple but inefficient implementation.

127 Summary Automatically converting regular expressions into efficient C code involves three main steps: 1. RE NFA (Thompson s Construction) 2. NFA DFA (Subset Construction) 3. DFA C Function

128 Summary Automatically converting regular expressions into efficient C code involves three main steps: 1. RE NFA (Thompson s Construction) 2. NFA DFA (Subset Construction) 3. DFA C Function

129 Theory & Practice In the next lecture, we will learn how to use a tool called Flex that puts the regular expression theory into practice.