Lexical analysis FORMAL LANGUAGES AND COMPILERS. Floriano Scioscia. Formal Languages and Compilers A.Y. 2015/2016

Transcription

1 Master s Degree Course in Computer Engineering Formal Languages FORMAL LANGUAGES AND COMPILERS Lexical analysis Floriano Scioscia 1

2 Introductive terminological distinction Lexical string or lexeme = meaningful character sequence in the source program matching with the pattern of a token. Token or symbol = abstraction of a class of lexical strings; it is the smallest element (not further decomposable) of a language; it describes a set of character strings having the same meaning: Example: id token associated with the class {instances of identifier lexemes} Pattern = description of the form lexemes of a token can take. Patterns are described through regular expressions. Symbol Instances Pattern while while while begin begin begin relop <, <=, >, >=,!=, =, == {<, <=, >, >=,!=, =, ==} id starting, time, m24, X2 letter then letters and/or digits num 3, 25, 3.5, 4,37E12 number constant strconst "Hello world!" character sequence between " A token represents a set of strings described by a pattern. id represents a set of strings starting with a letter and then including letters and digits. The actual string is called lexeme. Lexical analysis - Floriano Scioscia 2

3 Purpose of the lexical analyzer sequence of characters LEXICAL ANALYZER sequence of tokens alpha = (beta * 3) id, alpha assign, left, id, beta times, num, 3 right, Purpose of a lexical analyzer (a.k.a. scanner or lexer): given a sequence of characters of an alphabet forming a statement, check whether the sequence can be decomposed into a sequence of lexemes (symbols admitted by the language) If it is so, return the sequence of tokens for each lexeme in the source program Otherwise, return a lexical error For each lexeme, the scanner outputs one token. Lexical analysis - Floriano Scioscia 3

4 Role of the lexical analyzer (1/2) Lexicon describes the words a.k.a. lexical elements composing sentences. In artificial languages, lexical elements can be assigned to: Keywords: special fixed words characterizing various types of sentences or structures. E.g.: if, begin, subprogram, write. Keywords are unflectable. Delimiters, operators and composite characters: like above, they are fixed words, but composed also by non-alphabetic characters. E.g. comments in Ada language are preceded by two dashes -- ; the greater or equal operator is written as >= and in other languages as.gte. Open lexical classes: they comprise an infinite number of lexical elements, which must have the structure of a regular language. Typical examples include: names or identifiers of variables, subprograms or other language entities; e.g. in many languages identifiers are defined by the regular expression: identifier = letter (letter digit)* constants such as integer or real numbers and alphanumeric strings. Lexical analysis - Floriano Scioscia 4

5 Role of the lexical analyzer (2/2) There is an important difference between keywords and open lexical classes: keywords do not carry any information other than their name; lexical classes are usually strings belonging to a formal language of the regular type. Both identifiers and constants denote entities endowed with a value and/or other properties, which will be called semantic attributes hereafter. Lexical analysis - Floriano Scioscia 5

6 Tokens, patterns, lexicon and attributes (1/2) A token is described through a triplet: 1. Name (id, keyword, etc.) 2. Attribute (either a value or a pointer to the symbol table) 3. Position (optional) Token Lexemes Pattern Id count a123 String starting with a letter and containing letters and digits num 123 Any numeric constant e10 if if The keyword if - the string a123 is the value of a token of type identifier - the string 123 is the value of a token of type integer Lexical analysis - Floriano Scioscia 6

7 Tokens, patterns, lexicon and attributes (1/2) Sometimes the value is ignored: for example, a keyword or an arithmetic symbol can be specified through its name only. Token Pattern Attributes TOKEN_if Keyword if No attribute TOKEN_while Keyword while No attribute TOKEN_SUM + No attribute TOKEN_ASSIGN = No attribute TOKEN_ID Character sequence starting with a letter index Other attributes TOKEN_ID Character sequence starting with a letter X Other attributes TOKEN_NUM Sequence of digits Other attributes TOKEN_NUM Sequence of digits Other attributes Lexical analysis - Floriano Scioscia 7

8 Tokens and symbol table Examples of tokens 1. Keywords (e.g., IF) 2. Punctuation marks (e.g., ;) 3. Operators composed by single or multiple characters (e.g., =, ==, <=) 4. Identifiers 5. Numbers Integers Reals 6. Comments Since a token can match with one or more lexical elements, other information can be added to specify it (semantic attributes). For simplicity, a token can have just one attribute grouping all the pertaining information. For identifiers, this attribute is a pointer to the symbol table; the symbol table contains the actual attributes of the token. alpha = (beta * 3) id, alpha assign, left, id, beta times, num, 3 right, Lexical analysis - Floriano Scioscia 8

9 Role of the lexical analyzer (1/2) Primary role = abstraction process: characters symbols There is no need to separate lexical and syntax analysis, only the greater ease of management of the separate modules. When the scanner and the parser work at the same time, the scanner does not return a list of tokens, but one token at a time, when the parser requests it. source program Lexical analyzer symbol next() Syntax analyzer Benefits of separating lexical and syntax analysis: 1. Design simplicity 2. Efficiency 3. Portability Symbol table Lexical analysis - Floriano Scioscia 9

10 Role of the lexical analyzer (2/2) Since the scanner is part of the compiler reading the source code, it can perform other tasks besides identifying lexemes and tokens for the parser: Removing whitespace and comments; Inserting symbols into the symbol table; Deleting/inserting/replacing input characters or swapping two adjacent characters; Numbering code lines, so as to associate to a possible error message the corresponding line number; Providing a way to isolate the low-level rules from the structures forming the language syntax. Lexical analysis - Floriano Scioscia 10

11 Lexical analyzer operation (1/2) The lexical analyzer recognizes as a token the longest possible string. Example newval -- n ne new newv newva newval Normally, an end-of-token character is not defined. What is the end of a token? Is there an end-of-token marker character? If the number of token characters is fixed, no problem: (example: + - in Pascal, but not in C) Delimiters/spacers: characters on which a string representing a symbol ends (i.e. token boundaries ): (blank tab newline comment)+ Some lookahead is therefore needed, that is a maximum number of characters within which token delimiters must be found. Lexical analysis - Floriano Scioscia 11

12 Lexical analyzer operation (2/2) Since a formal language of the regular type is recognizable by finitestate automata, the scanner is an algorithm implementing the transition function of a finite-state automaton. More precisely, the lexical analyzer must not only check whether a substring of the source text matches with a valid lexical element, but it must also translate it into a proper encoding facilitating further processing (syntax analysis) by the compiler or interpreter. The encoding must contain two information elements: the identifier of the lexical class the element belongs to and semantic attributes (if the class needs them). For example, string is recognized as a real number constant and transformed into a ( real_constant, constant_value) pair. Lexical analysis - Floriano Scioscia 12

13 Scanner implementation (1/2) In order to implement a scanner, three approaches can be used: 1. Procedural or hand-coded implementation: an ad-hoc program for grammar G regular grammar ad-hoc program regular expression ad-hoc function 2. Table-driven interpreted implementation: a data structure (table) representing the recognizer DFA for grammar G and a program independent from the particular grammar (driver) regular grammar DFA regular expression DFA 3. Automatic, with a Scanner Generator Lexical analysis - Floriano Scioscia 13

14 Scanner implementation (2/2) Regular expressions are a formal notation powerful enough to describe the variety of tokens adopted by modern programming languages. Furthermore, they can be used as a specification to automatically generate finite-state automata, which can recognize the sets defined by regular expressions. This interpretation of regular expressions is at the foundation of scanner generators, programs which produce a working scanner, once the specification of tokens is given. This kind of programs is clearly a valuable tool in compiler development. Alternatively, scanners can be hand-coded (without the support of a tool) to recognize tokens of a particular language. The latter approach could be justified by considering sometimes more work and time are needed for learning the use of a scanner generator than for writing the scanner directly. Lexical analysis - Floriano Scioscia 14

15 Hand-coded vs table-driven implementation Hand-coded scanners have been a common practice until recently, as they were thought to be faster than table-driven ones created by generators. Every extra overhead can be relevant, as scanning represents a significant fraction of the compilation time. For example, some compilers were reported to spend 20% of their time just skipping spaces. Recent investigations evidenced that table-driven scanners can always be faster than ones coded without support tools. A further benefit is that the same driver can be used for several scanners, just changing the tables. In conclusion, we can say that, once one masters the use of scanner generators, scanner implementation becomes much easier and more effective. Lexical analysis - Floriano Scioscia 15

16 Hand-coded implementation (1/3) Regular expression Code Write: a sequence for each concatenation a selection for each union an iteration for each Kleene closure Example: r = c (c 0)* 0 { read(character); if (character == c ) { read(character); while (character == c character == 0 ) read(character); return ok;} if (character == 0 ) return ok; return error; } Lexical analysis - Floriano Scioscia 16

17 Hand-coded implementation (2/3) Regular grammar Code Write a function for each nonterminal; Write a selection for each alternative in the right-hand side of a derivation; For each nonterminal in the right-hand side of a derivation, call the corresponding function. Lexical analysis - Floriano Scioscia 17

18 Hand-coded implementation (3/3) Example: grammar V = {S, A} T = {0, 1, 2, 3, 4, 5, 6, 7, 8, 9} P = { S 0 S 1A 2A 3A 4A 5A 6A 7A 8A 9A A 0A 1A 2A 3A 4A 5A 6A 7A 8A 9A } int S () { read(character) if (character == 0) { read(character) if (character == ) return OK else return ERROR } else if (character == 1...9) return A() else return ERROR } int A () { read(character) if (character == ) return OK else if (character == 0..9) return A() else return ERROR } Lexical analysis - Floriano Scioscia 18

19 Table-driven implementation Regular expression DFA DFA Input string Generic driver code Decision letter other * letter 1 2 digit other 3 / 1 2 * 3 * other 4 / 5 C letter digit other F [3] 3 Blank cells undefined transitions C / * other F Lexical analysis - Floriano Scioscia 19

20 Lexical error recovery A character sequence not corresponding to any valid token is a lexical error. Lexical errors are infrequent, but they must be managed by the scanner. Stopping the compilation process for such kind of error is inappropriate. Typical lexical error management approaches are: Removing characters read up to the error and resume scanning; Remove the first character read by the scanner and restart scanning from the next character. A lexical error is usually due to the presence of some illegal character, often at the beginning of a token. In that case, the two above approaches are equivalent. Lexical analysis - Floriano Scioscia 20