Compiler Construction

Size: px
Start display at page:

Download "Compiler Construction"

Transcription

1 Compiler Construction Regular expressions Scanning Görel Hedin Reviderad a 2013 Compiler Construction 2013 F02-1

2 Compiler overview source code lexical analysis tokens intermediate code generation intermediate code syntactic analysis attributed AST optimization AST semantic analysis analysis intermediate code machine code generation synthesis machine code Compiler Construction 2013 F02-2

3 Compiler phases source code scanner regular expressions tokens lexical analysis parser context free grammar parse tree (implicit) AST builder AST (explicit) syntactic analysis Compiler Construction 2013 F02-3

4 Typical tokens Reserved words (keywords) Identifiers example lexemes Literals Operators Separators Compiler Construction 2013 F02-4

5 Typical tokens Reserved words (keywords) Identifiers example lexemes if then else begin i alpha k10 Literals A text Operators + ++!= Separators ;, ( Compiler Construction 2013 F02-5

6 Non-tokens whitespace example lexemes comments Compiler Construction 2013 F02-6

7 Non-tokens whitespace comments example lexemes tab newline formfeed /* comment */ //eol comment Compiler Construction 2013 F02-7

8 Formal languages An alphabet, Σ, is a set of symbols. (Nonempty and finite) A string is a sequence of symbols. (Each string is finite) A formal language, L, is a set of strings. (Can be infinite) Compiler Construction 2013 F02-8

9 Formal languages An alphabet, Σ, is a set of symbols. (Nonempty and finite) A string is a sequence of symbols. (Each string is finite) A formal language, L, is a set of strings. (Can be infinite) We would like to have rules or algorithms for deciding if a certain string over the alphabet belongs to the language or not. Compiler Construction 2013 F02-8

10 Example: Languages over binary numbers Suppose we have the alphabet Σ = {0, 1} Example languages: The set of all possible combinations of zeros and ones: L 0 = {0, 1, 00, 01, 10, 11, 000,...} All binary numbers without unnecessary leading zeros: L 1 = {0, 1, 10, 11, 100, 101, 110, 111, 1000,...} All binary numbers with two digits: L 2 = {00, 01, 10, 11}... Compiler Construction 2013 F02-9

11 Example: Languages over UNICODE Here, the alphabet Σ is the set of UNICODE characters Example languages: All possible Java keywords: { class, import, public,...} All possible lexemes corresponding to Java tokens. All possible lexemes corresponding to Java whitespace. All binary numbers... Compiler Construction 2013 F02-10

12 Example: Languages over Java tokens Here, the alphabet Σ is the set of Java tokens Example languages: All syntactically correct Java programs All that are syntactically incorrect All that are compile-time correct All that terminate Compiler Construction 2013 F02-11

13 Example: Languages over Java tokens Here, the alphabet Σ is the set of Java tokens Example languages: All syntactically correct Java programs All that are syntactically incorrect All that are compile-time correct All that terminate (undecidable)... Compiler Construction 2013 F02-11

14 Defining languages using rules Increasingly powerful: Regular expressions Context-free grammars Attribute grammars Compiler Construction 2013 F02-12

15 Regular expressions (core notation) RE read is called a a symbol M N M or N alternative M N M followed by N concatenation ɛ the empty string epsilon M zero or more M repetition (Kleene star) (M) where a is a symbol in the alphabet (e.g., {0, 1} or UNICODE) and M and N are regular expressions Each regular expression defines a language over the alphabet (a set of strings that belong to the language). Priorities: M N P = M (N (P )) Compiler Construction 2013 F02-13

16 Example a b c means Compiler Construction 2013 F02-14

17 Example a b c means {a, b, bc, bcc, bccc,...} Compiler Construction 2013 F02-14

18 Regular expressions with extended notation RE read is called a a symbol M N M or N alternative M N M followed by N concatenation ɛ the empty string epsilon M zero or more M repetition (Kleene star) (M) means M + at least one... M M M? optional... ɛ M [a za Z] one of... a b... z A... Z [0 9] not... one character, but not anyone of those listed a+b the string... a \+ b Compiler Construction 2013 F02-15

19 Exercise Write a regular expression that defines the language of all decimal numbers, like But not numbers lacking an integer part. And not numbers with a decimal point but lacking a fractional part. So not numbers like Leading and trailing zeros are allowed. So the following numbers are ok: a) Use the extended notation. b) Then translate the expression to the core notation. c) Then write an expression that disallows leading zeros. Compiler Construction 2013 F02-16

20 Solution a) [0 9] + (. [0 9] + )? b) (0... 9) (0... 9) (ɛ (. ((0... 9) (0... 9) ))) c) 0 (. [0 9] + )? [1 9] [0 9] (. [0 9] + )? Compiler Construction 2013 F02-17

21 JavaCC notation Appel JavaCC ɛ M + M+ M? M? [a za Z] [ a - z, A - Z ] [ 0-9 ] a b c abc \n \n Compiler Construction 2013 F02-18

22 JavaCC example... /* Skip whitespace */ SKIP : { " " "\t" "\n" "\r" } /* Reserved words */ TOKEN [IGNORE_CASE]: { < IF: "if"> < THEN: "then"> } /* Identifiers */ TOKEN: { < ID: (["A"-"Z", "a"-"z"])+ > } Compiler Construction 2013 F02-19

23 ... JavaCC example /* Literals */ TOKEN: { < INT: (["0"-"9"])+ > } /* Operators and separators */ TOKEN: { < ASSIGN: "=" > < MULT: "*" > < LT: "<" > < LE: "<=" >... } Compiler Construction 2013 F02-20

24 Ambiguous lexical definitions The scanned text can match tokens in more than one way Which tokens match <=? Compiler Construction 2013 F02-21

25 Ambiguous lexical definitions The scanned text can match tokens in more than one way Which tokens match <=? Which tokens match if? Compiler Construction 2013 F02-21

26 Extra rules for resolving the ambiguities Longest match If one rule can be used to match a token, but there is another rule that will match a longer token, the latter rule will be chosen. This way, the scanner will match the longest token possible. Compiler Construction 2013 F02-22

27 Extra rules for resolving the ambiguities Longest match If one rule can be used to match a token, but there is another rule that will match a longer token, the latter rule will be chosen. This way, the scanner will match the longest token possible. Rule priority If two rules can be used to match the same sequence of characters, the first one takes priority. Compiler Construction 2013 F02-22

28 Implementation of scanners Finite automata 1 i f 2 3 state IF a transition start state INT final state Compiler Construction 2013 F02-23

29 More automata WHITESPACE ID Compiler Construction 2013 F02-24

30 More automata WHITESPACE 1 \t \n ID a z a z 1 2 Compiler Construction 2013 F02-25

31 Automaton for the whole language Combine the automata for individual tokens merge the start states re-enumerate the states so they get unique numbers mark each final state with the token that is matched Compiler Construction 2013 F02-26

32 Example 1 Combine IF and INT Compiler Construction 2013 F02-27

33 Example 1 1 i f IF INT Compiler Construction 2013 F02-28

34 Example 2 Combine IF and ID Compiler Construction 2013 F02-29

35 Example 2 1 i f 2 3 a z0 9 IF a z 4 ID Compiler Construction 2013 F02-30

36 Is the automaton deterministic? Compiler Construction 2013 F02-31

37 Translating a RE to an NFA a MN M N M ɛ Compiler Construction 2013 F02-32

38 DFA vs. NFA Deterministic Finite Automaton (DFA) A finite automaton is deterministic if all outgoing edges from any given node have disjoint character sets there are no ɛ edges (the empty string) Can be implemented efficiently Compiler Construction 2013 F02-33

39 DFA vs. NFA Deterministic Finite Automaton (DFA) A finite automaton is deterministic if all outgoing edges from any given node have disjoint character sets there are no ɛ edges (the empty string) Can be implemented efficiently Non-deterministic Finite Automaton (NFA) An NFA may have two outgoing edges with overlapping character sets ɛ-edges DFA NFA Compiler Construction 2013 F02-33

40 Each NFA can be translated to a DFA Simulate the NFA keep track of a set of current NFA-states follow ɛ edges to extend the current set (take the closure) Construct the corresponding DFA Each such set of NFA states corresponds to one DFA state If any of the NFA states are final, the DFA state is also final, and is marked with the corresponding token. If there is more than one token to choose from, select the token that is defined first (rule priority) (Minimize the DFA for efficiency) Compiler Construction 2013 F02-34

41 Translate an NFA to a DFA NFA start 1 f 2 3 i a-z a-z 4 ID IF Compiler Construction 2013 F02-35

42 Translate an NFA to a DFA NFA 2 f 3 i IF start 1 a-z a-z 4 ID DFA 2,4 f 3,4 i a-eg-z IF start 1 a-z a-z a-hj-z 4 ID Compiler Construction 2013 F02-36

43 Construction of a Scanner Define all tokens as regular expressions Construct a finite automaton for each token Combine all these automata to a new automaton (an NFA in general) Translate to a corresponding DFA Minimize the DFA Implement the DFA Compiler Construction 2013 F02-37

44 Implementation alternatives for DFAs Table-driven Represent the automaton by a table Additional table to keep track of final states and token kinds A global variable keeps track of the current state Compiler Construction 2013 F02-38

45 Implementation alternatives for DFAs Table-driven Represent the automaton by a table Additional table to keep track of final states and token kinds A global variable keeps track of the current state Switch statements Each state is implemented as a switch statement Each case implements a state transition as a jump (to another switch statement). The current state is represented by the program counter Compiler Construction 2013 F02-38

46 Error handling Add a dead state (state 0), corresponding to erroneous input Add transitions to the dead state for all erroneous input Generate an Error token when the dead state is reached Compiler Construction 2013 F02-39

47 Table-driven implementation DFA for IF and ID 1 2,4 3,4 4.. a-e f g-h i j-z.. final kind true ERROR Compiler Construction 2013 F02-40

48 Table-driven implementation DFA for IF and ID.. a-e f g-h i j-z.. final kind 0 true ERROR ,4 4 0 false 2, , true ID 3, true IF true ID Compiler Construction 2013 F02-41

49 Design Token int kind String string Parser Scanner Token next() Reader int get() Compiler Construction 2013 F02-42

50 Scanner (sketch) class Scanner { PushbackReader reader; boolean isfinal [ ]; int edges [ ] [ ]; int kind [ ]; Token next() { } } Compiler Construction 2013 F02-43

51 Scanner (sketch) class Scanner { //does not handle: PushbackReader reader; //longest match boolean isfinal [ ]; //eof int edges [ ] [ ]; //whitespace int kind [ ]; //token strings Token next() { int state = 1; //start state while(!isfinal[state]) { char ch = reader.read(); state = edges[state][ch]; } return new Token(kind[state]); } } Compiler Construction 2013 F02-44

52 Handling End-Of-File (EOF) and non-tokens Compiler Construction 2013 F02-45

53 Handling End-Of-File (EOF) and non-tokens EOF construct an explicit EOF token when the EOF character is read Non-tokens (Whitespace & Comments) Errors view as tokens of a special kind scan them as normal tokens, but don t create token objects for them loop in next() until a real token has been found construct an explicit ERROR token to be returned when no valid token can be found. Compiler Construction 2013 F02-45

54 Handling longest match When a token is matched (a final state reached), don t stop scanning. Keep track of the latest matched token in a variable. Continue scanning until we reach the dead state Restore the input stream. Use PushbackReader to be able to do this. Return the latest matched token (or return the ERROR token if there was no latest matched token) Compiler Construction 2013 F02-46

55 Scanner with longest match class Scanner {... Token next() { } } Compiler Construction 2013 F02-47

56 Scanner with longest match class Scanner {... Token next() { int state = 1, lastfinalstate = 0; int lastfinalstr = "": StringBuilder builder = new StringBuilder(); while(state!=0) { char ch = reader.read(); builder.append(ch); state = edges[state][ch]; if(isfinal[state]) { lastfinalstate = state; lastfinalstr = builder.tostring(); } } reader.unread(... ); //unused characters return new Token(kind[lastFinalState], lastfinalstr); } Compiler Construction 2013 F02-48

57 JavaCC as a scanner generator toy.jj sum.toy javacc ToyTokenManager.java ToyConstants.java...java tokens Compiler Construction 2013 F02-49

58 Other scanner generators tool author implementation generates lex Schmidt, Lesk, 1975 table-driven C code flex Paxon switch statements C code jlex table-driven Java code jflex switch statements Java code Compiler Construction 2013 F02-50

59 Additional functionality Lexical actions extra (Java) code written in the specification for adapting the generated scanner e.g., create special token objects, count tokens, keep track of position in input text (line and column numbers), etc. Compiler Construction 2013 F02-51

60 Additional functionality Lexical actions extra (Java) code written in the specification for adapting the generated scanner e.g., create special token objects, count tokens, keep track of position in input text (line and column numbers), etc. Lexical states ( start states ) gives the possibility to switch automata during scanning makes it easier to scan certain tokens, e.g., multi-line comments makes it possible to scan certain combined languages, e.g., HTML Compiler Construction 2013 F02-51

61 Lexcial states, example Scanning of /*... */ Compiler Construction 2013 F02-52

62 Summary questions What is meant by an ambiguous lexical definition? Give some typical examples of this and how they may be resolved. Construct an NFA for a given lexical definition Construct a DFA for a given NFA What is the difference between a DFA and an NFA? Implement a DFA in Java. How is rule priority handled in the implementation? How is longest match handled? EOF? Whitespace? Compiler Construction 2013 F02-53

63 What s next? Tomorrow: Seminar 1 (Scanning) Next week: Parsing and Assignment 1 (Scanning) Compiler Construction 2013 F02-54

Lexical analysis FORMAL LANGUAGES AND COMPILERS. Floriano Scioscia. Formal Languages and Compilers A.Y. 2015/2016

Lexical analysis FORMAL LANGUAGES AND COMPILERS. Floriano Scioscia. Formal Languages and Compilers A.Y. 2015/2016 Master s Degree Course in Computer Engineering Formal Languages FORMAL LANGUAGES AND COMPILERS Lexical analysis Floriano Scioscia 1 Introductive terminological distinction Lexical string or lexeme = meaningful

More information

Scanner. tokens scanner parser IR. source code. errors

Scanner. tokens scanner parser IR. source code. errors Scanner source code tokens scanner parser IR errors maps characters into tokens the basic unit of syntax x = x + y; becomes = + ; character string value for a token is a lexeme

More information

03 - Lexical Analysis

03 - Lexical Analysis 03 - Lexical Analysis First, let s see a simplified overview of the compilation process: source code file (sequence of char) Step 2: parsing (syntax analysis) arse Tree Step 1: scanning (lexical analysis)

More information

Lexical Analysis. Scanners, Regular expressions, and Automata. cs4713 1

Lexical Analysis. Scanners, Regular expressions, and Automata. cs4713 1 Lexical Analysis Scanners, Regular expressions, and Automata cs4713 1 Phases of compilation Compilers Read input program optimization translate into machine code front end mid end back end Lexical analysis

More information

Regular Expressions. Languages. Recall. A language is a set of strings made up of symbols from a given alphabet. Computer Science Theory 2

Regular Expressions. Languages. Recall. A language is a set of strings made up of symbols from a given alphabet. Computer Science Theory 2 Regular Expressions Languages Recall. A language is a set of strings made up of symbols from a given alphabet. Computer Science Theory 2 1 String Recognition Machine Given a string and a definition of

More information

Monday, August 27, 12. Scanners

Monday, August 27, 12. Scanners Scanners Scanners Sometimes called lexers Recall: scanners break input stream up into a set of tokens Identifiers, reserved words, literals, etc. What do we need to know? How do we define tokens? How can

More information

COMP 356 Programming Language Structures Notes for Chapter 4 of Concepts of Programming Languages Scanning and Parsing

COMP 356 Programming Language Structures Notes for Chapter 4 of Concepts of Programming Languages Scanning and Parsing COMP 356 Programming Language Structures Notes for Chapter 4 of Concepts of Programming Languages Scanning and Parsing The scanner (or lexical analyzer) of a compiler processes the source program, recognizing

More information

Programming Languages CIS 443

Programming Languages CIS 443 Course Objectives Programming Languages CIS 443 0.1 Lexical analysis Syntax Semantics Functional programming Variable lifetime and scoping Parameter passing Object-oriented programming Continuations Exception

More information

Massachusetts Institute of Technology Department of Electrical Engineering and Computer Science

Massachusetts Institute of Technology Department of Electrical Engineering and Computer Science Massachusetts Institute of Technology Department of Electrical Engineering and Computer Science 6.035, Fall 2005 Handout 7 Scanner Parser Project Wednesday, September 7 DUE: Wednesday, September 21 This

More information

Lexical Analysis and Scanning. Honors Compilers Feb 5 th 2001 Robert Dewar

Lexical Analysis and Scanning. Honors Compilers Feb 5 th 2001 Robert Dewar Lexical Analysis and Scanning Honors Compilers Feb 5 th 2001 Robert Dewar The Input Read string input Might be sequence of characters (Unix) Might be sequence of lines (VMS) Character set ASCII ISO Latin-1

More information

Programming Assignment II Due Date: See online CISC 672 schedule Individual Assignment

Programming Assignment II Due Date: See online CISC 672 schedule Individual Assignment Programming Assignment II Due Date: See online CISC 672 schedule Individual Assignment 1 Overview Programming assignments II V will direct you to design and build a compiler for Cool. Each assignment will

More information

Introduction to Automata Theory. Reading: Chapter 1

Introduction to Automata Theory. Reading: Chapter 1 Introduction to Automata Theory Reading: Chapter 1 1 What is Automata Theory? Study of abstract computing devices, or machines Automaton = an abstract computing device Note: A device need not even be a

More information

Objects for lexical analysis

Objects for lexical analysis Rochester Institute of Technology RIT Scholar Works Articles 2002 Objects for lexical analysis Bernd Kuhl Axel-Tobias Schreiner Follow this and additional works at: http://scholarworks.rit.edu/article

More information

Theory of Computation

Theory of Computation Theory of Computation For Computer Science & Information Technology By www.thegateacademy.com Syllabus Syllabus for Theory of Computation Regular Expressions and Finite Automata, Context-Free Grammar s

More information

Programming Languages Third Edition

Programming Languages Third Edition Programming Languages Third Edition Chapter 6 Syntax Objectives Understand the lexical structure of programming languages Understand context-free grammars and BNFs Become familiar with parse trees and

More information

Compilers Lexical Analysis

Compilers Lexical Analysis Compilers Lexical Analysis SITE : http://www.info.univ-tours.fr/ mirian/ TLC - Mírian Halfeld-Ferrari p. 1/3 The Role of the Lexical Analyzer The first phase of a compiler. Lexical analysis : process of

More information

Definition: String concatenation. Definition: String. Definition: Language (cont.) Definition: Language

Definition: String concatenation. Definition: String. Definition: Language (cont.) Definition: Language CMSC 330: Organization of Programming Languages Regular Expressions and Finite Automata Introduction That s it for the basics of Ruby If you need other material for your project, come to office hours or

More information

Programming Language Syntax

Programming Language Syntax Programming Language Syntax Natural language vs. programming language No ambiguity allowed in programming languages in form (syntax) and meaning (semantics) Thus, programming language specifications use

More information

Scanning and parsing. Topics. Announcements Pick a partner by Monday Makeup lecture will be on Monday August 29th at 3pm

Scanning and parsing. Topics. Announcements Pick a partner by Monday Makeup lecture will be on Monday August 29th at 3pm Scanning and Parsing Announcements Pick a partner by Monday Makeup lecture will be on Monday August 29th at 3pm Today Outline of planned topics for course Overall structure of a compiler Lexical analysis

More information

Pushdown automata. Informatics 2A: Lecture 9. Alex Simpson. 3 October, 2014. School of Informatics University of Edinburgh als@inf.ed.ac.

Pushdown automata. Informatics 2A: Lecture 9. Alex Simpson. 3 October, 2014. School of Informatics University of Edinburgh als@inf.ed.ac. Pushdown automata Informatics 2A: Lecture 9 Alex Simpson School of Informatics University of Edinburgh als@inf.ed.ac.uk 3 October, 2014 1 / 17 Recap of lecture 8 Context-free languages are defined by context-free

More information

Automata and Computability. Solutions to Exercises

Automata and Computability. Solutions to Exercises Automata and Computability Solutions to Exercises Fall 25 Alexis Maciel Department of Computer Science Clarkson University Copyright c 25 Alexis Maciel ii Contents Preface vii Introduction 2 Finite Automata

More information

Context Free and Non Context Free Languages

Context Free and Non Context Free Languages Context Free and Non Context Free Languages Part I: Pumping Theorem for Context Free Languages Context Free Languages and Push Down Automata Theorems For Every Context Free Grammar there Exists an Equivalent

More information

Computing Functions with Turing Machines

Computing Functions with Turing Machines CS 30 - Lecture 20 Combining Turing Machines and Turing s Thesis Fall 2008 Review Languages and Grammars Alphabets, strings, languages Regular Languages Deterministic Finite and Nondeterministic Automata

More information

Regular Expressions in JLex

Regular Expressions in JLex Regular Expressions in JLex To define a token in JLex, the user to associates a regular expression with commands coded in Java. When input characters that match a regular expression are read, the corresponding

More information

CSCI-GA Compiler Construction Lecture 4: Lexical Analysis I. Mohamed Zahran (aka Z)

CSCI-GA Compiler Construction Lecture 4: Lexical Analysis I. Mohamed Zahran (aka Z) CSCI-GA.2130-001 Compiler Construction Lecture 4: Lexical Analysis I Mohamed Zahran (aka Z) mzahran@cs.nyu.edu Role of the Lexical Analyzer Remove comments and white spaces (aka scanning) Macros expansion

More information

Programming Project 1: Lexical Analyzer (Scanner)

Programming Project 1: Lexical Analyzer (Scanner) CS 331 Compilers Fall 2015 Programming Project 1: Lexical Analyzer (Scanner) Prof. Szajda Due Tuesday, September 15, 11:59:59 pm 1 Overview of the Programming Project Programming projects I IV will direct

More information

Automata and Languages

Automata and Languages Automata and Languages Computational Models: An idealized mathematical model of a computer. A computational model may be accurate in some ways but not in others. We shall be defining increasingly powerful

More information

Automata Theory and Languages

Automata Theory and Languages Automata Theory and Languages SITE : http://www.info.univ-tours.fr/ mirian/ Automata Theory, Languages and Computation - Mírian Halfeld-Ferrari p. 1/1 Introduction to Automata Theory Automata theory :

More information

The Halting Problem is Undecidable

The Halting Problem is Undecidable 185 Corollary G = { M, w w L(M) } is not Turing-recognizable. Proof. = ERR, where ERR is the easy to decide language: ERR = { x { 0, 1 }* x does not have a prefix that is a valid code for a Turing machine

More information

University of Toronto Department of Electrical and Computer Engineering. Midterm Examination. CSC467 Compilers and Interpreters Fall Semester, 2005

University of Toronto Department of Electrical and Computer Engineering. Midterm Examination. CSC467 Compilers and Interpreters Fall Semester, 2005 University of Toronto Department of Electrical and Computer Engineering Midterm Examination CSC467 Compilers and Interpreters Fall Semester, 2005 Time and date: TBA Location: TBA Print your name and ID

More information

6.045: Automata, Computability, and Complexity Or, Great Ideas in Theoretical Computer Science Spring, 2010. Class 4 Nancy Lynch

6.045: Automata, Computability, and Complexity Or, Great Ideas in Theoretical Computer Science Spring, 2010. Class 4 Nancy Lynch 6.045: Automata, Computability, and Complexity Or, Great Ideas in Theoretical Computer Science Spring, 2010 Class 4 Nancy Lynch Today Two more models of computation: Nondeterministic Finite Automata (NFAs)

More information

CS412/CS413. Introduction to Compilers Tim Teitelbaum. Lecture 18: Intermediate Code 29 Feb 08

CS412/CS413. Introduction to Compilers Tim Teitelbaum. Lecture 18: Intermediate Code 29 Feb 08 CS412/CS413 Introduction to Compilers Tim Teitelbaum Lecture 18: Intermediate Code 29 Feb 08 CS 412/413 Spring 2008 Introduction to Compilers 1 Summary: Semantic Analysis Check errors not detected by lexical

More information

Compiler I: Syntax Analysis Human Thought

Compiler I: Syntax Analysis Human Thought Course map Compiler I: Syntax Analysis Human Thought Abstract design Chapters 9, 12 H.L. Language & Operating Sys. Compiler Chapters 10-11 Virtual Machine Software hierarchy Translator Chapters 7-8 Assembly

More information

Finite Automata. Reading: Chapter 2

Finite Automata. Reading: Chapter 2 Finite Automata Reading: Chapter 2 1 Finite Automaton (FA) Informally, a state diagram that comprehensively captures all possible states and transitions that a machine can take while responding to a stream

More information

Regular Expressions and Automata using Haskell

Regular Expressions and Automata using Haskell Regular Expressions and Automata using Haskell Simon Thompson Computing Laboratory University of Kent at Canterbury January 2000 Contents 1 Introduction 2 2 Regular Expressions 2 3 Matching regular expressions

More information

Chapter Four: DFA Applications. Formal Language, chapter 4, slide 1

Chapter Four: DFA Applications. Formal Language, chapter 4, slide 1 Chapter Four: DFA Applications Formal Language, chapter 4, slide 1 1 We have seen how DFAs can be used to define formal languages. In addition to this formal use, DFAs have practical applications. DFAbased

More information

(IALC, Chapters 8 and 9) Introduction to Turing s life, Turing machines, universal machines, unsolvable problems.

(IALC, Chapters 8 and 9) Introduction to Turing s life, Turing machines, universal machines, unsolvable problems. 3130CIT: Theory of Computation Turing machines and undecidability (IALC, Chapters 8 and 9) Introduction to Turing s life, Turing machines, universal machines, unsolvable problems. An undecidable problem

More information

CSCI 3136 Principles of Programming Languages

CSCI 3136 Principles of Programming Languages CSCI 3136 Principles of Programming Languages Faculty of Computer Science Dalhousie University Winter 2013 CSCI 3136 Principles of Programming Languages Faculty of Computer Science Dalhousie University

More information

Introduction to Lex. General Description Input file Output file How matching is done Regular expressions Local names Using Lex

Introduction to Lex. General Description Input file Output file How matching is done Regular expressions Local names Using Lex Introduction to Lex General Description Input file Output file How matching is done Regular expressions Local names Using Lex General Description Lex is a program that automatically generates code for

More information

Semantic Analysis: Types and Type Checking

Semantic Analysis: Types and Type Checking Semantic Analysis Semantic Analysis: Types and Type Checking CS 471 October 10, 2007 Source code Lexical Analysis tokens Syntactic Analysis AST Semantic Analysis AST Intermediate Code Gen lexical errors

More information

CSC4510 AUTOMATA 2.1 Finite Automata: Examples and D efinitions Definitions

CSC4510 AUTOMATA 2.1 Finite Automata: Examples and D efinitions Definitions CSC45 AUTOMATA 2. Finite Automata: Examples and Definitions Finite Automata: Examples and Definitions A finite automaton is a simple type of computer. Itsoutputislimitedto yes to or no. It has very primitive

More information

Lex The Idea Program Structure What Lex does Using Lex Lex Operators States in Lex

Lex The Idea Program Structure What Lex does Using Lex Lex Operators States in Lex Lex The Idea Program Structure What Lex does Using Lex Lex Operators States in Lex The Lex Idea Lex is an implementation that uses an extension of RE notation as well as states and C/C++ Extending REs

More information

Operator precedence parsing

Operator precedence parsing Operator precedence parsing Bottom-up parsing methods that follow the idea of shift-reduce parsers Several flavors: operator, simple, and weak precedence. In this course, only weak precedence Main di erences

More information

Compiler Construction

Compiler Construction Compiler Construction Lecture 1 - An Overview 2003 Robert M. Siegfried All rights reserved A few basic definitions Translate - v, a.to turn into one s own language or another. b. to transform or turn from

More information

Motivation. Automata = abstract computing devices. Turing studied Turing Machines (= computers) before there were any real computers

Motivation. Automata = abstract computing devices. Turing studied Turing Machines (= computers) before there were any real computers Motivation Automata = abstract computing devices Turing studied Turing Machines (= computers) before there were any real computers We will also look at simpler devices than Turing machines (Finite State

More information

Evaluation of JFlex Scanner Generator Using Form Fields Validity Checking

Evaluation of JFlex Scanner Generator Using Form Fields Validity Checking ISSN (Online): 1694-0784 ISSN (Print): 1694-0814 12 Evaluation of JFlex Scanner Generator Using Form Fields Validity Checking Ezekiel Okike 1 and Maduka Attamah 2 1 School of Computer Studies, Kampala

More information

Introduction. Compiler Design CSE 504. Overview. Programming problems are easier to solve in high-level languages

Introduction. Compiler Design CSE 504. Overview. Programming problems are easier to solve in high-level languages Introduction Compiler esign CSE 504 1 Overview 2 3 Phases of Translation ast modifled: Mon Jan 28 2013 at 17:19:57 EST Version: 1.5 23:45:54 2013/01/28 Compiled at 11:48 on 2015/01/28 Compiler esign Introduction

More information

Automata and Formal Languages

Automata and Formal Languages Automata and Formal Languages Winter 2009-2010 Yacov Hel-Or 1 What this course is all about This course is about mathematical models of computation We ll study different machine models (finite automata,

More information

Lecture 9. Semantic Analysis Scoping and Symbol Table

Lecture 9. Semantic Analysis Scoping and Symbol Table Lecture 9. Semantic Analysis Scoping and Symbol Table Wei Le 2015.10 Outline Semantic analysis Scoping The Role of Symbol Table Implementing a Symbol Table Semantic Analysis Parser builds abstract syntax

More information

Grammars & Parsing. Lecture 13 CS 2112 Fall 2014

Grammars & Parsing. Lecture 13 CS 2112 Fall 2014 Grammars & Parsing Lecture 13 CS 2112 Fall 2014 Motivation The cat ate the rat. The cat ate the rat slowly. The small cat ate the big rat slowly. The small cat ate the big rat on the mat slowly. The small

More information

Textual Modeling Languages

Textual Modeling Languages Textual Modeling Languages Slides 4-31 and 38-40 of this lecture are reused from the Model Engineering course at TU Vienna with the kind permission of Prof. Gerti Kappel (head of the Business Informatics

More information

Context-Free Grammars

Context-Free Grammars Context-Free Grammars Describing Languages We've seen two models for the regular languages: Automata accept precisely the strings in the language. Regular expressions describe precisely the strings in

More information

the lemma. Keep in mind the following facts about regular languages:

the lemma. Keep in mind the following facts about regular languages: CPS 2: Discrete Mathematics Instructor: Bruce Maggs Assignment Due: Wednesday September 2, 27 A Tool for Proving Irregularity (25 points) When proving that a language isn t regular, a tool that is often

More information

What The Regex_Cat Dragged In

What The Regex_Cat Dragged In What The Regex_Cat Dragged In Kyle Hamilton COS210 - MCCC, Prof. Bostain December 2013 Regular Expression: From Wikipedia, the free encyclopedia In computing, a regular expression (abbreviated regex or

More information

Finite Automata. Reading: Chapter 2

Finite Automata. Reading: Chapter 2 Finite Automata Reading: Chapter 2 1 Finite Automata Informally, a state machine that comprehensively captures all possible states and transitions that a machine can take while responding to a stream (or

More information

An Overview of a Compiler

An Overview of a Compiler An Overview of a Compiler Department of Computer Science and Automation Indian Institute of Science Bangalore 560 012 NPTEL Course on Principles of Compiler Design Outline of the Lecture About the course

More information

Syntaktická analýza. Ján Šturc. Zima 208

Syntaktická analýza. Ján Šturc. Zima 208 Syntaktická analýza Ján Šturc Zima 208 Position of a Parser in the Compiler Model 2 The parser The task of the parser is to check syntax The syntax-directed translation stage in the compiler s front-end

More information

If-Then-Else Problem (a motivating example for LR grammars)

If-Then-Else Problem (a motivating example for LR grammars) If-Then-Else Problem (a motivating example for LR grammars) If x then y else z If a then if b then c else d this is analogous to a bracket notation when left brackets >= right brackets: [ [ ] ([ i ] j,

More information

Finite Automata and Formal Languages

Finite Automata and Formal Languages Finite Automata and Formal Languages TMV026/DIT321 LP4 2011 Lecture 14 May 19th 2011 Overview of today s lecture: Turing Machines Push-down Automata Overview of the Course Undecidable and Intractable Problems

More information

Formal Languages and Automata Theory - Regular Expressions and Finite Automata -

Formal Languages and Automata Theory - Regular Expressions and Finite Automata - Formal Languages and Automata Theory - Regular Expressions and Finite Automata - Samarjit Chakraborty Computer Engineering and Networks Laboratory Swiss Federal Institute of Technology (ETH) Zürich March

More information

Programming Languages: Syntax Description and Parsing

Programming Languages: Syntax Description and Parsing Programming Languages: Syntax Description and Parsing Onur Tolga Şehitoğlu Computer Engineering,METU 27 May 2009 Outline Introduction Introduction Syntax: the form and structure of a program. Semantics:

More information

5HFDOO &RPSLOHU 6WUXFWXUH

5HFDOO &RPSLOHU 6WUXFWXUH 6FDQQLQJ 2XWOLQH 2. Scanning The basics Ad-hoc scanning FSM based techniques A Lexical Analysis tool - Lex (a scanner generator) 5HFDOO &RPSLOHU 6WUXFWXUH 6RXUFH &RGH /H[LFDO $QDO\VLV6FDQQLQJ 6\QWD[ $QDO\VLV3DUVLQJ

More information

Reading 13 : Finite State Automata and Regular Expressions

Reading 13 : Finite State Automata and Regular Expressions CS/Math 24: Introduction to Discrete Mathematics Fall 25 Reading 3 : Finite State Automata and Regular Expressions Instructors: Beck Hasti, Gautam Prakriya In this reading we study a mathematical model

More information

Parse Trees. Prog. Prog. Stmts. { Stmts } Stmts = Stmts ; Stmt. id + id = Expr. Stmt. id = Expr. Expr + id

Parse Trees. Prog. Prog. Stmts. { Stmts } Stmts = Stmts ; Stmt. id + id = Expr. Stmt. id = Expr. Expr + id Parse Trees To illustrate a derivation, we can draw a derivation tree (also called a parse tree): Prog { Stmts } Stmts ; Stmt Stmt = xpr = xpr xpr + An abstract syntax tree (AST) shows essential structure

More information

The IC Language Specification. Spring 2006 Cornell University

The IC Language Specification. Spring 2006 Cornell University The IC Language Specification Spring 2006 Cornell University The IC language is a simple object-oriented language that we will use in the CS413 project. The goal is to build a complete optimizing compiler

More information

Chapter 2. Finite Automata. 2.1 The Basic Model

Chapter 2. Finite Automata. 2.1 The Basic Model Chapter 2 Finite Automata 2.1 The Basic Model Finite automata model very simple computational devices or processes. These devices have a constant amount of memory and process their input in an online manner.

More information

CS 164 Programming Languages and Compilers Handout 8. Midterm I

CS 164 Programming Languages and Compilers Handout 8. Midterm I Mterm I Please read all instructions (including these) carefully. There are six questions on the exam, each worth between 15 and 30 points. You have 3 hours to work on the exam. The exam is closed book,

More information

csce4313 Programming Languages Scanner (pass/fail)

csce4313 Programming Languages Scanner (pass/fail) csce4313 Programming Languages Scanner (pass/fail) John C. Lusth Revision Date: January 18, 2005 This is your first pass/fail assignment. You may develop your code using any procedural language, but you

More information

Honors Class (Foundations of) Informatics. Tom Verhoeff. Department of Mathematics & Computer Science Software Engineering & Technology

Honors Class (Foundations of) Informatics. Tom Verhoeff. Department of Mathematics & Computer Science Software Engineering & Technology Honors Class (Foundations of) Informatics Tom Verhoeff Department of Mathematics & Computer Science Software Engineering & Technology www.win.tue.nl/~wstomv/edu/hci c 2011, T. Verhoeff @ TUE.NL 1/20 Information

More information

About the Tutorial. Audience. Prerequisites. Copyright & Disclaimer. Compiler Design

About the Tutorial. Audience. Prerequisites. Copyright & Disclaimer. Compiler Design i About the Tutorial A compiler translates the codes written in one language to some other language without changing the meaning of the program. It is also expected that a compiler should make the target

More information

How to make the computer understand? Lecture 15: Putting it all together. Example (Output assembly code) Example (input program) Anatomy of a Computer

How to make the computer understand? Lecture 15: Putting it all together. Example (Output assembly code) Example (input program) Anatomy of a Computer How to make the computer understand? Fall 2005 Lecture 15: Putting it all together From parsing to code generation Write a program using a programming language Microprocessors talk in assembly language

More information

Programming Languages

Programming Languages Programming Languages Programming languages bridge the gap between people and machines; for that matter, they also bridge the gap among people who would like to share algorithms in a way that immediately

More information

NFAs with Tagged Transitions, their Conversion to Deterministic Automata and Application to Regular Expressions

NFAs with Tagged Transitions, their Conversion to Deterministic Automata and Application to Regular Expressions NFAs with Tagged Transitions, their Conversion to Deterministic Automata and Application to Regular Expressions Ville Laurikari Helsinki University of Technology Laboratory of Computer Science PL 9700,

More information

CA4003 - Compiler Construction

CA4003 - Compiler Construction CA4003 - Compiler Construction David Sinclair Overview This module will cover the compilation process, reading and parsing a structured language, storing it in an appropriate data structure, analysing

More information

Compilers I - Chapter 2: An introduction to syntax analysis (and a complete toy compiler)

Compilers I - Chapter 2: An introduction to syntax analysis (and a complete toy compiler) Compilers I - Chapter 2: An introduction to syntax analysis (and a complete toy compiler) Lecturers: Part I: Paul Kelly (phjk@doc.ic.ac.uk) Office: room 423 Part II: Naranker Dulay (nd@doc.ic.ac.uk) Office:

More information

Properties of Context-Free Languages. Decision Properties Closure Properties

Properties of Context-Free Languages. Decision Properties Closure Properties Properties of Context-Free Languages Decision Properties Closure Properties 1 Summary of Decision Properties As usual, when we talk about a CFL we really mean a representation for the CFL, e.g., a CFG

More information

University of Wales Swansea. Department of Computer Science. Compilers. Course notes for module CS 218

University of Wales Swansea. Department of Computer Science. Compilers. Course notes for module CS 218 University of Wales Swansea Department of Computer Science Compilers Course notes for module CS 218 Dr. Matt Poole 2002, edited by Mr. Christopher Whyley, 2nd Semester 2006/2007 www-compsci.swan.ac.uk/~cschris/compilers

More information

Flex/Bison Tutorial. Aaron Myles Landwehr aron+ta@udel.edu CAPSL 2/17/2012

Flex/Bison Tutorial. Aaron Myles Landwehr aron+ta@udel.edu CAPSL 2/17/2012 Flex/Bison Tutorial Aaron Myles Landwehr aron+ta@udel.edu 1 GENERAL COMPILER OVERVIEW 2 Compiler Overview Frontend Middle-end Backend Lexer / Scanner Parser Semantic Analyzer Optimizers Code Generator

More information

Automata Theory. Şubat 2006 Tuğrul Yılmaz Ankara Üniversitesi

Automata Theory. Şubat 2006 Tuğrul Yılmaz Ankara Üniversitesi Automata Theory Automata theory is the study of abstract computing devices. A. M. Turing studied an abstract machine that had all the capabilities of today s computers. Turing s goal was to describe the

More information

Automata on Infinite Words and Trees

Automata on Infinite Words and Trees Automata on Infinite Words and Trees Course notes for the course Automata on Infinite Words and Trees given by Dr. Meghyn Bienvenu at Universität Bremen in the 2009-2010 winter semester Last modified:

More information

Program translation.

Program translation. Program translation. ffl On the very earliest computers programs were written and entered in binary form. ffl Some computers required the program to be entered one binary word at a time, using swithes

More information

Unit 6. Loop statements

Unit 6. Loop statements Unit 6 Loop statements Summary Repetition of statements The while statement Input loop Loop schemes The for statement The do statement Nested loops Flow control statements 6.1 Statements in Java Till now

More information

A New Parser. Neil Mitchell. Neil Mitchell

A New Parser. Neil Mitchell. Neil Mitchell A New Parser Neil Mitchell Neil Mitchell 2004 1 Disclaimer This is not something I have done as part of my PhD I have done it on my own I haven t researched other systems Its not finished Any claims may

More information

J a v a Quiz (Unit 3, Test 0 Practice)

J a v a Quiz (Unit 3, Test 0 Practice) Computer Science S-111a: Intensive Introduction to Computer Science Using Java Handout #11 Your Name Teaching Fellow J a v a Quiz (Unit 3, Test 0 Practice) Multiple-choice questions are worth 2 points

More information

CS5236 Advanced Automata Theory

CS5236 Advanced Automata Theory CS5236 Advanced Automata Theory Frank Stephan Semester I, Academic Year 2012-2013 Advanced Automata Theory is a lecture which will first review the basics of formal languages and automata theory and then

More information

Adjusted/Modified by Nicole Tobias. Chapter 2: Basic Elements of C++

Adjusted/Modified by Nicole Tobias. Chapter 2: Basic Elements of C++ Adjusted/Modified by Nicole Tobias Chapter 2: Basic Elements of C++ Objectives In this chapter, you will: Become familiar with functions, special symbols, and identifiers in C++ Explore simple data types

More information

F U N C T I O N A L P E A R L S Monadic Parsing in Haskell

F U N C T I O N A L P E A R L S Monadic Parsing in Haskell Under consideration for publication in J. Functional Programming 1 F U N C T I O N A L P E A R L S Monadic Parsing in Haskell Graham Hutton University of Nottingham Erik Meijer University of Utrecht 1

More information

Nondeterministic Finite Automata

Nondeterministic Finite Automata Chapter, Part 2 Nondeterministic Finite Automata CSC527, Chapter, Part 2 c 202 Mitsunori Ogihara Fundamental set operations Let A and B be two languages. In addition to union and intersection, we consider

More information

Syntactic Analysis and Context-Free Grammars

Syntactic Analysis and Context-Free Grammars Syntactic Analysis and Context-Free Grammars CSCI 3136 Principles of Programming Languages Faculty of Computer Science Dalhousie University Winter 2012 Reading: Chapter 2 Motivation Source program (character

More information

Basics of Compiler Design

Basics of Compiler Design Basics of Compiler Design Anniversary edition Torben Ægidius Mogensen DEPARTMENT OF COMPUTER SCIENCE UNIVERSITY OF COPENHAGEN Published through lulu.com. c Torben Ægidius Mogensen 2000 2010 torbenm@diku.dk

More information

CS 141: Introduction to (Java) Programming: Exam 1 Jenny Orr Willamette University Fall 2013

CS 141: Introduction to (Java) Programming: Exam 1 Jenny Orr Willamette University Fall 2013 Oct 4, 2013, p 1 Name: CS 141: Introduction to (Java) Programming: Exam 1 Jenny Orr Willamette University Fall 2013 1. (max 18) 4. (max 16) 2. (max 12) 5. (max 12) 3. (max 24) 6. (max 18) Total: (max 100)

More information

C++ Programming: From Problem Analysis to Program Design, Fifth Edition. Chapter 2: Basic Elements of C++

C++ Programming: From Problem Analysis to Program Design, Fifth Edition. Chapter 2: Basic Elements of C++ C++ Programming: From Problem Analysis to Program Design, Fifth Edition Chapter 2: Basic Elements of C++ Objectives In this chapter, you will: Become familiar with the basic components of a C++ program,

More information

1 (1x17 =17 points) 2 (21 points) 3 (5 points) 4 (3 points) 5 (4 points) Total ( 50points) Page 1

1 (1x17 =17 points) 2 (21 points) 3 (5 points) 4 (3 points) 5 (4 points) Total ( 50points) Page 1 CS 1621 MIDTERM EXAM 1 Name: Problem 1 (1x17 =17 points) 2 (21 points) 3 (5 points) 4 (3 points) 5 (4 points) Total ( 50points) Score Page 1 1. (1 x 17 = 17 points) Determine if each of the following statements

More information

Compiler Design July 2004

Compiler Design July 2004 irst Prev Next Last CS432/CSL 728: Compiler Design July 2004 S. Arun-Kumar sak@cse.iitd.ernet.in Department of Computer Science and ngineering I. I.. Delhi, Hauz Khas, New Delhi 110 016. July 30, 2004

More information

CS 4120 Introduction to Compilers

CS 4120 Introduction to Compilers CS 4120 Introduction to Compilers Andrew Myers Cornell University Lecture 5: Top-down parsing 5 Feb 2016 Outline More on writing CFGs Top-down parsing LL(1) grammars Transforming a grammar into LL form

More information

NFA and regex. the Boolean algebra of languages. non-deterministic machines. regular expressions

NFA and regex. the Boolean algebra of languages. non-deterministic machines. regular expressions NFA and regex cl the Boolean algebra of languages non-deterministic machines regular expressions Informatics School of Informatics, University of Edinburgh The intersection of two regular languages is

More information

Composability of Infinite-State Activity Automata*

Composability of Infinite-State Activity Automata* Composability of Infinite-State Activity Automata* Zhe Dang 1, Oscar H. Ibarra 2, Jianwen Su 2 1 Washington State University, Pullman 2 University of California, Santa Barbara Presented by Prof. Hsu-Chun

More information

Deterministic Push-Down Store Automata

Deterministic Push-Down Store Automata Deterministic Push-Down tore Automata 1 Context-Free Languages A B C D... A w 0 w 1 D...E The languagea n b n : a b ab 2 Finite-tate Automata 3 Pushdown Automata Pushdown automata (pda s) is an fsa with

More information

1 Introduction. 2 An Interpreter. 2.1 Handling Source Code

1 Introduction. 2 An Interpreter. 2.1 Handling Source Code 1 Introduction The purpose of this assignment is to write an interpreter for a small subset of the Lisp programming language. The interpreter should be able to perform simple arithmetic and comparisons

More information

CS106A, Stanford Handout #38. Strings and Chars

CS106A, Stanford Handout #38. Strings and Chars CS106A, Stanford Handout #38 Fall, 2004-05 Nick Parlante Strings and Chars The char type (pronounced "car") represents a single character. A char literal value can be written in the code using single quotes

More information