Αυτοματοποιημένη Αναγνώριση και Παράκαμψη Κανόνων Ασφαλείας Εφαρμογών Διαδικτύου

Transcription

1 ΕΘΝΙΚΟ ΚΑΙ ΚΑΠΟΔΙΣΤΡΙΑΚΟ ΠΑΝΕΠΙΣΤΗΜΙΟ ΑΘΗΝΩΝ ΣΧΟΛΗ ΘΕΤΙΚΩΝ ΕΠΙΣΤΗΜΩΝ ΤΜΗΜΑ ΠΛΗΡΟΦΟΡΙΚΗΣ ΚΑΙ ΤΗΛΕΠΙΚΟΙΝΩΝΙΩΝ ΠΤΥΧΙΑΚΗ ΕΡΓΑΣΙΑ Αυτοματοποιημένη Αναγνώριση και Παράκαμψη Κανόνων Ασφαλείας Εφαρμογών Διαδικτύου Ιωάννης Γ. Στάης Επιβλέπων: Αγγελος Κιαγιάς, Επίκουρος Καθηγητής ΕΚΠΑ ΑΘΗΝΑ ΝΟΕΜΒΡΙΟΣ 2011

2 ΠΤΥΧΙΑΚΗ ΕΡΓΑΣΙΑ Αυτοματοποιημένη Αναγνώριση και Παράκαμψη Κανόνων Ασφαλείας Εφαρμογών Διαδικτύου Ιωάννης Γ. Στάης Α.Μ.: ΕΠΙΒΛΕΠΩΝ: Αγγελος Κιαγιάς, Επίκουρος Καθηγητής ΕΚΠΑ

3 ΠΕΡΙΛΗΨΗ Η εργασία αυτή ασχολείται με την αυτοματοποίηση των διαδικασιών αναγνώρισης και παράκαμψης των προγραμμάτων προστασίας εφαρμογών διαδικτύου με την χρήση βασικών εννοιών της θεωρίας υπολογισμού. Τα προγράμματα προστασίας τοποθετούνται εξωτερικά σε μια εφαρμογή διαδικτύου και φιλτράρουν όλα τα εισερχόμενα δεδομένα, πραγματοποιώντας αναγνώριση μοτίβων που σχετίζονται με την ύπαρξη κακόβουλου κώδικα και σε περίπτωση που κάτι εντοπιστεί το απορρίπτουν. Όλα αυτά τα μοτίβα ορίζουν μία γλώσσα η οποία αναγνωρίζεται από το πρόγραμμα προστασίας και απορρίπτεται. Από τις αρχές της θεωρίας υπολογισμού, γνωρίζουμε ότι το συμπλήρωμα αυτής της γλώσσας είναι το σύνολο των αποδεκτών εισόδων από το πρόγραμμα προστασίας. Επίσης γνωρίζουμε ότι μπορούμε να κατασκευάσουμε αυτόματα για την αναγνώριση αυτών των δυο γλωσσών. Γενικεύοντας το πρόβλημα μπορούμε να κατασκευάσουμε το αυτόματο που αναγνωρίζει την γλώσσα που απορρίπτει ένα πρόγραμμα προστασίας ως επικίνδυνη και το αυτόματο που αναγνωρίζει την γλώσσα που ένα άλλο πρόγραμμα προστασίας αποδέχεται ως ασφαλή, και να πραγματοποιήσουμε τη τομή των δύο αυτομάτων, παίρνοντας σαν αποτέλεσμα το αυτόματο που αναγνωρίζει την γλώσσα που αποδέχεται ως ασφαλή το ένα αλλά ως επικίνδυνη το άλλο. Οι είσοδοι που ανήκουν στην γλώσσα αυτή μας επιτρέπουν να ξεχωρίζουμε ποιό από τα παραπάνω προγράμματα χρησιμοποιείται, κάνοντας απλές δοκιμές και ελέγχοντας αν οι είσοδοι θα απορριφθούν ή όχι, αλλά και να αξιοποιούνται για την πραγματοποίηση επιθέσεων. Σε δεύτερη φάση, η εργασία αναφέρεται στην κατασκευή ενός εργαλείου για την παράκαμψη αυτών των προγραμμάτων προστασίας, παραλλάσσοντας τον επιβλαβή κώδικα. Το εργαλείο αυτό χρησιμοποιεί αυτόματα για την αναγνώριση γλωσσών χωρίς συμφραζόμενα, με σκοπό να αναλύσει τη δομή του επιβλαβούς κώδικα και να πραγματοποιήσει τις κατάλληλες επιτρεπτές αλλαγές ΘΕΜΑΤΙΚΗ ΠΕΡΙΟΧΗ: Ασφάλεια Υπολογιστικών Συστημάτων, Θεωρία Υπολογισμού ΛΕΞΕΙΣ ΚΛΕΙΔΙΑ: αυτόματα, τομή, προγράμματα προστασίας, αναγνώριση, παράκαμψη

4 ABSTRACT The essay refers to the automation of the identification and bypass of web application firewalls, using the basic concepts of computation theory in computer security. These web application firewalls filter all the incoming and outgoing data, performing pattern recognition in order to identify and reject malicious code. All these patterns define a language which is recognized by the wafs and discarded. Because of the closure of the regular languages, the complement of the above language is the language accepted by the application, which consists of all the valid inputs. Achieving the intersection between the regular language that intercepts a web application firewall and the complement of another regular language, that accepts another web application firewall, will give us the regular language that the second firewall should probably also not accept. So we can discover which waf from the above a web site uses, by checking if these inputs are accepted or rejected by the site. Moreover, since theses inputs are assumed as malicious by the first waf, they can be used to attack any web application protected by the second waf. In the second phase, the work refers to the construction of a tool that tries to bypass these protection programs, obfuscating the malicious code that user provides. This tool uses push down automata to analyze the structure of malicious code and to make the available transformations SUBJECT AREA: Computer Security, Computation Theory KEYWORDS: automata, intersection, web application firewalls, distinguish, bypass

5 Στους γονείς μου.

6 ΕΥΧΑΡΙΣΤΙΕΣ Θα ήθελα καταρχήν να ευχαριστήσω τον καθηγητή κ. Άγγελο Κιαγιά, που μου έδωσε την ευκαρία να ασχοληθώ με αυτό το θέμα, και την άψογη συνεργασία και καθοδήγησή του. Σημαντικό ρόλο στην εκπόνηση της παρούσας πτυχιακής εργασίας διαδραμάτισε ο υπεύθυνος της πτυχιακής εργασίας και φίλος μου Γιώργος Αργυρός, τον οποίο και ευχαριστώ για την καθοδήγηση του και την εξαιρετική και εποικοδομητική συνεργασία που είχαμε. Θα ήθελα ακόμα να ευχαριστήσω τους γονείς μου Γεώργιο και Μαρία για την καθοδήγηση και την ηθική συμπαράσταση που μου προσέφεραν στα μαθητικά και φοιτητικά μου χρόνια. Τέλος θα ήθελα να ευχαριστήσω την αδερφή μου Ελένη για την συμπαράσταση και τις συμβουλές της για τις απφάσεις μου στην περίοδο αυτή της ζωής μου.

7 ΠΕΡΙΕΧΟΜΕΝΑ PROLOGUE INTRODUCTION CyberSecurity Protection Threats Target of Theses Chapters: Organisation and Classification FUNDAMENTALS OF COMPUTATION THEORY Languages Grammars Regular Grammars Regular Expressions Operations on Languages Automata Finite State Automata Closure Deterministic Finite Automata and Non-Deterministic Finite Automata Converting NFA to DFA Subset Construction Algorithm Minimize Deterministic Finite Automata Classical Minimization Algorithm Hopcroft Algorithm Automata Complement Automata Intersection Cartesian Product Method Optimal Intersection Algorithm Converting Regular Expressions to Automata Thompson Construction Algorithm Converting Automata to Regular Expressions Converting Deterministic Finite Automata to Regular Expressions Transitive Closure Method State Removal Method

8 Brzozoswki Algebraic Method WEP APPLICATION ATTACKS AND COUNTERMEASURES Attacks SQL Injections Cross Site Scripting Cross Site Request Forgery Remote File Inclusion Local File Inclusion Web Application Firewalls IMPLEMENTATION From Concept To Implementation Tools for producing Deterministic Finite State Automata Flex Transforming Web Application Firewalls rulesets Intersection Algorithm Transition Function of Flex Automaton Checking Acceptance of Flex Automaton Complement of Flex Automaton Mapping Pairs of States Deterministic Finite Automata and States Graph Traversing Regular Expressions from Automata Heuristic Functions Replace Large Classes of Characters Accept Only Shortest Kleene Star Accept Only Shortest Regular Expressions Prevent Loops in Same State Heuristics Combination WEB SERVICE Presentation MALICIOUS CODE OBFUSCATION

9 6.1 Syntax Obfuscation Lexical Obfuscation Errors Exploitation FUNDAMENTALS OF PARSING Grammars Context-Free Grammars Push-Down Automata Context-Free Grammar to Push Down Automata IMPLEMENTATION OF AUTOMATED BYPASS Input Parsing Bison Parser Input Obfuscation Heuristic Functions CONCLUSION Project Breakthroughs Promising Results Future Work and Open Problems ΠΙΝΑΚΑΣ ΟΡΟΛΟΓΙΑΣ ΣΥΝΤΜΗΣΕΙΣ ΑΡΚΤΙΚΟΛΕΞΑ ΑΚΡΩΝΥΜΙΑ ΑΝΑΦΟΡΕΣ

10 ΚΑΤΑΛΟΓΟΣ ΕΙΚΟΝΩΝ Εικόνα 1: Automata Complement: Initial Εικόνα 2: Automata Complement: Final Εικόνα 3: Cartesian Product Method Εικόνα 4: State Removal Method: Initial Εικόνα 5: State Removal Method: First Step Εικόνα 6: State Removal Method: Second Step Εικόνα 7: Brzozoswki Algebraic Method Εικόνα 8: Attacks Εικόνα 9: OSI Layers and WAF Εικόνα 10: WAF2-WAF Εικόνα 11: WAF1-WAF Εικόνα 12: Brzozowski Algebraic method Εικόνα 13: Intersection Results Graph Εικόνα 14: Intersection Results Graph Εικόνα 15: Intersection Results Graph Εικόνα 16: Intersection Results Graph Εικόνα 17: Intersection Results Graph Εικόνα 18: Intersection Results Graph Εικόνα 19: Web Service Presentation Εικόνα 20: Web Service Presentation Εικόνα 21: Web Service Presentation Εικόνα 22: Web Service Presentation Εικόνα 23: Web Service Presentation Εικόνα 24: Web Service Presentation Εικόνα 25: Web Service Presentation

11 ΚΑΤΑΛΟΓΟΣ ΠΙΝΑΚΩΝ Πίνακας 1: Thompson Construction Algorithm Πίνακας 2: Web Application Firewall Πίνακας 3: Web Application Firewall Πίνακας 4: Intersection Expected Results Πίνακας 5: Intersection Expected Results Πίνακας 6: Intersection Results Πίνακας 7: Intersection Results Πίνακας 8: Intersection Results Πίνακας 9: Intersection Results Πίνακας 10:Intersection Results Πίνακας 11:Intersection Results Πίνακας 12:Intersection Results Πίνακας 13:Intersection Results Πίνακας 14:Intersection Results Πίνακας 15:Intersection Results Πίνακας 16:Intersection Results Πίνακας 17:Intersection Results

12 PROLOGUE I completed my undergraduate thesis during the summer of 2011 in Athens. My thesis involved the creation of a web application service that would be capable of making the intersection of the language that accepts a web application firewall with the language that rejects another web application firewall, and the creation of another syntax based obfuscating application. The aim of the research project was to be able to distinguish which firewall a website uses, to evaluate it according to its ruleset, and to try to bypass it using obfuscating malicious code.

13 1. INTRODUCTION 1.1 CyberSecurity Nowadays computer scientists and ethical hackers battle to prevent an information security apocalypse. Hackers are constantly experimenting with a wide variety of techniques to penetrate into websites and web applications. These are the most frequent victims of attacks by malicious users since they are the front door for making further more complicated attacks under the local machine zone, allowing the acquisition of the system, and of valuable informations or even the transformation of the system into a zombie machine. It is obvious the great importance of better security protection on such applications as they are the forefront of the whole system. The web application security is a branch of information security that deals specifically with security of websites and web applications. The existence of billions of web applications, which use more than one web technologies and are based in different implementations, leads to the existence of a variety of different ways to confront these attacks. These protections are the main target of hackers, who are trying to distinguish and bypass them. 1.2 Protection Web applications are carrying out security checks on all their inputs and outputs, which is the only way an attacker can interact with the specific application without having further access to the server. Since the inputs and outputs are strings, these checks are accomplished using pattern recognition in those strings, filtering the incoming characters. There are two different approaches to carrying out these checks, the white-listing and the blacklisting. White-list is a term used to describe a list of entities that are accepted from the web application as valid. On the contrary, blacklist is a list of entities that are unaccepted from the web application as potential attacks. Without loss of generality the report will focus on blacklisting, which appears more frequently, because it can be applied over all inputs and outputs omitting the obvious limitations created by the use of white-listing. Web developers are likely to create a specific sanitize function to do their respective checks and to implement it to every input and output that they consider as a vulnerable one. They can also use a web application firewall, which undertakes to control all the inputs and outputs and also the access from, to, or by a web application or service. This is an appliance, server plug-in, or filter that operates by monitoring and potentially blocking the incoming and outgoing elements to an http conversation which do not meet the policy of the firewall that is configured by an appropriate set of rules. In both sanitize functions and web application firewalls, these security filters reject specific inputs, which are part of a language, that the web application wants to reject as a potential attack. Ιωάννης Γ. Στάης 13

14 1.3 Threats Generally, these rules cover common attacks such as Cross Site Scripting, SQL Injection,Cross Site Request Forgery, Remote File Inclusion, Local File Inclusion, Denial of Service, Protocol Violation and Anomalies, Discovering Uploaded Trojans and other inbound and outbound activities that are considered as malicious by the web application. 1.4 Target of Theses Having access to a wide variety of open source web applications and web application firewalls, in the first part we will try to create an application capable of: Identifying the fingerprints of a web application firewall or of a specific sanitize function used in a web application Discovering security holes in sanitize functions in comparison with known web application firewalls Finding security vulnerabilities in web application firewalls through the comparison between them The concept of the project is based in the principles of computation theory. Alot of open source projects, give us the ability to access the security checks and convert them into regular expressions, defining the specific regular language that the web application reject. Also, because of the closure of the regular languages, the complement of the above language is the language accepted by the application, which consists of all the valid inputs. Achieving the intersection between the regular language that intercepts a web application firewall and the complement of another regular language, that accepts another web application firewall, will give us the regular language that the second firewall should probably also not accept. This language consists of inputs that are intercepted as attacks by the first web application firewall and are valid and safe according to the second web application firewall. So we can discover which waf from the above a web site uses, by checking if these inputs are accepted or rejected by the site. Moreover, since theses inputs are assumed as malicious by the first waf, they can be used to attack any web application protected by the second waf. In the second part, we will try to create an application that will: Obfuscate the input in order to make the attacking source code difficult to understand Bypass the waf s security checks The target of the tool will be to produce strings based on user input, that can bypass the waf s security checks. This can be achieved using the latest obfuscation techniques combined with the knowledge of the waf s rejected language. Ιωάννης Γ. Στάης 14

15 1.5 Chapters: Organisation and Classification The chapters are classified as follows: The second chapter provides the fundamentals of computation theory that are needed to continue with the construction of the solution. The third chapter contains an introduction to web attacks and countermeasures, and a wide range of popular techniques. The forth chapter consists of the implementation details of the above ideas and descibes the creation of the web application. The fifth chapter presents the available features of the web application. The sixth part describes common obfuscation techniques. The seventh chapter provides more details over computation theory that are needed for the second part of the concept. The eigth part describes the implementation of the application that tries to bypass the wafs. The last part contains the conlusion results about the project. In the last part, there is a total evaluation of the above work, the prommising results and future possibilities of the project. Ιωάννης Γ. Στάης 15

16 2. FUNDAMENTALS OF COMPUTATION THEORY 2.1 Languages A language contains certain words(finite strings of letters, symbols, or tokens). The set from which these letters are taken is the alphabet over which the language is defined.the symbol Σ is for the alphabet and the symbol Σ for all the possible words of that alphabet. 2.2 Grammars A grammar is a set of formation rules for strings in a language, which describe how to form strings from the language s alphabet that are contained in the language and is usually thought of as a language generator. A grammar does not describe the meaning of the strings or what can be done with them. The rules consist of starting, non-terminal, terminal or empty symbols. These rules are then applied in any order, until a string that contains neither the start symbol nor designated non-terminal symbols is produced Regular Grammars If the left hand side is a single non-terminal symbol and the right side is empty, or a single terminal symbol, or a single terminal symbol followed by a non-terminal symbol then the grammar named regular grammar and the language a regular language Regular Expressions A regular expression, is a pattern that describes a set of strings without having to list all the strings. Regular expression are equivalent to regular grammar and capable of describing a regular language with a simply way Operations on Languages Union: The union of one or more languages consists of all the strings that are contained in one or more of the these languages. Concatenation: The concatenation consists of all the strings of the form σ 1 σ 2 where σ 1 is a string from one language and σ 2 is a string from another. Complement: The complement consists of all strings formed from the alphabet of the language, but not contained in it. Intersection: The intersection of two languages consists of all the strings that are in both languages. Difference: The difference A B of two languages A and B, consists of all the strings that are contained in the language A but not in B. Symmetric Difference: The symmetric difference of two languages A and B, consists Ιωάννης Γ. Στάης 16

17 of all the strings that are a member of exactly one of A and B. Cartesian product: The cartesian product of two languages A and B is the language whose members are all possible ordered pairs (a, b) where a is a member of A and b is a member of B. Kleene Star: The concatenations of zero or more words of the original language. 2.3 Automata Automata are abstract mathematical machines or systems that can solve computation problems. Automaton means something that is doing something by itself and is used as a finite representation of a formal language that may be an infinite set Finite State Automata The theoretical model for the recognition of a language is an automaton. Generally it allows us to identify in a practical manner whether a string belongs to the strings of a language or not. It consists of states and transitions. For every input, it makes a transition from it s current state to another state(it can be one or more available to move), according to its transition function. The languages recognized by finite automata are called regular languages. For different definitions of automata, the recognizable languages are different Closure The class of languages accepted by finite automata is closed under: Union Concatenation Kleene star Complement Intersection Deterministic Finite Automata and Non-Deterministic Finite Automata The finite automata are defined by five T = (R, S, D, S, F): K: Finite set of states Σ: Finite input alphabet δ: transition function from a state Q to a Q based on an input symbol σ Σ: δ(q, σ) = Q s: Start condition F: Total final states If the function δ corresponds to exactly one state then automatically called deterministic finite automaton. Conversely, if for some values (Q, σ) of δ there are many possible Ιωάννης Γ. Στάης 17

18 new states Q then automatically called non-deterministic finite automaton. Also, nondeterministic finite automata can have empty transitions which are labeled as ε transitions Converting NFA to DFA NFA s can be converted to DFA s. The max states of the corresponding automaton will be 2 n where n are the states of the NFA Subset Construction Algorithm Consider that we have the non deterministic automaton M = (K, Σ, δ, s, f). We expand all the transitions in order to contain only one symbol each in M = (K, Σ, δ, s, f ). We want to convert it in the deterministic M = (K, Σ, δ, s, f ). First, we search and remove from the automaton the group of states containing transitions that are not accepted by a deterministic automaton. Such transitions are those who are ε. We can achieve this using Depth First Search algorithm for all the ε transitions. For a given state s define the ε-closure(s) to be the set of states reachable from s using only ε-transitions. the new start state will be: s = ε closure(s ) = q(s, ε) M (q, ε) For a given set of states K, define ε-closure(k), as the set of states reachable from any state in K using only ε-transitions. the new transition function will be: δ (Q, σ) = ε closure(δ (Q, σ)) = q (q, σ) M (q, ε) In the second part, we use the accepted transitions of Σ to find new accepted transitions for the groups mentioned before. We may also create new states for these transitions. To do the mapping, we use the transition function δ and check the triplets using for starting state every state of the not accepted groups and for every symbol of Σ checking the end state. We repeat the procedure described before for all the inserted new states using a list. if the mapped states already exist, we skip them Minimize Deterministic Finite Automata A M DFA is called minimal if there is no other M DFA, where L(M) = L(M ), and M has fewer states. Yingjie XU in his work presents the following two most common algorithms for DFA minimization Classical Minimization Algorithm According to Yingjie XU s work, Given a DFA D = (Q, Σ, δ, s, F), two states q1, q2 Q are said to be equivalent if for every w Σ?, δ(q1, w) F δ(q2, w) F. Two states that are not equivalent are called distinguishable. The key idea of the algorithm is to find all equivalence class of the indistinguishability Ιωάννης Γ. Στάης 18

19 relation and join all states in each class into one state. Of course all the unreachable states must be previously removed. The first step of the algorithm is to mark all pairs p,q, where p F and qf.then, for each unmarked pair, check for each σ Σ if the new pair (δ(q, σ), δ(p, σ)) is marked, and in case it is true, then mark the previous pair too. The running time of the algorithm is obviously O(n 2 ) Hopcroft Algorithm Begin by partitioning the states of the DFA into two groups: the final states and the nonfinal states. For every group from the partition and for each symbol σ Σ get the set of end states for any existing transition from every state of that group using σ. If that set is a subset of one of the members of the partition,then we don t do nothing else that group must be further partitioned into subgroups so that all the states in each of the subgroups have end states to the same subgroup on every incoming symbol. Consider U = {U1,.., Um}(1 m Q ) the classes created. Then U = i=1m Ui Q log Q so the running time of the algorithm is O(nlogn) Automata Complement The complement is achieved simply by converting all the final states into non-final, and all the non-final into final. start q. 1 q 2 Εικόνα 1: Automata Complement: Initial start q. 1 q 2 Εικόνα 2: Automata Complement: Final Automata Intersection If M 1 and M 2 are FSAs, then there is an FSA that accepts L(M 1 ) L(M 2 ). The proof of the above theorem can be achieved by using the classic cartesian product method Cartesian Product Method The intersection between two finite automata, can be accomplished easily by constructing pairs of states. Consider two automata M 1 and M 2. We can construct the intersection Ιωάννης Γ. Στάης 19

20 automaton M, accepting the language L(M 1 ) L(M 2 ). Let M 1 = (k1, Σ, δ1, s1, F1) and M 2 = (k2, Σ, δ2, s2, F2) deterministic finite automata equivalent to the given automata M 1 and M 2 respectively. We simulate this parallel operation of automata M 1 and M 2 in order to achieve the acceptance of strings accepted by both the M 1 and the M 2. For each symbol input, will execute the corresponding transition for the M 1, but also for Εικόνα 3: Cartesian Product Method the M 2, using for states for the automaton M, pairs of the form < q1, q2 > where q1 k1 and q2 k2. To achieve the parallel operation of M 1 and M 2, we use the function for state transition: δ(< q1, q2 >, σ) = (δ1(q1, σ), δ2(q2, σ)). Similarly, the initial state of the automaton will be < s1, s2 >, and the final states, in order to be accepted by both the M 1 and M 2 will be: F = {< q1, q2 > q1 F1 and q2 F2} Overall, we construct the automaton: M = (k, Σ, δ, s, F), where: k = {< q1, q2 > q1 k1 and q2 k2} Σ : the alphabet accepted of the automata M 1 and M 2. δ : δ(< q1, q2 >, σ) = (δ1(q1, σ), δ2(q2, σ)) where q1 k1, q2 k2, and σ Σ. s =< s1, s2 > F = {< q1, q2 > q1 F1 and q2 F2} Optimal Intersection Algorithm Consider the above two automata M 1 and M 2 and let M 1 be the number of states of M 1 and M 2 be the number of states of M 2. The FSM M, accepting the language L(M 1 ) L(M 2 ) will have at most M 1 M 2 states. Ιωάννης Γ. Στάης 20

21 So the complexity is O( M 1 M 2 ) The previous algorithm is decades old but we still can t proove that algorithm result is optimal. Does a better intersection algorithm exist? George Karakostas and Anastasios Viglas in one of their previous related works support that there is no reason to believe this. They focus on the corollary problem: If M 1 and M 2 are FSAs, then there is an algorithm that runs in time O( M 1 M 2 ) and decides whether or not L(M 1 ) L(M 2 ) is empty. They try to find a better algorithm that solves the emptiness problem for L(M 1 ) L(M 2 ) in linear time given that there are FSAs M 1 and M 2 so that the smallest number of states of a machine M, where L(M) = L(M 1 ) L(M 2 ), is the product of the number of states of M 1 and M 2. Subsequently, this could only happened if the new algorithm could avoid the step of building the product automata, which is the main problem since it is not possible to restrict this step. So the proof that there is no better intersection algorithm, could be based either in proving the assumption that it is unlikely to exist any other algorithm capable of beating the Cartesian product algorithm, or approaching the problem by studying the consequences of that existence in the computation theory. In their work, they followed the second way and prove that the assumption that there exists a better algorithm solving the FSA intersection emptiness problem implies that nondeterministic time is in sub-exponential deterministic time and also separates NL from P, which under a non-uniform variant of the assumption mentioned above implies that NL NP. Karakostas and Viglas assumed the Product Emptiness Problem (PEP): Suppose that M 1,..., M k are FSA each with at most n states. We what would the consequences be if there was an algorithm that ran in time n o(k) and determined whether or not there was an input they all accepted. where k should be fixed otherwise according to a previous proof of Dexter Kozen, in this case the problem is PSPACE complete. There is also the option to allow algorithms that run in time 2 k n k But based the previous assumption, and considering k as fixed, the following theorems can be prooved: There is an algorithm solving knapsack in O(2 εn ) for all ε > 0 Integer factorization can also be solved in O(2 εn ) for all ε > 0 NTIME(t) DTIME(2 εt ) for all ε > 0 Also a slight modification of assumption, allows the separation of NL from P The cartesian product method is obviously the best-known intersection algorithm, even if we can not proove if it is optimal. Ιωάννης Γ. Στάης 21

22 Converting Regular Expressions to Automata Since we are able to convert a NFA to a DFA automaton with the algorithm mentioned previously, we will focus on a way converting regular expressions to NFA, which is easier to implement because of the use of ε move and the multiple edges for the same symbols Thompson Construction Algorithm Thompson s algorithm permits the conversion of Regular Expressions to Non Deterministic Finite Automata. The concept of the algorithm is the creation of an NFA pattern for each symbol and each operator and the association of them with ε moves maintaining the priority, in order to have a final finite automaton with one final state and one start state. Guided by the syntactic structure of the regular expression we develop partial components for its sub-expressions: The NFA for the entire regular expression definition is a combination Symbol Alternation Kleene Star Kleene Plus Concatenation has one new start and one new final state connected with a transition that is labelled with that symbol. has a new start state and a new final state which are connected with ε transitions to the automata of the sub-expressions. has a new start state and a new final state connected with an ε transition. Also the automaton of the repeated sub-expression is connected with the new start and the new final state( the start state with the start state of the automaton of sub-expression and the final state of the automaton of sub-expression with the final state). Finally there is an ε transition from the final state of the automaton of subexpression to the start state of the automaton of sub-expression for the loops. has a new start state and a new final state which on the contrary with the example above, they are not connected. The automaton of the repeated sub-expression is connected with the new start and the new final state( the start state with the start state of the automaton of sub-expression and the final state of the automaton of sub-expression with the final state) and there is an ε transition from the final state of the automaton of sub-expression to the start state of the automaton of sub-expression for the loops. the start state is the automaton of the first expression and the final state is the automaton of the second expression connected with an ε. Πίνακας 1: Thompson Construction Algorithm from the above intermediate partial components. It uses the recursive definitions of regular expressions to build a finite automaton for a compound expression from the finite automata for the base expressions. Ιωάννης Γ. Στάης 22

23 Converting Automata to Regular Expressions Because of the equality of DFA and NFA, we will focus on a way converting a DFA to regular expressions, which is easier to implement Converting Deterministic Finite Automata to Regular Expressions We can also get regular expressions from automata. The Kleene proved the equality of regular expressions and finite automata, providing us with the first method for the conversion of automata to regular expressions (transitive closure method). Later, Janusz A. Brzozowski [4] extended the method of Kleene inserting the concept of derivatives of regular expressions. The Ding-Shu Du and Ker-I Ko [3] also showed another removal method of states (State removal method) which was later improved by Peter Linz [6]. Christoph Neumann[1] had analyse the most popular ways mentioned above: Transitive Closure Method Consider a state q i and a state q j. The path of the transition from q i to q j is the concatenation of the regular expressions that correspond to the transitions of each intermediate state. Nevertheless, in advanced automata, there is not only one transition from the state q i to q j. But using the union and the repetition we can solve the above problem. Let R ij be the sum of the regular expressions that produce strings that cause the transition from state q i to state q j, without pass over state q k during the intermediate transitions. We can construct the R ij by constructing all the R 1 ij, R 2 ij...r m ij, where: R k ij = R k 1 ik (R k 1 kk ) Rk 1 kj + R k 1 ij, assuming that: R 0 ij = r, for i! = j and r transitions from qi to qj, R 0 ij = r + l, for i == j and r transitions from qi to qj, R 0 ij = empty, other So we can construct all regular expressions as a union of R f where q is the initial state and f M f (final states of the automaton). The problem with this algorithm is that it creates very large regular expressions State Removal Method This state removal approach recognizes patterns within the graph and removes them, constructing regular expressions along each transition. The advantage of this technique against the Transitive Closure method is that it is easier to visualize. First, the different possible transitions are converted to a single transition that contains the union of these. Ιωάννης Γ. Στάης 23

24 i q 2 n q 3 start q. 1 t q 4 o q 5 \( q 6 \s Εικόνα 4: State Removal Method: Initial Suppose that the from state q 1 to q 2 there is one transition i and from state q 2 to q 3 there is one transition n. Such a move will be a direct transition from q 1 to q 3 in the form in. Then, in each possible sub-graph of the original graph, we remove all the possible states, and replace them with the corresponding transitions in their concatenation into one transition. In case that a transition that corresponds to a state that is subtracted, is a loop, it is replaced with a Kleene star symbol. in q 2 q 3 start q. 1 q 4 to(\s)*\( q 5 q 6 Εικόνα 5: State Removal Method: First Step start q. in (to(\s)*\() 1 q 9 Εικόνα 6: State Removal Method: Second Step This method is excellent to use in an FSA, which is visualized as a graph, but it is not so practical to implement in a program. Ιωάννης Γ. Στάης 24

25 Brzozoswki Algebraic Method The Brzozowski method [4] which was presented by G. Berry and R. Sethi [2] has a unique approach in creating regular expressions. It creates a system of regular expressions with an unknown regular expression for each state in M, and then solve the system R l where R l is the regular expression corresponding to the starting condition q l. These equations are the characteristics equotations of M. The construction of the characteristic equation is simple: For each state q i in M, the equation for R i is a union of all conditions. Each term can be constructed in this way: for a transition from q i to q j, the term is ar J. Also: whether the term R i is a final state, a λ is added as being one of the arguments. This leads to a system of equations in the form: R 1 = a 1 R 1 + a 2 R R 2 = a 1 R 1 + a 2 R R 3 = a 1 R 1 + a 2 R λ... = = =... R m = a 1 R 1 + a 2 R λ where ax = blank if there is no transition from R i to R j. The system can be solved by simple replacement, except in the case where an unknown argument appears on the right and left side of the equation. This situation occurs when there is a loop in the state q i. Arden s theorem is the key to resolving these situations. The theorem is as follows: Given an equation of the form X = AX + B where l does not belong to A, the equation has solution x = A B Using the above equation to isolate the R i in the left member of equation and then replacing some of the other characteristic equations is the solution to this issue. We continue this procedure until you eliminate the unknowns from the right members of equation. For example, for the FSA: The characteristic equations are: q 1 = iq 2 q 2 = nq 3 q 3 = nq 3 + \sq 4 q 4 = lq 5 + \sq 4 q 5 = iq 6 q 6 = kq 7 q 7 = eq 8 q 8 = λ Ιωάννης Γ. Στάης 25

26 start q. i n 1 q 2 q 3 n \s q 4 l q 5 \s i q 6 k q 7 q 8 e Εικόνα 7: Brzozoswki Algebraic Method From q 5, q 6, q 7 and q 8 we conclude that q 5 = ike. We solve the q 4 using the theorem of Arden and as mentioned above, it will be: q 4 = lq 5 + \sq 4 q 4 = like + \sq 4 q 4 = (\s) like Then replace it in q 3 and we have: q 3 = nq 3 + \s(\s) like using the theorem of Arden again: q 3 = (n) \s(\s) like Then, from q 1, q 2 and q 3 we conclude: q 1 = in(n) \s(\s) like This method is quite simple to implement, and in contrast to the transitive closure is great for languages without creating large regular expressions. Ιωάννης Γ. Στάης 26

27 3. WEP APPLICATION ATTACKS AND COUNTERMEASURES 3.1 Attacks There are many different types of attacks that are used by hackers in the wild, such as Cross Site Scripting,SQL Injection,Cross Site Request Forgery, Remote File Inclusion, Local File Inclusion, Denial of Service, Bruteforce attacks, Exploiting Protocol Anomalies, Uploading Trojans and other attacks on inbound and outbound activities. Many of them are caused by naive construction of the web sites since developers do not consistently employ secure coding practices. The main target of the attackers, is to execute their own malicious code either on the server or on the connected clients. The only way that they have in order to interact with the web application, are its inputs and outputs. Moreover, their malicious code, must belong both in the language that accepts the site, the language that doesn t rejects the waf that the site uses, and the language that is accepted by the server or the client browser for execution. The most common attacks that the majority of Εικόνα 8: Attacks the web applications are trying to protect from, are the following: SQL Injections Most websites contain an online database where they can store informations about users, products and services. Only the web application has access on this database in order to save and recover these informations which are defined by web users needs. So the web application is responsible for responding to the users demands by transfering them the correct records from the database or adding new records to the database. The communication between the database and the web application is made using queries. These queries are commands that have a specific vocabulary and syntax which define the database language. In order to read and respond to users demands, the web pages take parameters and use them to configure the queries that will recover from the database the corresponding information. For instance if a user wants to search for a specific product, he will send Ιωάννης Γ. Στάης 27

28 the name of the product to the web application and it will execute a query based on that name in the products table of the database. Then it will return the results to the user. Other times, it may obtain records from the database based on some parametres only to analyse them and use them for configuring it s behavior without presenting them to the user. For example, when a user login, the web application gets that user name and password and makes a query to the database to check if a user has valid name and password. A query like this would be: SELECT * FROM users WHERE username = given name In this case, the queries are created naively on the fly by the web application. An evil user could exploit the fact that is able to alter the queries and send a crafted user name and password field that will change the SQL query and thus grant as a result something else. For the previous instance, attacker could give as the name parametre or 1 = 1 altering the query: SELECT * FROM users WHERE username = or 1 = 1 which will return a valid result although the user did not give a correct and registered name. If the website has any kind of protection against attacks, the parameters that the user gives would be checked for sql injections, The final parameters that are accepted by the web application belong to the language that it accepts. In order the attack to be successful the combination of the initial query and the parameters must also be accepted by the database as a valid query. In other words, it must belongs to the language that the database accepts. Today these injection have been widely written and used in the wild. SQL injections give the attackers the ability to access the online database and all the confidential informations that it contains. Moreover if the users accounts informations are saved in that database without being encrypted or if they are encrypted with an algorithm that is no longer secure, like md5 hashing algorithm, attackers can easily steal them. Also if an account with privileges is one of them, then the whole site is vulnerable for further attack Cross Site Scripting The evolution of the internet technologies with the introduction of web 2.0, created a whole new level of attacks in web applications. Until now, web sites generated only static pages and the servers were able to have full control over how the browser interprets these pages. Nowadays, web sites use dynamic pages which permit clients to have a better interaction but also more capabilities in the interpretation of the web pages. More and more dynamic pages let users add their own informations, or alter them based on their demands. For example, forums and blogs let users add their opinion, search pages present results based on users demands, and many other web applications let users personalize the pages. Attackers use these inputs for embedding malicious code and get a user s browser to execute this code. This is achieved by adding the malicious code as informations in a Ιωάννης Γ. Στάης 28

29 dynamic page, which will be later visited by another user. The user is waiting to read real informations, but his browser will be tricked to execute the evil code. Usually the code itself is written in Html, Javascript, Cascading Style Sheets, Document Object Model, VBScript, ActiveX, Java, Flash, or any other browser-supported technology. The problem is that if the malicious code has been injected in a page, neither the server nor the client has enough information to distinguish the injection from the real web page. The attack targets on the trust that a user has for a particular site. The injected code, since it is executed from the victim s browser, has the same privileges and access rights with his browser. The trust relationship between an authenticated user and a web application is compromised. For instance, if a user is authenticated in a website, then the malicious code that is embedded in one of the pages of the website, will also have access to the authenticated session and will be able to send these credentials(cookies) back to the attacker, having his account hijacked. Of course it can also redirect the victim s browser or change the contents of the page. There are many different ways of implementing a cross site scripting attack. The first way is the persistent attack, where the attacker adds malicious code in a permanent place in one of the pages on the server. Every time that a user visits that page, he also downloads the malicious code, without further integration with the attacker. For example, a forum site saves the user s posts in the database and recovers them every time that a user visit the forum. An attacker could post its malicious payload and let the web site save it permanent in the database. Every user that visits the forum will also get the payload from the database. Another way is the non-persistent xss, where the malicious code is not saved in the database, is temporary used and print in the web page by the server. It may be saved as a url parameter, a cookie, a post type variable or any type of variable used in the specific protocol. For example there are websites that have search pages that alter the result based on the search variable which is sent using one of the above ways and print the variable somewhere in the page. Also the web sites may offer a more personalized view using informations temporary saved in that way. Since the malicious code is not stored in the web application, the victim need to be tricked to follow a crafted link containing the malicious code as a parameter in case of an xss in a url parameter, or to send a form in case of an xss in a post variable with a little help of social engineering in order to deliver the payload to the web page. Of course these attacks can be obfuscated and executed without the victim s knowledge. The last way of implementing a cross site scripting attack looks like the non-persistent method, but without the server s interaction. In this way the attacker applies a runtime embedding of the malicious code in the page using the DOM parts. For instance such parts are the username and password part in the authorization header, the referrer object, in the referrer header, the query path of the url object and the fragment part of the url object, which is not even sent by the server. Generally the DOM attacks target on using Ιωάννης Γ. Στάης 29

30 raw non-html data for passing the malicious code, bypassing the server processing Cross Site Request Forgery In this type of attack, the victim is tricked to execute an unwanted action. Since the action is made by the user or the user s browser, the action has the same privileges with the user, according to the same origin policy that the web browsers follow. If the user is authenticated by a web application, then the action in that web application will be executed within the rights of the user. For example, an administrator could be tricked by an attacker to add the attacker as administrator too. Same to non-persistent xss, the victim again need to be tricked to follow a certain link, or to send a form with a little help of social engineering and again these attacks can be obfuscated and executed without the victim s knowledge. The difference with the nonpersistent xss is that in this attack there is no malicious code and none payload that needs to be delivered and executed to the victims browser. In contrast, in the CSRF the attacker wants to execute a malicious request, and the victim needs only to be tricked to make this request, exploiting the trust that a site has in a user s browser. The attack is based in the policy of same origin that the browsers implement, according to which, browsers automatically include in web sites requests and upload any credentials associated with the site on the same domain. Also it is based in the fact that many web application accept users action without making any dynamic check on the requests, but relying on a simple authentication check of the session. There are also ways of transforming a CSRF into a payload. This can help the attackers store the attack in a web page Remote File Inclusion Modern web technologies support file inclusion. File inclusion permits a source code file to encompass another source code file which contains code that may be executed by calling specific procedures. It is mainly used for packaging common code into separate files. Complicated web sites contain many web pages that share common code parts using file inclusion. For example all the web pages may include the same code for the header and the footer. Attackers exploit this characteristic and include remote custom files which contain malicious code. This attack is achieved only if the selection of the modules that will be included, are able to be altered by evil users. For example, if the name of a file that contains the header source code, is obtained by an url parametre or generally any input accessible by the user(post variables, cookies, etc) then the user can replace the name with an external path to another malicious file. Using the RFI attack an evil user can achieve running malicious code on the server since any code in the included malicious files will be executed by the server and this could lead to a complete system compromise. Also it can achieve running malicious code on clients Ιωάννης Γ. Στάης 30