Checkpointing-Based Testing

Transcription

1 Master Thesis Institute of Information Security, Department of Computer Science, ETH Zurich Checkpointing-Based Testing by Tristan Buchs Autumn Spring 2015 ETH student ID: address: [email protected] Supervisor: Prof. Dr. David Basin Advisors: Dr. Mohammad Torabi Dashti Marco Guarnieri Petar Tsankov Date of submission: April 2, 2015

2

3 Declaration of Originality I hereby declare that this written work I have submitted is original work which I alone have authored and which is written in my own words. With the signature I declare that I have been informed regarding normal academic citation rules and that I have read and understood the information on Citation etiquette : https: // leistungskontrollen/plagiarism-citationetiquette.pdf The citation conventions usual to the discipline in question here have been respected. This written work may be tested electronically for plagiarism. Zurich, April 2, 2015 Signature: Tristan Buchs a a See printed declaration iii

4

5 Abstract Testing web applications is challenging as they usually have deep and large state spaces and very long execution traces. We develop Checkpointing-Based Testing, a novel approach for efficiently testing web applications. Our approach is based on the idea that many test cases share common prefixes and we can significantly reduce the test execution time by avoiding the re-execution of the common prefixes. We implement our approach in a system called WebCheck and empirically evaluate it through case studies over three real-world PHP web applications. Our results demonstrate that WebCheck can achieve up to 19x speedup in test execution time and also detects 10 previously unknown errors. v

6

7 Acknowledgements I would like to express my sincere gratitude to my mentor, Prof. Dr. David Basin, for giving me the opportunity to write my thesis in his research group. I would like to thank my supervisors Marco Guarnieri and Petar Tsankov for their valuable feedbacks and continuous guidance during the whole duration of this thesis. I would also like to thank Dr. Mohammad Torabi Dashti for his insightful suggestions on the subject. Finally, I thank all my family and friends who unconditionally supported me during my studies at ETH, especially Armand Kurum for proofreading my work. vii

8

9 Contents 1 Introduction 1 2 Related Work Application Checkpointing Web Application Testing Fuzz testing Web Applications Technical Background Web Applications Context Structure Storage Stateful Web Applications Web Application Frameworks Web Application Security Web Application Testing Context Approaches Tools Fuzz Testing Checkpointing and Databases Checkpointing Databases Prerequisites Definitions Stateful Web Applications Test Cases Test Suites Snapshot of web application Design Checkpointing Goal Web Application State Checkpoint Methods ix

10 Contents Solution WebCheck Goal Fuzzing Framework Fuzz Operators Test Cases Oracles Implementation Checkpointing Target Applications WebCheck Proxy Application-level Instrumentation Complexity WebCheck Fuzzer Input Parsing Fuzzer Fuzz Operators Oracles Empirical Evaluation Hypothesis Experimental Setup Evaluation of H Evaluation of H First Set of Experiments Second Set of Experiments Evaluation of H Design of Experiments Results & Discussion Conclusion 55 Bibliography 57 x

11 1 Introduction Web applications are nowadays ubiquitous and handle all sort of sensitive data, from banking information to health related data. Their correctness is, therefore, of utmost importance. Testing is, in practice, the most used technique for gaining confidence about a web application s correctness. It consists of exercising the web application with a finite number of inputs (which are typically sequences of HTTP requests) to check whether its behavior conforms to its specification. However, testing is in general incomplete as most web applications accept infinitely many inputs. Therefore, web applications must be thoroughly tested with a large number of inputs to gain a sufficient level of confidence in their correctness. Web applications accept HTTP requests sent by the clients and reply with HTTP responses containing the requested content (e.g. a web page). They typically process long sequences of such events. For instance, to buy a book on Amazon, the client has to (1) provide credentials to authenticate to Amazon, (2) select the book to purchase and add it to the shopping cart, (3) go through all the checkout steps such as entering the shipping and payment information, and, finally, (4) pay for the book. This interaction consists of a large number of HTTP requests and HTTP responses, circa 60, which are individually processed by the web application as separated events. The event-driven nature of web applications results in deep and large state spaces and very long execution traces. These aspects pose serious limitations to web application testing. First, due to the large state space, a tester must execute a large number of test cases to gain any reasonable confidence in the software. Second, due to the long execution traces, executing a single test case may take a long time. For instance, testing Amazon s payment phase requires to first execute the login, the book selection, and the checkout phases, as they must be executed before the payment. Executing even a single test case can therefore be quite expensive. Furthermore, to thoroughly test the payment phase, the tester must check how the application handles different kinds of payment requests, such as payments with insufficient funds or with missing payment information. Each payment request results in a separate test case for which the login procedure, the book selection procedure, and the checkout procedure are always executed at the beginning. In this thesis, we develop Checkpointing-Based Testing, a novel approach for efficiently testing web applications. Our key insight is that many test cases share common prefixes and we can significantly reduce the test execution time by avoiding 1

12 1 Introduction the re-execution of common prefixes. In contrast to all state-of-the-art testing techniques, which execute all requests in a test case, including the common prefix, our technique checkpoints the application s state right after executing a common prefix and restores the application s state instead of re-executing the prefix. As shown by the Amazon example, the common prefixes can be long and expensive to execute, in particular if some steps (e.g. authenticating the client) in the common prefix are computationally intensive. Note that our approach is not confined to web applications and is also applicable to any event-driven applications. To evaluate our Checkpointing-Based Testing approach, we design and implement a novel system, called WebCheck. WebCheck has a specialized checkpointing component for taking and restoring snapshots of web-application states. To capture a web application s state, WebCheck stores all state-relevant information, such as data stored in a database and session information. WebCheck builds upon the transaction mechanisms, which are readily available in virtually all SQL databases, to efficiently rollback to previous database states. Through experiments, we show that WebCheck is efficient in terms of the time required for restoring the states of real-world web applications as well as scalable in the size of the state (e.g. the database size). WebCheck has a modular design and supports arbitrary fuzz operators for generating test cases using valid inputs and arbitrary test oracles. We have pre-configured it with standard fuzz operators for modifying the sequence of requests in a test case as well as with standard oracles for detecting generic web-application errors such as unhandled interpreter errors and malformed SQL queries. We empirically evaluate WebCheck through case studies over several popular PHP web applications. Our results demonstrate that WebCheck achieves up to 19x speedup in test execution time compared to state-of-the-art solutions and discovers 10 previously unknown vulnerabilities in our test subjects. Structure In Chapter 2 and 3, we present background information on the subject at hand as well as the main definitions for recurring terms in this thesis. In Chapter 4, we present both our checkpointing solution and our WebCheck fuzzing framework. We explain and motivate our main design choices. In Chapter 5, we discuss the implementation of every component of the WebCheck system. In Chapter 6, we discuss our main hypotheses and justify them with a set of experiments. For each hypothesis, we present each case study and review the results. 2

13 2 Related Work 2.1 Application Checkpointing Application checkpointing is a technique whose goal is to make computer systems more fault tolerant. The state-of-the-art checkpointing approaches [3] [4] [8] [15] usually save the state of the computation on disk, in such a way that the computation can be restored when a subsequent failure occurs. The computation s state consists of the memory data, the process identifiers of the application, and a lot of additional components, e.g., file locks or open tcp connections. Application checkpointing is used, in general, in parallel and distributed applications. [8] [12] [1] [2] However, approaches exist also solely for single-node applications. [15] In the following, we focus on single-node applications, as they are relevant for this thesis. There are a few available tools and libraries focusing on single-node checkpointing, namely Cryopid 1, CRIU 2 and Libckpt [15]. Those tools allow to suspend the execution of a running process and take a snapshot of its state, a checkpoint, into a bunch of files on disk. The process execution can then be restored from the checkpoint. Although existing checkpointing tools are captivating, they are not mature enough to checkpoint web applications. After a few experiments, we encountered some issues. For instance, checkpointing tools cannot handle applications spanning over too many processes, as it is the case with web applications. The reason seems to be that the checkpointing tools do not support shared memory segments within the targeted applications. Furthermore, they have strict limitations in dealing with network connections. Therefore, we cannot integrate existing checkpointing tools to our design, as they cannot be used with web applications Page 3

14 2 Related Work 2.2 Web Application Testing In the context of web application testing, there are multiple important prerequisites that come into play. First of all, selecting complex test cases that exercise the web application as desired is not trivial. [17] User-session-based techniques can help tremendously as they allow the tester to record users interactions (HTTP requests) to generate test cases. [6] Web application testing is often neglected or done very quickly as thoroughly testing a web application is time consuming and often lacks significant payoff. [13] Moreover, most commercially available tools are not covering all functionalities of web applications and are, most of the time, addressing accessibility and portability issues. This is a reason why well-defined web testing strategies should be designed to point web application testing in the right direction. [5] The emphasis of current research, rather than being on executing test suites faster, is on optimizing the way test suites are executed [16] or on optimizing test scheduling. [19] There does not seem to be previous work on increasing test suites runtime efficiency. 2.3 Fuzz testing Web Applications Input validation in web applications has often been shown to be a hard task. Many weaknesses in web applications are a direct consequence of it. Moreover, it is infeasible to exhaustively test web applications, due to the large number of possible inputs. Fuzz testing web application tries to address this issue by leveraging a tool to automatically generate test data, which is referred to as fuzzing. [14] [11] The fuzzing tools solely focus on altering HTTP requests (GET and POST) and their parameters, trying to expose web application flaws, such as reflected and stored cross-site scripting (XSS) vulnerabilities and SQL injections bugs. Alternative ways of applying fuzz testing, other than modifying the single requests, have been successfully applied to other domains. For instance, mutating entire traces of messages has been successfully applied to security protocols. [18] Despite the fact that, to the best of our knowledge, this approach has not yet been applied to fuzz testing web applications. 4

15 3 Technical Background In this chapter, we present background information on web applications, web application testing, and fuzz testing. First, we discuss web applications and their inner working with an emphasis on topics relevant to the thesis. Afterwards, we describe web application testing and detail fuzz testing and its different usage methods. Finally, we present relevant background on checkpointing and databases. 3.1 Web Applications Context Over the past two decades, we witnessed the quick expansion of the World Wide Web, which enables users to access web pages over the network via a web browser. Web services are mostly provided by web applications Structure Figure depicts the main components of a typical web application. The web browser is a software application that is mainly used to retrieve and display information, such as web pages, images, videos, and files from web servers. A web server hosts web sites and answers HTTP(s) requests from clients by delivering static or dynamically created content. The web server identifies the resource to be retrieved by the request s URL and maps it to the corresponding file or program. Popular web servers include Apache, Microsoft IIS, and nginx. Application servers usually run the web application s business logic. They act as a bridge between the front-end and the back-end (storage). Within web applications, application servers are often responsible to create dynamic web pages - that is to say, using server-side scripts assembling web pages based on many parameters. The database server runs a database management system (DBMS) and offers its 1 Source: Web Engineering ( L), ETH Zürich 5

16 3 Technical Background Figure 3.1: Typical structure of a web application. services to other applications that need to manage persistent data. The choice of an appropriate DBMS is based on the requirements of the web application. From an architectural perspective, a web application is made of three or more tiers as depicted in Figure The application tier can be divided in as many tiers as needed by the developers. The application tier is often divided in multiple components. The web server and the application logic are the more common ones and make it possible to have a working web application. On top of that, other components can be added to integrate the web application within a bigger system (e.g. enterprise business logic) or simply the model instances for web applications designed using the model-view-controller (MVC) architectural pattern. Figure 3.2: A web application s tier list. 2 Source: Web Engineering ( L), ETH Zürich 6

17 3.1 Web Applications Storage As we have seen above, web applications usually store persistent data in databases. There are also other means for storing data persistently, such as files, but they are rarely used in practice. SQL databases are the most popular storage engines used by web applications. Typically, a database server contains one or more database, each one consisting of a number of tables. The data stored in the database is managed using SQL queries. Most programming languages feature APIs for interacting with the database server using built-in functions. SQL queries are used to both read and write the database. Web application developers have to be very thorough when dealing with those as different checks have to be made depending on the nature of the query Stateful Web Applications The communication between the web server and the client runtime is built on HTTP/HTTPS 3. The HTTP protocol is stateless, which raises the following inquiry: how can web applications be stateful when they are built on top of a stateless protocol? The answer to this question is simple: sessions. A session is defined as a series of related browser requests that come from the same client during a certain time period. Session tracking ties together a series of browser requests think of these requests as pages that may have some meaning as a whole, such as a shopping cart application. 4 In the following, we detail how PHP handles sessions. Other programming languages and frameworks may implement sessions differently. We review here the main concepts about session management that are relevant to the thesis. A session can be thought of as a key-pair data structure. Each session identifier is linked to the corresponding data, such as the user name or the user identifier. Both the server and the client each keep track of some of the session data. There are two ways to store the session information: either on the client or on the server. HTTP Cookies usually store the session information on the client. They are a small piece of data from a website store inside the browser. In contrast, session information is usually stored in a file, in memory, or in a database on the server. When the session information is stored on the server, the client has to store a reference to its session information in the form of a session identifier in the corresponding cookie, as we can see in Figure In addition, the web server keeps track of the user s state /wls/docs92/webapp/sessions.html 5 Source: gif 7

18 3 Technical Background using sessions. Each session ordinarily has an expiration date rendering the session obsolete. Sessions can also become obsolete after a predefined time of inactivity. The web server configuration dictates when an obsolete session has to be discarded. Figure 3.3: Setting up an HTTP cookie Web Application Frameworks Web application frameworks are designed to help with the development of web applications. They apply different techniques and patterns from software engineering directly to web development. There are a number of web frameworks families, which can be anything from frameworks, languages or environments, namely: PHP, Python, Ruby, Microsoft.NET, Java VM and Javascript. The list is not complete and represents a selection of some well-known (or widely used) web frameworks families. Each framework has its strengths and weaknesses, it is up to the developers to choose which one suits them best. Figure depicts the framework usage statistics for the top 10k web sites. Figure 3.4: Framework usage statistics for Top 10k web sites. 6 Source: 8

19 3.2 Web Application Testing Web Application Security Web application security is an important subject because web applications typically store sensitive data, such as confidential business information or private user data, which must be handled with great caution. A fundamental weakness is that the web server must process any requests, including requests that are sent by malicious users. Malicious users can submit carefully crafted requests in order to exploit security flaws in the web application. The web application must therefore correctly validate all requests so as to avoid security threats. 3.2 Web Application Testing Context Finding faults in web applications has always been a challenge. The wide variety of heterogeneous technologies used to implement web applications as well as a lack of well-defined test strategies complicate effectively testing of web applications. Moreover, the wide variety of possible failures makes it cumbersome to reveal the most critical failures within web applications. Over the last few years, web testing has grown fast by trying to overcome the limitations (broad variety of failures and undefined testing strategies) that initially made finding errors in web applications difficult. There is a vast landscape of diverse tools that allow the tester to scan a web application in a reliable way. The goal of web application testing is to reveal failures, which is the inability of an application to perform its required functions, such that the underlying faults - a weakness in an implementation that can lead to a failure - can be fixed to later have a more robust and secure web application. There are a lot of distinct aspects, such as: usability testing; user acceptance testing; performance testing; security testing; functional testing and interface testing. 7 All those techniques have precise testing guidelines and methods. In this thesis, we mainly focus on security testing. Security testing s main goal is to uncover faults that, if exploited by attackers, might lead to data leaks, such as unauthorized access to confidential data or protected resources, and integrity violations, such as corrupting the web application s state. Faults are usually difficult to uncover and the web application needs to be thoroughly tested to reveal them. Therefore, testing for different kinds of failures might take a lot of time. Different features can be tested. A list of tasks to perform associated 7 9

20 3 Technical Background with security testing can be found at [ 8 ]. We cannot test for all kinds of failures, exploring all aspects of security testing is far too broad of a domain. Security testing in web applications can be broken down into several methodologies such as penetration testing, fuzz testing,... Fuzz testing has already been used to successfully find many security problems in real world applications. Therefore, we solely focus on fuzz testing Approaches There are mainly two standard approaches to test web applications: black-box testing and white-box testing. Black-box testing considers the system as a black box. There are entry points and exit points but the content of the inside of the box is unknown. The tester can test the inputs and outputs. Black-box testing is principally used to ensure that the system works according to the specifications. In contrast, white-box testing implies that the tester has a complete knowledge of the inner working of the web application. This knowledge can be used to target tests to a specific region of the web application code Tools A number of automated tools exist for security testing in web applications. Some of them are black-box oriented meaning that they primarily focus on the inputs and outputs of the web applications, whereas others can be customized to fit into a more white-box oriented testing approach using knowledge from the implementation of the web applications. On top of that, some tools target a very specific methodology of web application security testing such as, for example, penetration testing. A list containing most of the web application testing tools can be found at [ 9 ]. 3.3 Fuzz Testing Here, we describe fuzz testing in detail. Fuzz testing is a software testing technique that tests programs by feeding them with invalid, malformed, unexpected, or random inputs. The tester uses monitoring tools, such as Valgrind 10, to identify the possible failures that may reveal security vulnerabilities. 8 Application Security Testing Cheat Sheet 9 A: Testing Tools

21 3.3 Fuzz Testing Fuzz testing is ordinarily used within black-box testing techniques. It gives a high benefit-to-cost ratio [9], as fuzz testing sometimes reveals critical exploitable faults and is not difficult to set up. For instance, fuzz testing can be used to test the robustness of error-handling routines. There are distinct approaches to fuzz testing: mutation-based and generationbased fuzz testing. With the mutation-based approach, a fuzzer, a tool used to test parameters of an application, mutates existing data whereas with the generationbased approach, a fuzzer generates new data based on its prior knowledge of the application s data. Moreover, successful fuzzers usually leverage good fuzz operators with deep knowledge of the application or protocol under test. A fuzz operator is a function being used to mutate a well-formed input. The mutated input is subsequently likely to expose vulnerabilities. Fuzz operators can, in the case of white-box testing, be extracted from the specification of the protocol itself. Using the specifications of a program to generate mutated inputs has the advantage that the fuzz operators can target a specific part of the specifications directly. [18] On one hand, the main goal of fuzz testing is to help find faults to be fixed during the development phase of a software. On the other hand, seen from the attacker perspective, fuzz testing is a reliable option to find vulnerabilities in software. There are a few limitations to fuzz testing. First of all, it is not possible to test softwares based on their specification at an early stage in the development. Testing is always done on the system, not on the specifications. Additionally, the tester does not always have access to the entire code or specification of the system under test (black-box testing). This complicates fuzz testing as it becomes harder to target specific application areas. It leads to fuzz testing mostly finding simple errors. Mutation-based and generation-based fuzz testing both have their set of advantages and disadvantages: Mutation-based PROs: It mutates existing inputs that are meaningful to the application. There is no need to extract any kind of specification from the system under test. Mutation-based CONs: Inputs are well-formed meaning that, by mutating those, we may miss some aspects that require more mutations. It is suited for black-box testing only. It requires a set of system inputs to work with. Generation-based PROs: By generating inputs itself, it can target specific parts of the system. It is suited for white-box testing. It does not require existing system inputs. Generation-based CONs: It is required to extract some specifications from the system under test in order to generate meaningful inputs. 11

22 3 Technical Background 3.4 Checkpointing and Databases Checkpointing Context Originally, checkpointing application has been investigated to improve fault-tolerance in applications. It works by taking a snapshot of the application s state and storing it in a bunch of files on the disk. In case of failure, the application can be restarted from the saved state by restoring it from its files. Our checkpointing method varies from the standard approach we just described, as it does not store the state of a web application completely on the disk but mostly in memory. State manipulation Here, we describe the inner working of checkpointing processes, detailing the necessary manipulations on the processes states. A snapshot of a process is a dump of all the process relevant data. It includes the following elements: a copy of the process s address space; the state maintained by the kernel and the system server for that process. [7] The snapshot is stored into files on a reliable storage (e.g. hard drive) that can sustain failures of the running process. In case of failure, the process can be restored to the snapshotted state. The state information are read from the files and each component of the snapshot is restored to replicate the process s state at the time of the snapshot Databases Context In the previous sections, we mention that web applications interact with databases a lot. Here, we give a little bit more details about databases inner workings and, specifically, we focus on transactional databases. Nowadays, most database engines have native support for transactions. A transaction can be seen as an unit of work where one or more actions are being performed against the database at once. 12

23 3.4 Checkpointing and Databases Transactions A transaction is a group of operations having some special properties in common, they are atomic, consistent, isolated, and durable (ACID) 11. This implies that either all operations are executed or none of them are. The database will not end up in an inconsistent state following the execution of the transaction. The intermediate operations are completely isolated from the rest of the world until the transaction s completion. In most databases, a transaction is started using a specific command such as BEGIN or START TRANSACTION. Following this command, all the queries performed against the database will be isolated and seen only from inside of the transaction. The transaction is successful when the command COMMIT is executed, meaning that all the changes made during the transaction are now stored on the database. The transaction can be aborted. In this case, the database will issue a ROLLBACK command, undoing everything that was done inside the transaction. There are other implemented mechanisms giving transactions even more flexibility. It is possible to issue commands to create a save-point inside the transaction, such as CREATE SAVEPOINT savepoint name. This command offers a mechanism to rollback a smaller portion of a transaction by executing the command ROLLBACK TO savepoint name, as long as there have not been any auto-committed queries issued in between

24

25 4 Prerequisites In this Chapter, we give the main definitions for recurring terms in this thesis. 4.1 Definitions Stateful Web Applications A web page displayed on a web browser is made of multiple elements. Some elements are statics (e.g. text and images) whereas others are dynamic and can be interacted with (e.g. links, buttons, and input fields). When interacting with a dynamic element, an event is triggered which causes a reaction. For example, a new web page appears or some provided data is now stored by the web application. A stateful web application keeps track of the executed events and may react differently depending on what events have already been executed. Its state machine (or automaton) defines which user-triggered events are enabled in a given state. For example, a user first has to login prior to changing his sensitive personal information and therefore in the initial state the only enable event is login Test Cases A test case is a trace of events (or messages). Events are defined by HTTP requests. Our test cases only contain those events that change the state of the web application. Consequently, HTTP requests used to load images, JavaScript, and style sheets are not part of the test case Test Suites A test suite is a collection of test cases that will be used to test a web application. 15

26 4 Prerequisites Snapshot of web application When speaking about a snapshot of the web application, it means that we froze the state information from the web application into the snapshot. We can restore the snapshot to bring the web application back to this prior state at a later point in time. 16

27 5 Design In Sections 5.1 and 5.2, we discuss the design of our tool. We present both our checkpointing solution and our WebCheck fuzzing framework. We also explain and motivate our main design choices. 5.1 Checkpointing Goal Our goal is to develop an efficient technique for testing stateful web applications. The key insight is that many of the test cases in a given test suite share a common prefix of events, for example, sending a login request to a web application. We build upon the ideas from the state-of-the-art techniques for checkpointing applications, discussed in Chapter 2, which can be used to take snapshots of different application states. To execute a test case, we can then restore a particular state, as opposed to starting the application from its initial state.the difficulty lies in designing a checkpointing technique tailored to web applications: web applications are typically large systems spanning over multiple layers (e.g. database and application tiers) and the state-of-the-art checkpointing approaches do not scale to such complex systems. We expect our approach, called checkpoint-based testing, to reduce the overall test case execution time because it eliminates the need to execute the events that are part of the prefix shared among multiple test cases. Our approach also comes with several challenges. Using existing checkpointing tools to capture the states of web applications is nontrivial as they have been recently introduced. They are still unstable and they do not scale to large web applications with databases. Therefore we cannot readily use existing checkpointing tools to achieve our goal and we need to come up with an alternative solution. The notion of state must be explicitly defined to guarantee that the behavior of a checkpointed web application is similar to that observed when the web application is restarted from its initial state. The web application does not depend on elements outside of the state (e.g. time) and those can be controlled if it is absolutely vital to the well-being of our design. 17

28 5 Design Web Application State We now formalize states for web applications. There are several key factors to consider. A web application, as others event-driven programs, does not operate as most standalone applications that can be found in an operating system. It does not distinctly start and end, it runs endlessly (excluding maintenance). Therefore, the state of a web application doesn t depend on the web application itself, but instead on its interaction with external entities. Web applications usually process events or requests sent by those entities and the events themselves impel the web application to modify its running state. Hence, the web application s state is associated to the events it receives. Therefore, it s easier to pinpoint exactly where to look at to define the state of the web application. Web application s states can be represented using automata theory. Figure 5.1 depicts such an automaton, a state machine consisting of three distinct states. Figure 5.1: Simple automaton. Starting in the initial state A, the web application can transition into other states (B and then C) according to predefined transitions. We define the transitions to be events received by the web application. We can map this simple automaton to a real life example; take the following use case: a client logs into a cart application and adds an item to his cart. We have the following states and transitions: 1 st state A: no client is logged in (initial state of the web application) 2 nd state B: client is logged in 3 rd state C: client has an item in its cart 1 st transition A to B: client submits a login request with its credentials 2 nd transition B to C: client adds an item to its cart Firstly, the client has to log into the web application, this is usually handled by the web application s session management utility as described in Chapter 3. Afterwards, the client adds an item to its cart. Web applications usually store their users and products data in a persistent storage such as a database. 18

29 5.1 Checkpointing From this example, we can extract two key elements required to precisely define a web application s state: the session data and the database. Keeping track of those two components permits to capture the web application s state, defined by the pair (<session,database>), where the session data has been extracted and saved from the web applications and a database dump has been saved to its current state. The state can subsequently be restored to erase everything that has been done in the mean time Checkpoint Methods Here, we describe the checkpoint methods that we are going to use starting with the database component and following with the session data. Databases Most web applications use a database management system (DBMS) to persistently store their data. The naive approach to database checkpointing is to dump the content of the entire database. While trivial to implement, this approach is inefficient and does not scale well with the size of the database. We will use this naive approach as a baseline in Chapter 7, where we compare different approaches for database checkpointing. To achieve a robust, efficient, and highly scalable checkpointing solution, we leverage the power of transactional databases by isolating the execution of test cases inside transactions. The transactions can be rollbacked to the initial state to start with the execution of the next test case. Rather than start the execution of each test case from the beginning, we use the save-point mechanism of transactional databases to only execute the common prefix of similar test cases once. We set a save-point after the N th request to save the state of the database. For each test case sharing those same first N requests, we directly rollback to the save-point which save us the execution of the first N requests. The save-point acts as a snapshot of the database after the Nth request. All the subsequent requests are erased when executing a rollback to the stored save-point. There is no trace of the previous test case execution after restoring the state at the N th request. Sessions Let s address the session data. Session data can be stored in a database, a file or a cookie. For the case where the session information is stored in the database, refer to 19

30 5 Design the previous section. For the cases where the session information is stored in a file or in a cookie, we proceed as follows: to take a snapshot when the session data is stored in a file, which can be a cookie or a session file, we make a copy of the file. To restore a snapshot, we simply replace the current session file with the saved copy. It is important to note that the cookie identifier (shared at the HTTP level) needs to be stored too, when dealing with session management on the server side. The cookie identifier permits to authenticate the incoming requests and to attach them to their corresponding session data Solution Our solution to checkpoint web applications, called WebCheck, is to use a proxy between the database driver of the web application and the database itself. The idea, depicted in Figure 5.2, is to extend the database driver in such a way that, instead of executing database queries itself, database queries are redirected to an external proxy and the proxy returns queries results as would the standard database driver. Extended Driver Web Application Native Database Driver Proxy Database Figure 5.2: WebCheck s design. The WebCheck proxy opens a persistent connection to the database to allow transactions to be used. All the received queries are forwarded through the persistent connection which gives the WebCheck proxy full control over all communications to the database. In particular, the WebCheck proxy can makes save-points and rollbacks for checkpointing purposes. The results are parsed and returned in a format that is understandable for the web application s database driver. The extended database driver strictly communicated with the WebCheck proxy and does not have any interaction with the database anymore. Session are being dealt with within the WebCheck proxy. The WebCheck proxy is tuned on a per-application basis to match the methods with which the session are handled. Consequently, each test case can be run in total isolation from the others as the WebCheck proxy will rollback the transaction to an initial state following each test case. Another good reason to do a proxy is that we gain modularity: all the checkpointing logic is implemented in the driver and, to support different kinds of databases, we simply need to extend their drivers. The details and specifications of this approach will be discussed in Chapter 6. 20

31 5.2 WebCheck 5.2 WebCheck Goal In addition to showing the feasibility of checkpointing applied to testing, our secondary goal is to use our newly developed approach within a fuzz testing framework for web applications, the WebCheck fuzzing framework. Fuzz testing web application tends to make use of an immense set of test cases, where each test case contains a slightly different alteration compared to the previous one. Many of those test cases share a common prefix as fuzz testing a web application is not done using single-request test cases because some elements to fuzz may require previous events to happen, such as being logged into the web application. Leveraging our checkpointing techniques within fuzz testing can significantly increase the quantity of test cases one can execute over a fixed amount of time. Web applications have deep state spaces. To thoroughly test them, we need long traces of events. We must, therefore, fuzz entire traces in opposition to individual requests. We apply the same technique that is used to fuzz-test protocols, as seen in [18], where the full traces are mutated. Providing invalid or unexpected sequences of requests to a web application can potentially lead to erroneous behaviors. Our goal is to investigate in that direction and see what can be uncovered Fuzzing Framework The WebCheck fuzzing framework permits to efficiently fuzz-test web applications. From a set of test cases and a set of fuzz operators, it generates a test suite to execute against a web application. It tests the resilience of the web application when exposed to unexpected sequence of events. The test suite contains altered test cases built from original test cases being passed to and transmuted by the fuzz operators. The responses from the web application are redirected to an oracle capable of discovering various failures and report them. The WebCheck fuzzing framework extensively uses the checkpointing method designed in Section 5.1. After each run, when the web application responses have been analyzed by the oracle, we rollback the state of the web application to the most recent save-point, erasing all the modifications performed by the test case. We achieve test cases isolation, which means that each test case is being run independently from the others. The benefits of this isolation of test cases is that if a test case were to put the database in an inconsistent state, the next test case would not be affected. Putting the database back into a consistent state takes resources. This scenario can be avoided by having execution isolation of test cases. Moreover, having execution 21

32 5 Design isolation allows to easily reproduce the results, as there is no need to run multiple days worth of test cases to reach the interesting result. A drawback is that the interaction between different test cases is harder to test for Fuzz Operators Here, we specify what our fuzz operators are. As mentioned earlier, our goal is to focus on altered traces, rather than on malformed individual requests. Consequently, our fuzz operators are targeting and mutating full traces. With this goal in mind, we design the following fuzz operators: INSERT: is used to insert at least one arbitrary request at a given position in the provided trace. REPEAT: is used to repeat the request at a given position in the provided trace. SWAP: is used to switch two consecutive requests with one another from a given position in the provided trace. REMOVE: is used to remove at least one request from a given position in the provided trace. REPLACE: is used to replace a request with an arbitrary request at a given position in the provided trace Test Cases The WebCheck framework, due to its checkpoint-based test execution engine, enables the efficient execution of a large number of long test cases. In addition, we require a set of test cases which covers as many functionalities of the web application as possible. Higher coverage allows our fuzzing framework to potentially discover more failures in the web application. Each web application has its proper set of specific test cases Oracles The WebCheck framework relies on an oracle to catch failures during altered test cases executions. We start by specifying the kind of web application misbehavior we are looking for. 22

33 5.2 WebCheck Our primary goal is to focus on navigation errors. A navigation error occurs when an unexpected chain of event is sent to the web application triggering unanticipated responses. With current web browsers, it is possible to save a web page in a bookmark, allowing the user to quickly access the saved web page by requesting the bookmark directly. This feature may have unforeseen consequences. Web pages typically require context, for example some actions have to be performed before requesting a specific web page. Without context, requesting the web page can lead to a navigation error. There is not a single root cause for navigation errors. Web applications should handle those special cases and prevent unpredicted order of requests to be correctly executed. Although it has been observed that web applications do not always handle it correctly. [10] Furthermore, a similar behavior can be achieved when using the back button of most web browsers. When using this feature, web browsers usually do not resend the request for the web page to the web application. They simply load the web page back from their caches. It leads to similar navigation inconsistencies, comparable to the ones stated above. The aforementioned faults can lead to more or less critical security issues. For example, missing access control checks can allow attackers to access pages that they should not be able to visit otherwise. The error pages may also leak data, such as providing a dump of the database or they could hint at how to inject a malicious query. To uncover this kind of web applications failures, we design specialized oracles capable of exposing such inconsistencies. Our solution is to create navigation state machines (NSM) for web applications. A NSM is state machine that describes a set of authorized transitions coming from a specific navigation state. A navigation state represents a web page and the transitions between navigation states are defined by the allowed events to be issued from this web page. We extract the NSM directly from the web applications by following the available hyper-links on web pages and defining which events trigger a transition in the NSM. A simple example is depicted in Figure 5.3. The WebCheck fuzzer generates altered traces. If a trace belongs to the NSM s language, the oracle does not trigger any errors, otherwise the oracle issues a navigation error. In addition, if the trace has been flagged as illegal, the oracle inspects the web application s responses to see if the illicit request has been properly handled by the web application. A web application properly handles an illicit request when it recovers from it by, for example, redirecting the request to a legitimate web pages. The oracle reports an error if and only if the web application does not correctly handle the illegal trace. 23

34 5 Design Figure 5.3: A simple NSM. The nature of the failure has to be verified by hand to decide on its kind (accesscontrol issues, web server errors,... ). 24

35 6 Implementation In Sections 6.1 and 6.2, we discuss the implementation of every component of the WebCheck system, discussed in Chapter 5, as well as the little refinements of the initial design that are specific to the concrete choice of web applications supported by our implementation. 6.1 Checkpointing Target Applications We focus solely on PHP based web applications that use MySQL as database. Note that a number of existing web applications fall in this category (Facebook.com 1, Yahoo 2 and Wordpress.com 3 ), as shown also by the prominent market share of PHP (see Figure 3.4) and MySQL WebCheck Proxy During the design phase, we have seen in Figure 5.2 what our ideal proxy should look like. First of all, we briefly address the reason why the proxy could not be implemented directly into the database driver of the web application. With PHP, every visit to a PHP script invokes a new instance of the PHP interpreter. Nothing is shared between those instances. Therefore, keeping and reusing a persistent connection to the database cannot be done as the database driver is reset at the start of every instance. On top of that, with web servers such as Apache, the database connection pools are linked to a unique web server child process, thus there is no guarantee that a user s HTTP request will be serviced by the process containing the desired persistent database connection. The web server processes do not share the connections. Finally,

36 6 Implementation transactions cannot span over multiple database connections. These factors are the reason we chose to implement a proxy to handle the persistent connection to the database. Our initial design, as seen in Figure 5.2, is simple, yet lightweight as the queries are executed once and the only overhead latency comes from sending the queries and results to and from the proxy. Implementing our straightforward design is, however, not so simple when dealing with PHP web applications. This issue is specific to PHP and should not happen with other common web frameworks. The issue specific to PHP web applications occurs when retrieving the result of a database query through the standard database driver. The result variable is stored in a PHP resource, rather than in an array or some other standard data structure. The PHP documentation states that a resource is a special variable, holding a reference to an external resource. Resources are created and used by special functions. 5 This rather imprecise definition does not give more insights into what a resource exactly is. Moreover, it is also mentioned that: As resource variables hold special handlers to opened files, database connections, image canvas areas and the like, converting to a resource makes no sense. 6 The main problem is that our proxy has to return the result of the database query it received and the result of a database query should be passed as a PHP resource. There is a resource type called mysql result which contains the outcome of the query executed against the database 7. Looking deeper inside the actual implementation of the PHP MySQL database driver, we discover that the mysql result resource is in fact a complicated C struct containing multiple arrays and variables. Moreover, PHP resources cannot be stored persistently due to their inability to be serialized. They cannot be sent over a communication channel. Consequently, we have to use the native PHP database driver at some point to get the result from the database as there is no other way to build this resource. The refined designed of our PHP proxy is depicted in Figure 6.1. The idea is to overcome the fact that building a PHP resource is complex and time consuming while leveraging the persistent connection from the proxy. The WebCheck proxy receives a query q from the web application and runs it against the database DB inside a persistent database connection. Instead of directly returning the result r to the web application, the database proxy stores the result r in a temporary database DB. The proxy creates a new query q to retrieve the stored result r from DB. Afterwards, the query q is returned to the

37 6.1 Checkpointing Extended Driver Web Application Native Database Driver 1. q 5. q' Proxy 2. q 3. r 6. q' 4. r DB 7. r DB' Figure 6.1: WebCheck s design for PHP. extended database driver of the web application and executed using the native PHP database driver. The result r is recovered as a PHP resource. In more details, here are the specifics of the setup: The WebCheck proxy is written in Python. Python contains the features to easily build the proxy and simplifies the implementation and understandability of the proxy over alternatives such as C++ and Java. The communication channel between the extended database driver and the proxy is done over basic client and server sockets using TCP connections. The server listens on a predefined port and waits for incoming requests. The database proxy can also receive messages from the fuzzer, notifying it when to start or rollback transactions and when to setup save-points. Therefore, the persistent connection to the database is always in the desired state. An incoming query is parsed and subsequently executed inside the persistent database connection using the standard Python MySQL connector. Afterwards, the result is parsed and a new table containing the corresponding data is created. A new query to retrieve the data stored in the temporary table is produced and sent back to the extended database driver of the web application. The new query that is received by the web application is executed using the native PHP database driver over a standard non-persistent database connection. In this setup, the proxy can handle the incoming queries and successfully works as an intermediary between the web application and the database. 27

38 6 Implementation Application-level Instrumentation As explained in the previous section, to intercept all database queries using our proxy, we need to extend the database driver of the web application. In practice, there are two possible use cases, either the web application uses its own implementation of the database driver on top of the native PHP database driver or the web application simply uses the native PHP database driver. Both cases are handled similarly, the only difference being that in the first case, we only have to do one centralized code injection in the database driver file, whereas we have to modify all the web application s files with our code in the second case. First of all, we need a brief insight into how database calls are handled in PHP. There are three different PHP APIs for accessing a MySQL database: mysql, mysqli and PDO. Our technique can be used with each one of them. We are going to describe it in details for the MySQL API. The MySQL API 8 contains functions allowing the developer to query and retrieve data from the database. We are especially interested by the function for sending a MySQL query. The function definition 9 is: mixed mysql query(string $query[, resource $link identifier = N U LL]) Which can be used using, for example: $result = mysql query( SELECT FROM table WHERE 1 = 1, db connection); The query is directly sent to the underlying native PHP database driver, which transmits it to the database and returns the result. Our goal is to intercept the query before it is sent to the native database driver. A PHP extension called runkit allows to do exactly this. The runkit extension provides means to modify constants, user-defined functions, and user-defined classes. It also provides for custom superglobal variables and embeddable sub-interpreters via sandboxing 10. Listing 6.1 shows how we use the runkit extension within our system

39 6.1 Checkpointing Listing 6.1: Database driver extension code. 1 runkit_function_copy ( mysql_query, mysql_query_old ); 2 runkit_function_redefine ( mysql_query, $query, $link_identifier = null, 3 4 // Connect to proxy $client = stream_socket_client (" tcp :// :12321 ", $errno, $errormessage ); 6 7 if ( $client === false ) { 8 throw new UnexpectedValueException (" Failed_to_connect : $errormessage "); 9 } fwrite ( $client, $query ); // Get response from proxy. 14 $response = stream_get_contents ( $client ); if ( $response == " true ") { 17 return true ; 18 } elif ( $reponse == " false ") { 19 return false ; 20 } else { 21 $newquery = $response ; 22 if ( is_null ( $link_identifier )) { 23 $result = mysql_query_old ( $newquery ); 24 } else { 25 $result = mysql_query_old ( $newquery, $link_identifier ); 26 } 27 } 28 // Close connection to proxy. 29 fclose ( $client ); return $result ;); The initial step is to copy the function that we want to redefine to a new one with a different name. We can redefine our target function (here mysql query) by overwriting it with our own code. We start by creating a client socket to connect to the database proxy. When the stream is successfully opened, we send the query to the proxy. We wait for a response from the proxy containing the new query used to retrieve the result using the native PHP database driver. There are a few cases where the query only returns a boolean variable, this happens with queries of the forms: INSERT INTO, UPDATE,... In those cases, we return the received Boolean value. If the response is a query, we use the copy of the mysql query function that we created previously. Using the original function is straightforward, we pass the new query and a non-persistent database connection, if one exists, and it returns the result as a PHP resource. We close the connection to the proxy before exiting Complexity The proposed solution for checkpointing web application comes at a price. The price to pay is an added complexity brought by the additional back and forth exchanges with the database. 29

40 6 Implementation When using the native PHP database driver, there is a single interaction D with the database, making the complexity O(D). In the proposed solution, depicted in Figure 6.1, there are two additional interactions with the database, making the complexity O(3 D). In addition, there is also the processing time due to underlying actions happening within the proxy. Let P be this additional time. The complexity is, therefore, O(3 D + P ). Finally, we have to take into account the communication time to pass the query from the extended database driver of the web application to the proxy. We define one round-trip time as C, which leaves us with a complexity of O(3 D + P + C) compared to the initial complexity of O(D) from using the default database driver. Note that the complexity is the per-query complexity and each request may contain multiple queries. 6.2 WebCheck Fuzzer Input Parsing The test cases are recorded using tcpdump 11 with the command: sudo tcpdump -i lo -w file name.pcap port 80. The test cases are stored in PCAP files containing all the necessary requests to replay the execution of the test case. Initially, test cases are recorded from a user interacting with a web browser. Consequently, the traces saved in the PCAP files contain a lot of requests that are irrelevant for testing. We are testing the behavior of the web application when altering the flow of user requests. Thus, only the requests that are part of this flow are relevant. Browsing a web application through a web browser generates requests for the various images, videos, and JavaScript files to be loaded by the web browser. Those requests are irrelevant as we are not scrutinizing the way a web page is displayed. Therefore, our first step after loading the traces from the various test case files is to filter out the irrelevant packets. The filter rejects requests for: JavaScript files, CSS files, icons, PNG images, GIF images, and JPG images to name a few. Afterwards, the unique requests used by the fuzz operators are extracted from the test cases. We focus on test cases covering as many functionalities from the web application as possible. A set of test cases that thoroughly exercise the web application will naturally result in more complete test cases generated by the fuzz operators. The test cases have to be generated accordingly

41 6.2 WebCheck Fuzzer Fuzzer The proxy needs to be started when the fuzzer uses checkpointing. The proxy is an external component independent from the fuzzer. Additionally, the web application must use the extended database driver to communicate with the proxy. The fuzzer has two distinct modes: normal mode and checkpointing mode. The normal mode simply executes all the test cases from starting message to fuzzed message without exploiting the benefits of checkpointing. The database has to be brought back to its initial state in between each round. On the other hand, the checkpointing mode uses all the functionalities provided by the checkpointing technique. The common prefixes among the test cases are executed only once and the fuzzer can leverage the rollback of the transaction to come back to the initial state. Listing 6.2 describes how the fuzzer alters the initial test cases to form new ones that are then executed. Listing 6.2: Pseudo-code of fuzzer execution. 1 for testcase in set_testcase 2 for position in testcase 3 if checkpointing_mode == true : 4 execute ( request, position - 1) 5 add ( savepoint ) 6 for fuzz_op in set_fuzzoperators 7 for unique_request in set_unique_request 8 altered_testcase = fuzz ( testcase, position, fuzz_op, unique_request ) 9 if checkpointing_mode == true : 10 execute ( altered_testcase, position ) 11 else : 12 execute ( altered_testcase ) There can be hundreds of altered testcases from a single test case. Every time we increment the position, the resulting set of altered testcases share a common prefix, that is to say the first position - 1 requests coming before the alteration. Consequently, the longer the test case, the longer the shared prefix become and the more requests we can save using checkpointing. A fuzzer instance starts by picking a test case from the set of test cases. It iterates through all the possible request s positions in the trace. For each position, the fuzzer applies each fuzz operator with all the unique requests at its disposal. Given that there are P positions in a test case, F fuzz operators, and R unique requests, the total number of unique altered testcases is P F R. When at position N, the N 1 requests are the same for all the altered testcases, using checkpointing implies that, instead of having to run those N 1 requests for all F R altered testcases, we can execute them only once and create a save-point. We can then rollback to this save-point and save: time(n 1 requests) (F R) + time(database restore) 31

42 6 Implementation The bigger the N, the more time is saved Fuzz Operators We use fuzz operators altering traces of requests. Traces are stored in lists, consequently, fuzz operators are operations on lists, modifying their content or order. The Python implementation of the fuzz operators is described in Listing 6.3. Listing 6.3: Python implementation of fuzz operators. 1 def fuzz_op ( operator, testcase, position, unique_request ): 2 3 if operator is " INSERT ": 4 altered_testcase = testcase [: position +1] + [ unique_request ] 5 6 elif operator is " REPEAT ": 7 altered_testcase = testcase [: position +1] + [ testcase [ position ]] 8 9 elif operator is " SWAP " and ( position < len ( testcase ) -1): 10 # swap positions 11 testcase [ position ], testcase [ position +1] = testcase [ position +1], testcase [ position ] 12 altered_testcase = testcase [: position +2] elif operator is " REMOVE ": 15 del testcase [ position ] 16 altered_testcase = testcase [: position +1] elif operator is " REPLACE ": 19 testcase [ position ] = unique_request 20 altered_testcase = testcase [: position +1] return altered_testcase Oracles Our fuzzing framework makes use of two oracles: a generic oracle and an application specific oracle. The generic oracle inspects the content of the responses received from the web application. We check for three kinds of errors: Server errors: happen when the status element in the header of the response is 500. It means that the server was confronted with an unexpected condition and could not complete the execution of the request. PHP errors: happen when the PHP interpreter encounters an unforeseen event and does not handle it correctly. 32

43 6.2 WebCheck Fuzzer Database errors: happen when the PHP interpreter faces difficulties while accessing the database. It results in a database error that can potentially leak important information on the database schema. The application specific oracle is a bit more complex and it works in pair with the generic one. In Section 5.2.5, we described our idea to build navigation state machines (NSM) for each web application. The NSMs are extracted manually from the web application. We navigate through the web pages and extract meaningful events as transitions between states. The NSM extraction takes a few hours for simple web applications. It does not scale well for bigger web applications, as they get a lot more complicated. There are automatic tools (e.g. web crawlers, model-based testing tools,... ) that can help with the NSM generation but this is out of the scope of this thesis. We start from the default page of the web application, for example the index page or login page, and select this to be the initial state of the NSM. Afterwards, we focus on determining which requests or events are central to the web application and can initiate a change of state. Those events have in common that they usually are prerequisites for other events, for example a login event has to happen before accessing some private data. Thus, requests can be categorized into two sets, the ones which are relevant to the flow of the web application and the ones which are not. The relevant requests make the transitions between the states whereas the others represent transitions to the same state. The requests from the second category usually have a request of the first category as a prerequisite but are never the prerequisite of another request. The requests of the first category are the prerequisite of requests of both categories. The altered test cases generated by the fuzz operators contain a trace of requests that is analyzed by the oracle using the NSM for the corresponding web application. In many cases, the oracle will issue false positives as altered traces often fall outside of the NSM s language. To solve this issue, the oracle examines the responses returned by the web application to see if the illegal sequence of events is handled properly. For example, trying to access some private content before being logged into the web application should redirect to the login page. Using this method, we can reduce the number of false positives from the oracle to when the web application inappropriately handle the out-of-place request. The two oracles allow the WebCheck fuzzing framework to uncover a specific kind of failures, namely web server errors and navigation errors. Although, it is important to note that the set of found failures is only as good as the oracle being used. The more effort being put into the design of the oracle, the better results we can expect from the fuzzing framework as a whole. 33

44

45 7 Empirical Evaluation In this Chapter, we discuss our main hypotheses and justify them with a set of experiments. For each hypothesis, we describe each case study and review the results. 7.1 Hypothesis In this thesis, (1) we study the problem of checkpointing web applications and (2) we build a fuzzing framework leveraging checkpointing of web applications to increase the efficiency of running large test suites. By increasing the efficiency, we mean running more test cases in a given amount of time. Concretely, we investigate the following hypotheses: H1: Applying checkpointing techniques to web applications is feasible. H2: Web application checkpointing reduces the time for running test suites. H3: Our fuzz testing approach effectively exposes web application error. 7.2 Experimental Setup The multiple experiments are conducted on the following three PHP web applications: Tomatocart , an open source professional ecommerce solution. ( 325k lines of code) BambooInvoice , an invoicing software intended for small businesses. ( 79k lines of code) Cuteflow , a web-based open source document circulation and workflow system. ( 50k lines of code)

46 7 Empirical Evaluation The web applications use a MySQL database (Version 14.14; Distribution ) as well as the original PHP MySQL API. We use PHP version 5.5, in addition the following module has to be installed: runkit 4. The Zend Engine version is also used. The web server runs Apache version on a Linux Ubuntu machine (kernel version: generic). The machine s specifications are: Intel Core i7-4600u (3.30GHz), 8GB DDR3L SDRAM 1600 MHz, 256GB SSD. 7.3 Evaluation of H1 Our first experiment aims to investigate whether checkpointing web applications is feasible. We compare the outcome of the two following use cases: 1. Run a test case from start to finish. 2. Restore a snapshot of the web application and then execute only the last request. We run this experiment on the TomatoCart web application, as it provides a good scenario to run and it is a mature, well-tested product. The test case we use goes as follows: a client logs into the web application, adds an item to his cart, and proceeds to the checkout process. The snapshot takes place before entering the checkout process, making it easy to detect if checkpointing the web application works. The snapshot is taken while recording the test case from the browser. If the checkpointing works, executing the last request will bring the client to the checkout process with an item into his cart, while being logged in. If the checkpointing does not work, the client will be redirected to the login page of the cart application. Results We run 100 test cases with and without checkpointing. In all cases, the state was restored properly. Discussion This small experiment shows that checkpointing web applications is indeed feasible. Restoring the pair <session data; database data>, which precisely captures the state, before the last request, proved to be sufficient to trick the web application into thinking that the events leading to this state happened previously

47 7.4 Evaluation of H2 7.4 Evaluation of H2 In our second set of experiments, we investigate the time required to run test cases with and without checkpointing. We divide our experiments into two parts First Set of Experiments Design of Experiments We compare our checkpointing solution to various techniques that can be used to execute test cases, namely: Selenium; executing the test case from the start at the HTTP level and, finally, restoring snapshots of the database. The goal is to compare the diverse runtime we get with all the methods. We use a simple test case using the TomatoCart application. The test case at hand is the following: the user connects to the home page of the application and logs in using the login page. Afterwards, the user searches for an item using the various menu and adds an item to his cart. Finally he verifies the content of his cart and proceed with the checkout process. We execute the test case multiple times using the following methods: Selenium 5 (a browser automation testing framework). Raw HTTP traffic (executing the entire test case at the HTTP level). Checkpointing with a snapshot of the full database. Checkpointing with a snapshot of the relevant database tables. Checkpointing with our solution leveraging transactions. Furthermore, we are going to adjust some parameters, such as the amount of data in the database, to see their effects on the various methods. The test case will be executed multiple times again to show what discrepancies a single parameter change can lead to. Results & Discussion The first experiment permits to compare the runtime of the different methods used to run a test case. The results are depicted in Figure

48 7 Empirical Evaluation Transaction HttpTraffic Selenium Table Restore Full DB Restore 3.5 Run duration (seconds) Test Run # Figure 7.1: Runtime for each method over 100 test cases. In addition to showing the feasibility of our different checkpointing methods, this experiments allow us to notice the discrepancies in runtime between them. The snapshot or save-point is taken before the last request which means that for a test case of N requests, the N-1 first requests are replaced by restoring the snapshot or save-point and the N th request is executed normally. We can observe the following behaviors from Figure 7.1: Selenium is slow which is expected as it runs the test case interactively within the web browser at a speed that is slow enough for a human being to follow. There is a huge different between restoring a snapshot of the entire database compared to restoring the snapshot of a subset of tables. Restoring the snapshot of the entire database is actually slower than executing all the requests from the test case in a row. Checkpointing using transaction is actually slower than restoring only the relevant tables for the test case. These results are encouraging but there is an important component that has been neglected. The amount of data in the database is neglected. In practice, the databases used for testing purposes are never completely empty. A 38

49 7.4 Evaluation of H2 common practice is to take a copy of a production database and execute test cases against it. Therefore, it is crucial to minimize the time taken to restore the snapshot. Our second experiment takes into account the missing parameter about the amount of data in the database. The results of this experiment are depicted in Figure Transaction Rollback HttpTraffic Tables Restore - 19KB Tables Restore - 11MB Tables Restore - 100MB Run duration (seconds) Transaction Rollback HttpTraffic Full DB Restore - 511KB Full DB Restore - 11MB Full DB Restore - 100MB Run duration (seconds) Test Case Run # Test Case Run # (a) Restoring Full Database. (b) Restoring Subset of Tables. Figure 7.2: Runtime for each method when varying the amount of data in the database. In both Figures, we compare our transactional checkpointing technique and the raw HTTP execution of the test cases to the database checkpointing techniques when varying the size of the database. We choose arbitrary amounts of data in the database to show how it affects the two dumping techniques. We can conclude from these experiments that the time required to restore the snapshot increases with the amount of data in the database. In addition, it is interesting to look deeper and understand where the time is actually spent. The processing time, i.e. the time it takes for the web server to process the last request of the test case, is constant (around 0.04 seconds) when using the two techniques that restore snapshots of the database. In contrast, restoring the snapshot takes significantly more time. It gets worse the bigger the snapshot is. In comparison, we observe that restoring the database state using a transaction rollback is very fast (around seconds independent from the amount of data in the database). Besides, it is important to note that the processing time for the last request is significantly bigger (around 0.2 seconds). This loss is due to the added complexity of our proxy, as discussed in Chapter 6. The more queries per request the web application issues, the bigger the processing time gap is. 39

50 7 Empirical Evaluation Second Set of Experiments Design of Experiments We run our WebCheck fuzzer on a set of test cases for all three web applications. As a benchmark, we perform the experiments using a normal approach to web application testing, executing each test case from the start up until the fuzzed part of the trace. Afterwards, we perform the same experiment, but this time using our checkpointing technique. We compare the runtime to execute the entire test suites and extract other important information from the results. We start by describing the setup required for each web applications. BambooInvoice The application-level modification are restricted to the in-application database driver located in /bamboo system files/database/drivers/mysql/mysql driver.php. To cover the functionalities of the whole application, we created four test cases of lengths 14, 11, 20 and 44, which permit to inspect most aspects of the web application behaviors. From those test cases, we extract 37 unique requests corresponding to all the pages and features of the application. From those initial test cases and unique requests, the WebCheck fuzzer generates and executes 13,117 fuzzed test cases that form the test suite. The runtime of the test suite is compared for the two approaches. CuteFlow The application-level modification have to be made in every single PHP file of the application as it does not implement its own database driver. To cover the functionalities of most of the application, we created three test cases of lengths 12, 48, and 29, which permit to inspect different behaviors of the web application. From those test cases, we extract 32 unique requests corresponding to most of the pages and features of the application. From those initial test cases and unique requests, the WebCheck fuzzer generates and executes 5,785 fuzzed test cases that form the test suite. The runtime of the test suite is compared for the two approaches. TomatoCart The application-level modification are restricted to the in-application database driver located in /includes/classes/database/mysql.php. To cover the functionalities of the whole application, we created six test cases of lengths 15, 24, 17, 24, 40 and 17, which permit to inspect most aspects of the web application behaviors. From those test cases, we extract 38 unique requests 40

51 7.4 Evaluation of H2 corresponding to most of the pages and features of the application. From those initial test cases and unique requests, the WebCheck fuzzer generates and executes 10,832 fuzzed test cases that form the test suite. The runtime of the test suite is compared for the two approaches. The goal is to compare the results and show that our checkpointing technique permits to run the test suite more efficiently. Results & Discussion BambooInvoice The results of the experiment on the BambooInvoice application are depicted in Figure 7.3. Figure 7.3: Average runtime for the experiment on BambooInvoice. For each test case, the left column represents the average runtime of the normal execution of the fuzzed test cases, whereas the right column represents the average runtime of the WebCheck framework. The average is calculated over the execution of all the fuzzed test cases generated from the depicted test case. In each case, both the time to restore or rollback the database and the time spent in the web server are superposed. Note that the rollback time is too small to appear on the graph. We can clearly see that our checkpointing approach allows to execute a lot more test cases in a shorter amount of time. The global speedup factor is 8.13x. 41

52 7 Empirical Evaluation The speedup factor is skewed by the fact that there is a costly-to-process request in the third test case, and, for such requests, checkpointing yields high speedups. To clarify that, Figure 7.4 depicts the total runtimes of the two approaches for each test case, and its corresponding test suite, separately. 800 Normal Execution Checkpointed Execution 600 Normal Execution Checkpointed Execution Total Runtime [seconds] Total Runtime [seconds] Fuzzed Test Case # Fuzzed Test Case # (a) Total Runtime Test Case 1 (b) Total Runtime Test Case Normal Execution Checkpointed Execution 6000 Normal Execution Checkpointed Execution Total Runtime [seconds] Total Runtime [seconds] Fuzzed Test Case # Fuzzed Test Case # (c) Total Runtime Test Case 3 (d) Total Runtime Test Case 4 Figure 7.4: Total runtime to execute every fuzzed test cases for each test case. Each graph displays the total time taken to execute all the fuzzed test cases generated by the mentioned test case. The red line represents the normal execution whereas the dashed blue line represents the WebCheck framework execution. The speedup factor for each test case is calculated in Table 7.1. We observe that, outlier aside, the speedup factor is correlated to the length of the test case. The longer the test case, the larger the speedup. This added benefit is due 42

53 7.4 Evaluation of H2 Test Case # Test Case Length Speedup Factor x x x x Table 7.1: Speedup Factor for BambooInvoice s test cases. to the fact that, while the time to execute a single request using our checkpointing technique is relatively constant, the execution time for all the requests increases with the length of the fuzzed test cases. The time gained when fuzzing position 1 is a lot smaller than the gain when fuzzing position 50. The third test case is an outlier with a tremendous speedup compared to what it should be. Analyzing the behavior of the web application in more details, we uncovered that this is due to a very expensive request being executed. In the normal execution of the fuzzed test cases, this expensive request has to be executed for every single test cases as soon as its position has been reached. Whereas with our checkpointing technique, this expensive request has to be executed only once. The save-point created after it prevents it from being executed again which results in a huge speedup. Another aspect to take into account is the fact that the added complexity from the proxy is principally due to the number of queries executed per request. In this experiment, 127,674 queries were handled by the proxy which, for 13,117 test cases, gives an average ratio of 9.73 queries-per-request. The conclusion from the experiment using the BambooInvoice application are the following: The speedup of our checkpointing technique increases with the length of the initial test cases. Our checkpointing technique is even more effective when a few requests are taking a tremendous amount of time when compared to the others. CuteFlow The results of the experiment on the CuteFlow application are depicted in Figure 7.5. As in the previous experiment, for each test case, the left column represents the average runtime of the normal execution of the fuzzed test cases, whereas the right column represents the average runtime of the WebCheck framework. The average is calculated over the execution of all the fuzzed test cases generated from the depicted test case. In each case, both the time to restore or rollback the database and the 43

54 7 Empirical Evaluation Figure 7.5: Average runtime for the experiment on CuteFlow. time spent in the web server are superposed. The rollback time is too small to appear on the graph. We observe that our checkpointing approach allows to execute a lot more test cases in a shorter amount of time. The global speedup factor is 5.05x. Compared to the previous experiment, the results are not skewed by a single test case. Figure 7.6 depicts the runtime of the two approaches for each test case separately. Each graph displays the total time taken to execute all the fuzzed test cases generated by the mentioned test case. The red line represents the normal execution whereas the dashed blue line represents the WebCheck framework execution. The speedup factor for each test case is calculated in Table 7.2. Test Case # Test Case Length Speedup Factor x x x Table 7.2: Speedup Factor for CuteFlow s test cases. We observe a slightly different result than in the previous experiment. Only looking at test cases 1 and 2, we see that increasing the length of the test case results 44

55 7.4 Evaluation of H2 800 Normal Execution Checkpointed Execution 600 Normal Execution Checkpointed Execution Total Runtime [seconds] Total Runtime [seconds] Fuzzed Test Case # Fuzzed Test Case # (a) Total Runtime Test Case 1 (b) Total Runtime Test Case Normal Execution Checkpointed Execution 6000 Total Runtime [seconds] Fuzzed Test Case # (c) Total Runtime Test Case 3 Figure 7.6: Total runtime to execute every fuzzed test cases for each test case. in a higher speedup which is as we expected it. However, test case 3 does not follow the same pattern. Upon further investigation, it seems that this is due to a very high number of queries per request, especially in test case 3. In this experiment, 217,475 queries were handled by the proxy which, for 5,785 test cases, averages to a ratio of 37.6 queries-per-request. This value is heavily skewed by test case 3 as the ratio of queries-per-request is a lot higher than for the other test cases. Moreover, when comparing the result of the CuteFlow application to the results of the BambooInvoice application, we observe that the speedup factor for test cases of similar length is not comparable. (Length 11 with 1.72x speedup for BambooInvoice compared to length 12 with 4.02x speedup for CuteFlow) These results have an 45

56 7 Empirical Evaluation explanation. The BambooInvoice application mostly uses cheap queries when addressing the database, whereas CuteFlow issues a lot of queries containing expensive components, such as a wide variety of joins. This, adding to the fact that CuteFlow s queries-perrequest ratio is almost four times higher than BambooInvoice s, explains the reason why running full test cases for CuteFlow is a lot slower for a similar test case length. The conclusion from the experiment using the CuteFlow application are the following: The speedup brought by our checkpointing technique increases with the length of the initial test case, but the speedup factor is highly affected by the nature of the test case. A test case having a huge queries-per-request ratio will not experience a comparable speedup to a test case with a normal queries-perrequest ratio. The higher the ratio of queries-per-request in a test case is, the slower the speedup get. Overall, at the web application scale, a higher queries-per-request ratio results in a higher speedup factor when comparing web applications to each other. TomatoCart The results of the experiment on the TomatoCart application are depicted in Figure 7.7. As in the previous two experiments, for each test case, the left column represents the average runtime of the normal execution of the fuzzed test cases, whereas the right column represents the average runtime of the WebCheck framework. The average is calculated over the execution of all the fuzzed test cases generated from the depicted test case. In each case, both the time to restore or rollback the database and the time spent in the web server are superposed. The rollback time is too small to appear on the graph. Notice that the experiment on TomatoCart leads to very different results. The reason being that TomatoCart is a much more mature, optimized and well-tested product. The execution time of request is an order of magnitude smaller than for the previous experiments. We observe that, compared to the previous experiments, the normal execution time is not too far behind the WebCheck execution time. The global speedup factor is 3.5x, when accounting for both the time to restore the database and the execution time, whereas the speedup factor drops to 1.58x when only taking the execution time into account. There are multiple reasons leading to this result. We can observe that, the longer the test case, the bigger the speedup is, as seen in Table 7.3 when only accounting for the execution time. The fact that our test suite 46

57 7.4 Evaluation of H2 Figure 7.7: Average runtime for the experiment on TomatoCart. is mostly made of short test cases has a bigger impact on the global speedup factor. It is good to notice that some test cases may contain more expensive requests which can lead to smaller speedup factors for similar lengths. Moreover, we see that, when considering the time to restore the database, the speedup increases a lot. Test Case # Test Case Length Execution Speedup Total Speedup x 3.03x x 3.48x x 3.08x x 3.47x x 4.24x x 3.13x Table 7.3: Speedup Factor for TomatoCart s test cases. Another important factor is the nature of the queries handled by the WebCheck proxy. Over the course of the experiment, 292,085 queries were handled by the proxy which, for 10,824 test cases, averages to a ratio of queries-per-request. We have seen in Chapter 6, that the higher the queries-per-request ratio, the bigger the slowdown from the proxy. In addition, the amount of data passed through the proxy play a significant role. In the two previous experiments, the majority of the queries were very simple and did not result in big amounts of data being passed around. With the TomatoCart application, there are some queries that induce a big amount 47

58 7 Empirical Evaluation of data being passed through the proxy. This reduce its performance significantly. The results show that executing a single request through the proxy can take as much time as executing around 10 requests using the native PHP database driver. The conclusions from the experiment using the TomatoCart application are the following: The speedup is bigger, when the test cases are longer. Efficient applications issuing a lot of expensive database queries significantly reduce the effectiveness of our checkpointing technique. When accounting for the time needed to restore a database to its initial state, our checkpointing technique shines again, as transaction rollbacks are extremely fast. Our implementation of the WebCheck proxy for PHP web application shows its limitations and, in the future, it might be a good idea to reconsider implementing a proper proxy, overcoming the difficulties we encountered. Global Conclusions In the previous three experiments, we have made multiple observations. First, the longer the test case, the more efficient our checkpointing technique. In addition, there is an even bigger speedup when a few requests are expensive to run and take up for most of the execution time. On the other hand, test cases having big queries-per-request ratios are significantly decreasing the effectiveness of our checkpointing technique, even more when big amounts of data have to be passed through the WebCheck proxy. 7.5 Evaluation of H3 Here, we investigate the effectiveness of our fuzzing testing approach in terms of exposing web application errors Design of Experiments We use the same experiments that were run for all three web applications in the previous section. We focus on the effectiveness of our fuzzing technique to find errors in the aforementioned web applications. For each web application, we have one application-specific oracle defined as a navigation state machine (NSM) looking for navigation issues within the web application and one generic oracle looking for generic errors, which cover generic PHP errors, 48

59 7.5 Evaluation of H3 and database errors. As the experiments are performed using a normal testing approach and using our checkpointing technique, we have two sets of results to analyze. However, in this section, we will only present a single set of results as both sets are identical. The fuzzer uses the same inputs and uses the same fuzz operators leading to identical outputs as there is no randomness involved. BambooInvoice It has been mentioned in [10] that this application contains a few navigation errors due to missing access control checks. Those can lead to web server failures expressing themselves to the user as generic PHP errors. CuteFlow As far as we know, there are no publicly disclosed vulnerabilities for the CuteFlow application, although community users have uncovered some bugs. TomatoCart The TomatoCart web application is more popular and is the shopping cart module in many real-world web sites. There are no publicly disclosed vulnerabilities related to our work Results & Discussion BambooInvoice are the following: The results of the experiment on the BambooInvoice application We find 10 generic errors caused by missing checks for undefined PHP objects. We find 14 navigation errors caused by missing navigation checks. There are many generic errors, as an example, we may get: A PHP Error was encountered. Trying to get property of non-object. Those errors do not compromise the web application. Many error messages are displayed when trying to display a page with such errors. The root cause for those errors is that the application does not check if a given object exists before trying to modify it. Navigation errors do not always trigger generic errors. It can be that only a blank page is displayed. Navigation errors are mostly caused by missing access-control checks. For example, trying to edit the information of a non-existing client should be forbidden, whereas, in this case, the corresponding web page can be loaded and is full of generic errors because the client object the application is trying to edit does not exist. 49

60 7 Empirical Evaluation From those two categories of errors, we can deduce the following: Generic errors are due to missing validation checks on the existence of an object in most cases. Navigation errors are due to missing access-control checks to establish if a web page can be loaded. All the generic errors happen are paired with a navigation error. The reason is probably that the developers were not expecting an uninitialized object to be accessed and the only way to access such an object is to trigger a navigation error. Moreover, the main issue in this case is that the erroneous web pages should not be possible to load. The application is missing some access-control checks to determine if all the conditions to request a web page are fulfilled. Many of the found errors are not security-relevant, as they do not give any extra features to a user exploiting them. There are no dump of the database happening and unauthorized users cannot modify the content of the web application as all the errors happen after login. We can observe that for both kinds of errors, there are a lot of duplicates to reach 878 and 1,041 errors from 10 and 14 unique errors. It is interesting to measure when each error is first encountered to see the effectiveness of our fuzz testing technique. Figure 7.8 depicts the amount of fuzzed test cases required to discover the various errors Generic Errors Navigation Errors 12 Number of unique errors found Fuzzed Test Case # Figure 7.8: Amount of fuzzed test cases required to discover errors. 50

61 7.5 Evaluation of H3 We observe that all of the errors are discovered early in the experiment, within the range [305:335] over 13,117 total fuzzed test cases. We notice that errors are not discovered before the 305 th fuzzed test case. This is because all the fuzzed test cases prior to this point happen before the user actually logs in. Whatever action happens before login is redirected to the login page, which explains that no errors can be found there. As soon as we start to fuzz traces at a position farther than the login position, we directly, within seconds, find all of the errors. Another interesting pattern to notice is that all the errors are found by the INSERT fuzz operator. This is because this fuzz operator permits us to directly insert a specific request, without using any existing requests from the trace it fuzzed. The results suggest that when testing for navigation errors and generic errors, often we can find most errors quickly as we can trigger the execution of any web pages without requiring any specific state. It is either missing all the checks or missing none of them. The results show that the errors are easy to find. Most of the errors can be found by relatively few test cases. Also, the test cases that find the errors are short in size (less than 4 HTTP requests). This suggests that found errors are simple and do not have a complex dependence on a specific state, i.e. they are revealed as long as the user is logged in. CuteFlow following: The results of this experiment on the CuteFlow application are the We find 0 generic errors. We find 20 navigation errors caused by missing access-control checks. We find 11,029 malformed queries coming to the proxy out of the 217,475 queries handled by the proxy during the experiment. The 11,029 malformed queries happen in conjunction with the 1,290 total navigation errors recorded, as there are many queries per request. We cannot find generic errors because the developers of this application did not omit the validation checks on elements used by the web pages. Navigation errors, in this case, are always caused by missing access-control checks. The whole application contains very few access-control checks that properly work. Consequently, almost every functionality of the web application can be accessed by a user directly from the home page of the web application. Malformed queries received by the proxy are a direct consequence of the navigation errors. When requesting an element that does not exist, the web application issues malformed queries. Those errors are not reflected as generic errors because the web 51

62 7 Empirical Evaluation application handles the exceptions triggered by such behaviors. In our experiment, the error handling on the malformed queries had to be done directly in the proxy, by altering the malformed queries to make them valid using dummy data. Figure 7.9 depicts the amount of fuzzed test cases required to discover the various errors. Navigation Errors 20 Number of unique errors found Fuzzed Test Case # Figure 7.9: Amount of fuzzed test cases required to discover errors. We observe that all of the errors are discovered at the start of the experiment, within the first 32 of 5,785 total fuzzed test cases. In contrast with the previous experiment, the navigation errors are all discovered before the login event happens. Accessing the inner content of the web application is possible before being logged in and there are no redirects to the login page as we would expect. As for the previous experiment, all the errors are found by the INSERT fuzz operator. Again, this is due to the fact that this fuzz operator permits to directly request any web page from anywhere in the trace. As in the previous test subject, most navigation errors can be found quickly simply because one can request any page at any position in the trace. It is therefore sufficient to send a single request to uncover those kinds of errors. From a security perspective, this web application is vulnerable, as any action can be executed by the attacker, at any point in time, due to the numerous missing access-control checks. A few examples of what can be achieved are described below: 52

63 7.5 Evaluation of H3 An attacker can create a new user and give it administrator rights without being logged in. An attacker can modify all the information of an existing user without being logged in. When a document flow has been created, anyone can validate the step of another person without being logged in. An attacker can delete a document flow even without read-access to the aforementioned flow.... As a conclusion to this experiment, we see that when dealing with broken web applications, fuzz testing and checkpointing are not bringing any additional benefits compared to standard testing approaches, as most of the problems can be uncovered without needing huge traces of requests. TomatoCart The results of the experiment on the TomatoCart application differ from the ones of the previous two experiments. As mentioned earlier, TomatoCart is a much bigger, mature and well-tested product, when compared to the other two candidates. It is used in multiple online e-commerce applications. Consequently, finding navigation errors and generic errors is far more unlikely and it is a good thing for the TomatoCart application that we did not find any of those errors. To conclude, we observe that our preliminary results using standard fuzz operator for reordering messages and standard generic oracles detect mostly simple errors that do not strongly depend on the application s state. Not surprisingly, such simple errors can be detected with a small number of short test cases. To fully justify the benefits of checkpoint-based testing, one needs to consider more complex fuzz operators and oracles that are able to expose and detect subtle errors that have complex dependence on the state, and thus require more thorough testing. For example, imagine a page requiring 10 previous conditions to be satisfied. There are 10 checks executed sequentially to allow the page to be displayed. The 10 th check might be faulty and leading to a generic error. In this case, the trace should be long enough to fulfill the first nine conditions if we aim at finding the error triggered by the 10 th one. Consequently, checkpointing would make sense. 53

64

65 8 Conclusion We proposed Checkpointing-Based Testing, a novel approach for efficiently testing web applications. We developed WebCheck, a Checkpoint-Based testing tool. We discussed its design and we empirically evaluated its performance and behaviour using several PHP applications. The results show that WebCheck achieves high speedup factors in test execution time compared to state-of-the-art solutions and it can discover numerous unknown vulnerabilities in real-world web applications. WebCheck significantly reduces the test execution time of long test cases that exercise web applications in depth and enables thorough testing of large and complex web applications. A natural next step is to consider more complex oracles, fuzz operators, and larger test subjects, to test for subtle errors that depend on the web application s state. A challenge in applying WebCheck to larger applications is the tedious and errorprone specification of application-specific oracles. An example is defining a website s navigation state machine which is used to detect missing navigation checks. One direction for future work is to enhance WebCheck with inference capabilities so as to reduce the tester effort and enable WebCheck to reason about navigation flaws in an automated way. 55

66

67 Bibliography [1] J. Ansel, K. Arya, and G. Cooperman. Dmtcp: Transparent checkpointing for cluster computations and the desktop. In Parallel & Distributed Processing, IPDPS IEEE International Symposium on, pages IEEE, [2] L. Bautista-Gomez, S. Tsuboi, D. Komatitsch, F. Cappello, N. Maruyama, and S. Matsuoka. Fti: high performance fault tolerance interface for hybrid systems. In Proceedings of 2011 International Conference for High Performance Computing, Networking, Storage and Analysis, page 32. ACM, [3] G. Bronevetsky, D. Marques, K. Pingali, and P. Stodghill. Automated application-level checkpointing of mpi programs. ACM Sigplan Notices, 38(10):84 94, [4] G. Bronevetsky, D. Marques, K. Pingali, P. Szwed, and M. Schulz. Applicationlevel checkpointing for shared memory programs. ACM SIGOPS Operating Systems Review, 38(5): , [5] G. A. Di Lucca, A. R. Fasolino, F. Faralli, and U. De Carlini. Testing web applications. In Software Maintenance, Proceedings. International Conference on, pages IEEE, [6] S. Elbaum, G. Rothermel, S. Karre, and M. Fisher. Leveraging user-session data to support web application testing. Software Engineering, IEEE Transactions on, 31(3): , [7] E. N. Elnozahy, D. B. Johnson, and W. Zwaenepoel. The performance of consistent checkpointing. In Reliable Distributed Systems, Proceedings., 11th Symposium on, pages IEEE, [8] E. N. Elnozahy and J. S. Plank. Checkpointing for peta-scale systems: A look into the future of practical rollback-recovery. Dependable and Secure Computing, IEEE Transactions on, 1(2):97 108, [9] P. Godefroid, M. Y. Levin, D. A. Molnar, et al. Automated whitebox fuzz testing. In NDSS, volume 8, pages , [10] S. Hallé, T. Ettema, C. Bunch, and T. Bultan. Eliminating navigation errors in web applications via model checking and runtime enforcement of navigation 57

68 Bibliography state machines. In Proceedings of the IEEE/ACM international conference on Automated software engineering, pages ACM, [11] R. Hammersland and E. Snekkenes. Fuzz testing of web applications. [12] P. H. Hargrove and J. C. Duell. Berkeley lab checkpoint/restart (blcr) for linux clusters. In Journal of Physics: Conference Series, volume 46, page 494. IOP Publishing, [13] E. Hieatt and R. Mee. Going faster: Testing the web application. Software, IEEE, 19(2):60 65, [14] S. McAllister, E. Kirda, and C. Kruegel. Leveraging user interactions for indepth testing of web applications. In Recent Advances in Intrusion Detection, pages Springer, [15] J. S. Plank, M. Beck, G. Kingsley, and K. Li. Libckpt: Transparent checkpointing under unix. Computer Science Department, [16] Z. Qian. Test case generation and optimization for user session-based web application testing. Journal of Computers, 5(11): , [17] F. Ricca and P. Tonella. Analysis and testing of web applications. In Proceedings of the 23rd international conference on Software engineering, pages IEEE Computer Society, [18] P. Tsankov, M. T. Dashti, and D. Basin. Secfuzz: Fuzz-testing security protocols. In Automation of Software Test (AST), th International Workshop on, pages 1 7. IEEE, [19] M. Woo, S. K. Cha, S. Gottlieb, and D. Brumley. Scheduling black-box mutational fuzzing. In Proceedings of the 2013 ACM SIGSAC conference on Computer & communications security, pages ACM,