Institutionen för systemteknik

Transcription

1 Institutionen för systemteknik Department of Electrical Engineering Examensarbete Measuring Accuracy of Vulnerablity Scanners An Evaluation with SQL Injections Examensarbete utfört i Datateknik vid Tekniska högskolan vid Linköpings universitet av Alexander Norström LiTH-ISY-EX--14/4748--SE Linköping 2014 Department of Electrical Engineering Linköpings universitet SE Linköping, Sweden Linköpings tekniska högskola Linköpings universitet Linköping

2

3 Measuring Accuracy of Vulnerablity Scanners An Evaluation with SQL Injections Examensarbete utfört i Datateknik vid Tekniska högskolan vid Linköpings universitet av Alexander Norström LiTH-ISY-EX--14/4748--SE Handledare: Examinator: Dr. Teodor Sommestad FOI, Totalförsvarets forskningsinstitut, Linköping Jan-Åke Larsson isy, Linköpings universitet Linköping, 14 mars 2014

4

5 Avdelning, Institution Division, Department Avdelningen för Informationskodning Department of Electrical Engineering SE Linköping Datum Date Språk Language Svenska/Swedish Engelska/English Rapporttyp Report category Licentiatavhandling Examensarbete C-uppsats D-uppsats Övrig rapport ISBN ISRN LiTH-ISY-EX--14/4748--SE Serietitel och serienummer Title of series, numbering ISSN URL för elektronisk version Titel Title Mätning av noggrannhet bland sårbarhetsskannrar Measuring Accuracy of Vulnerablity Scanners Författare Author Alexander Norström Sammanfattning Abstract Web application vulnerabilities of critical are commonly found in web applications. The arguably most problematic class of web application vulnerabilities is SQL injections. SQL injection vulnerabilities can be used to execute commands on the database coupled to the web application, e.g., to extract the web application s user and passwords data. Black box testing tools are often used (both by system owners and their adversaries) to discover vulnerabilities in a running web application. Hence, how well they perform at discovering SQL injection vulnerabilities is of importance. This thesis describes an experiment assessing detection capability for different SQL injection vulnerabilities under different conditions. In the experiment the following is varied: SQL injection vulnerability (17 instances allowing tautologies, piggy-backed queries, and logically incorrect queries), scanners (four products), exploitability (three levels), input vector (POST/GET), and time investment (three levels). The number of vulnerabilities detected is largely determined by the choice of scanner (30% to 77%) and the input vector (71% or 38%). The interaction between the scanner and input vector is substantial since two scanners cannot handle the POST-vector at all. Substantial differences are also found between how well different SQL injection vulnerabilities are detected and the more exploitable variants are detected more often, as expected. The impact of time spent with the scan interacts with the scanner - some scanners required considerable time to configure and other did not and as a consequence the relationship between time investments to detection capabilities is non-trivial. Nyckelord Keywords SQL Injection, Vulnerability Scanning

6

7 Abstract Web application vulnerabilities of critical are commonly found in web applications. The arguably most problematic class of web application vulnerabilities is SQL injections. SQL injection vulnerabilities can be used to execute commands on the database coupled to the web application, e.g., to extract the web application s user and passwords data. Black box testing tools are often used (both by system owners and their adversaries) to discover vulnerabilities in a running web application. Hence, how well they perform at discovering SQL injection vulnerabilities is of importance. This thesis describes an experiment assessing detection capability for different SQL injection vulnerabilities under different conditions. In the experiment the following is varied: SQL injection vulnerability (17 instances allowing tautologies, piggy-backed queries, and logically incorrect queries), scanners (four products), exploitability (three levels), input vector (POST/GET), and time investment (three levels). The number of vulnerabilities detected is largely determined by the choice of scanner (30% to 77%) and the input vector (71% or 38%). The interaction between the scanner and input vector is substantial since two scanners cannot handle the POST-vector at all. Substantial differences are also found between how well different SQL injection vulnerabilities are detected and the more exploitable variants are detected more often, as expected. The impact of time spent with the scan interacts with the scanner - some scanners required considerable time to configure and other did not and as a consequence the relationship between time investments to detection capabilities is non-trivial. iii

8

9 Contents 1 Introduction Purpose Outline SQL Injections and Vulnerability Scanners Different types of web attacks SQL injections Tautologies Incorrect Queries Piggy-Backed Queries Union Queries Stored Procedures Inference Alternate Encodings Scope of this study Variables influencing vulnerability scanning effectiveness Actions covered Time, experience, and tuning of scanners Protections techniques used by the application Scanner/Scan-method Scan barriers Input vector Database management system Previous studies of web and SQL vulnerability scanning Method Experimental design Dependent variables Independent variables SQL Injection Vulnerabilities Input vector Difficulty of Exploitation Scanner v

10 vi Contents 3.4 Nuisance variables Experience and time investment Choice of Database management system Scan Barriers Exploited Action Result Difficulty of Exploitation Scanner Input vector SQL vulnerabilities Interactions and variables importance The importance of input vector when comparing difficulty and scanners Difference in Detection rate between Vulnerabilities Discussion Summary of variable influence Input vector Scanner Vulnerabilities Difficulty of Exploitation Time Threats against Validity and Reliability Vulnerabilities Constant Database The choice of scanners Experience and skill Inconsistent scanning behavior Results of Input Sanitation versus Pattern matching Other variables not included Advice to practitioners and developers Consider POST Scanners Think about inconsistent behaviors Time is money Look at other scanners Plan beyond SQL injections Implement continuous scanning and automated testing Check for false positives Conclusion 37 A Complete Table of Observations 41 B Grouped tables of Observations 51 Bibliography 55

11 1 Introduction The web applications of tomorrow are all about data, collecting data from databases and processing it into new information for fun and profit. If attackers can get access to and influence this data, they can take the fun and profit for themselves. With the increasing popularity for the web as a platform for applications the threat posed against web applications need to be seriously considered. According to OWASP, a worldwide organization focused on improving the security of software, the biggest threat against web application are injection attacks, such as SQL (Structured Query Language) injections and cross-site scripting OWASP [2013]. Verizon states that web application is a popular target in larger organizations, 56% of all recorded breaches was against a web application and they account for 39% of the compromised records Verizon [2012]. As such the overall security of web applications is considered poor. SQL injection attacks is by far the most common and dangerous vulnerability in web applications. OWASP publishes a top 10 list of vulnerabilities every third year. In the top 10 report from 2007 SQL injection attacks scored a second place, but in the two following reports for 2010 and 2013 injection vulnerablities is now ranked first OWASP [2013]. In addition SQL injections are also the higest ranked category in the SANS institute list of the 25 most dangerous software errors Martin et al. [2011]. Fortunately, there are tools available that can be used to detect these vulnerabilities before they are exploited by an adversary. A vulnerability scanner can detect major problems, which in turn can be mitigated by the security team. However, these vulnerability scanners are not omniscient. Thus we need to know more about what these tools can and what they cannot do, in order to make some assumption on what we can use them for and to what extent they can be trusted. Studies of web application vulnerability scanners have shown that there is quite 1

12 2 1 Introduction a difference in the results between scanners. It has also been shown that scanners produce a large percentage of false positives and that they leave many vulnerabilities undetected Bau et al. [2010] Doupé et al. [2010] Fonseca et al. [2007]. These previous studies have looked at several types of vulnerabilities in web applications, such as: Cross-Site Scripting, Cross-Channel Scripting, Cross-Site Request Forgery, SQL Injections and Malware Presence. In this thesis we will focus on SQL injection because they represent the most common and dangerous vulnerabilities found in web applications and because these weaknesses in software system are easily and often exploited by attackers. In addition, SQL injection vulnerabilities have a low remediation cost and detection of them is quite easy, making vulnerability scanners an interesting defensive tool Martin et al. [2011]. This thesis describes an experiment in which a web application was constructed with a known set of vulnerabilities, and where the vulnerabilities were verified in advance. We chose to construct a new application to perform test against instead of using and existing to have better control over the constraints and vulnerabilities it included. The application was then used to perform a set of controlled experiments where different vulnerability scanners were tested against the application. The scanners are then evaluated based on which vulnerabilities they detected. The vulnerabilities present in the web application were chosen to represent a wide range of different conditions that could be present in real web applications. The result of these experiments will show how the detection capabilities depends on these conditions and the scanner in use. 1.1 Purpose The purpose with this thesis is to describe the current state of web vulnerability scanners when it comes to detecting SQL injections in web applications. Accordingly, our research question is to know to what extent we can rely on vulnerability scanners to discover and mitigate these vulnerabilities in our web applications under different conditions. The scope is limited to scanners that can be used without access to the source code of the application. This scanners are of particular interest as they report the same result for both the product owners and adversaries. 1.2 Outline The second chapter introduces concepts and background facts about SQL injections and vulnerability scanners. It establishes the taxonomy used later to define the experiment in the third chapter. The method used and definition of the experiment is described in third chapter. The forth chapter follows with the results obtained from the experiment. It presents the results from individually studying each of the independent variables that was varied during the experiment. Following that is an analysis of the interactions between the variables and a general discussion about the results is presented in the fifth chapter. The last chapter

13 1.2 Outline 3 then concludes and presents the important finding in this thesis. The rest is Appendixes containing data tables and references used in the thesis.

14

15 2 SQL Injections and Vulnerability Scanners In this chapter we give some background into the field of SQL Injections and Vulnerability Scanners. We also introduces the classification taxonomies which are used throughtout the thesis. 2.1 Different types of web attacks This section covers how different types of web attacks can be classified based on their attack life cycle. The taxonomy by Álvarez and Petrović [2003] defines the life cycle for a web attack in a structured and logical way. A taxonomy is a classification scheme that partitions a body of knowledge and defines the relationship between the parts Howard and Longstaff [1998]. The suggested life cycle is split into nine categories: 1. Entry point: The targeted application system. Our target is the web application itself. 2. Vulnerability: The weakness in the system that allows unauthorized actions. They define 5 types: Code injection, Canonicalization, HTML manipulation, Overflows and Misconfigurations. We are only interested in the subset of Code injection namely SQL injections. 3. Service (under threat): What type of security service the attack poses a threat against. This classification is of no interest to us. 4. Action: What vulnerability type the attack exploits. 5. Length: Number of arguments passed in the request. Can be Expected or Unexpected. Used to trigger buffer overflows and are therefore of no 5

16 6 2 SQL Injections and Vulnerability Scanners interest to us. 6. HTTP element: The user input fields used to perform the attack. 7. Target: The aim of the Attack. Can be either the web application or the platform. Only web applications is of interest where application data and functionality will be affected. 8. Scope: Whom the attack affects, such as a single user or a group of users. Can be either local (impact limited to single or small group of users) or universal (All users are affected, e.g. database manipulation). Our experiment will focus on Universal attacks. 9. Privileges: If the attack escalates privileges. Only applicable if the targeted service is Authentication which was of no interest therefore neither is this. This taxonomy describes SQL injections from an environmental viewpoint, but do not give a clear classification of the technical aspects involved by the SQL injection. However, it does still provide useful context on the overall vulnerability. It can be used to describe the attack vectors that are used by the attacker to reach his goal. To do this they refer to the concept of an attack life cycle, where life cycle is defined as the chain of events that the attack incurs when passing through the web stack of the application. These nine steps of the life cycle represent the order of commands the attacker has to complete. First, it has to pass the entry point to the application. The entry point would contain a vulnerability, which in turn would threaten a service. By exploiting an action in that service performed by the application using some input vector that is transmitted by the web stack to a target within the scope of the application, gaining some unintended privileges. 2.2 SQL injections SQL (Structured Query Language) is a special-purpose programming language used to query relational database management systems (DBMS) for data. The language is based upon relational algebra and consists of operations for data definition and manipulation. It includes operations for data creation, reading, updating, and deletion. Vendors that use SQL typically extend the language with additional functionality that includes, but are not limited to, schema and access control management. Furthermore, vendors often add procedural elements that make it possible to create programs within the DBMS itself. SQL injection is an attack in which malicious code is inserted into a string that are later passed as a query to the DBMS for parsing and execution. The attack is performed by including additional input that is crafted to resemble SQL statements into the entry fields of the web application. If there is a SQL injection vulnerability this input is then processed by the web application in an unsecure manner and the crafted input is assembled with the real SQL query to change the behavior of the web application. As an example of a SQL injection, let s consider the following SQL statement (\ denotes the line continues without linefeed):

17 2.2 SQL injections 7 SELECT * FROM users \ WHERE username = Alice AND password= secret This SQL statement tries to fetch a row from the DBMS where the username equals Alice and her password is secret. Let s say that if, both these fields are true the DBMS will return the user record for Alice and allow Alice to log on to the web application. If the attacker can provide the value for the username and password parameter in the example via a login page (i.e., change Alice and secret to something else) he would be able to inject his own code into the statement. For instance, the attacker can craft parameters that modify the statement to always return a match for the user Alice. He performs this attack by injecting database escape code into either the username or password parameter. As example, insert a username or password that abruptly ends the SQL statement by marking it as a programmer s comment. If the attacker would enter the username Alice -- the two dashes remove everything to the end of the line and the statement sent to the database would be the following: SELECT * FROM users \ WHERE username= Alice -- AND password= clueless This statement would return any record where the username is Alice, as the double dash (--) in the statement marks the rest of the row as a comment (i.e. something that is not to be executed). Another way the attacker can bypass the check for both fields is to add an additional criteria which would always evaluate to true. This is called a Tautology and will be explained in detail later. Such a statement could be constructed by escaping the password parameter with a single quote (i.e. ) followed by a true statement (e.g. clueless OR 1 = 1). The attacker would then modify the statement to something like this: SELECT * FROM users \ WHERE username= Alice AND password= clueless OR 1 = 1 The statement above would always evaluate to true because the condition for 1=1 will be used whenever the first conditions does not hold. Halfond et al. [2006] proposed a taxonomy for the classification of technical aspects involved in SQL injections. This taxonomy is described below. Alternative classification schemes can be found in Álvarez and Petrović [2003] Fong and Okun [2007] Shin and Williams [2008] Tautologies The goal with these attacks is to inject code into the conditional statements of the SQL query to modify their behavior to always evaluate to either true or false in every possible interpretation. Another name for this attack is Boolean-based attack. The intent behind this type of attack varies, but usually they are used to bypass authentication or to extract data from within the system Halfond et al. [2006].

18 8 2 SQL Injections and Vulnerability Scanners The injections are triggered by injecting invalid input into the conditional fields in the WHERE section of a query, but also be combined with input from other sections of the query such as the GROUP BY or ORDER BY sections. To successfully exploit the system, the injected input has to change the conditional statement of the WHERE-clause to always evaluate to the same result. In other words, the DBMS will interpret the query as either true or false regardless if the provided parameters (like username and password) matches the content of the database or not Incorrect Queries An attacker can try to inject invalid statements into the query so that the database system will disclose information about the underlying database schema and queries. An attack of this sort can be used by the attacker to gather information about the database system for further attacks. The attacker can extract information from the database system with these incorrect queries by observing the behavior of the application when executing them. If the application is not configured to handle errors in quiet way they can return a default error messages when an incorrect query is executed. The error message returned from the system is often so descriptive that the attacker can easily leverage information about injectable parameters in the query so that new exploits can be crafted to utilize these vulnerabilities and fulfill the attacker s intent Piggy-Backed Queries In some systems the database abstraction layer supports the evaluation of multiple SQL queries in each call. By supporting this functionality the developer can invoke several queries at the same time. However, it may also allow an attacker to append entirely different queries that the developer did not intend. In this attack the goal is not to modify the query into performing something different. Instead the attacker modifies the query by inserting additional queries using what is called a Query Delimiter. Therefore allowing the attacker to perform other queries then what is possible with for example tautologies. In the end the database will execute all queries as normal, including the attackers new queries, and therefore it is possible to perform queries that are not usually possible by modifying existing queries. This attack can be very harmful as it allows the attacker to issue any type of query that the current database user has access to execute, such as creating new administration accounts, invoking system commands, or installing backdoors in the database server Union Queries This type of attacks is used to extract data from the database by exploiting a parameter to change the resulting data set from the database. The attacker changes the result by injection a UNION SELECT statement into the existing one. Doing so the attacker append data from other tables and system variables onto the original data set the database would return. Examples include extracting account

19 2.2 SQL injections 9 details such as password or credit card numbers Stored Procedures Some types of database management systems allow for the creation and usage of stored procedures within a database. A stored procedure is a user defined function that has the power to execute statements on behalf of another user. This makes Stored Procedures useful to implement specialized access control. Let s make an example with a booking system, assume the anonymous user may not read or write to the reservation table. In this case the anonymous user can call a stored procedure that is executed as the receptionist user, which has write access, to make a new reservation. However if the stored procedure would be improperly implemented the anonymous user could escalate his privileges to get the receptionist access. An example of such a bad implementation is to concatenate SQL statements and evaluate them inside the stored procedure. The recommended practice is to bind data to variables and use these variables in queries instead. If the stored procedure evaluates SQL statements from strings, an attacker could execute any type of SQL statements in the database. A neat feature of stored procedures is that they can also be used as triggers on database event, such as INSERT INTO, UPDATE, and DELETE on tables. If the attack were to create such a trigger, he could insert a backdoor into the database, and possible collect input data before it s stored to disk. The attacker can then intercept and collect for example plaintext password and other secret data before it is encrypted and stored to disk Inference This attack classification is used by an attacker to discover vulnerable parameters, database schemas, or to extract data. The attack is performed by modifying the SQL statements to perform a specific action only if the condition evaluates to true or false. Even if the attacker cannot extract information directly from the database the attacker can use Inference to conclude what the actual data might be, by injecting statements that would trigger some response depending on the value of the actual data. Inference is therefore a useful tool when attacking applications that are configured to not respond with error codes or other messages to signal if an attack was successful or not. In comparison with the Incorrect query attack. Two types of Inference are Blind Injection and Timing Attacks. With a Blind Injection it s possible to deduce if the injected statement was successful based on the response from the server when asking true/false questions. Even if the server does not response with error, the outcome can be inferred by the fact that the response from a site will contain slight different output for a normally functioning page versus a page that is expected to be broken. If, for some reason, the responses would not differ between a functional and a broken page, a time based attack can be used instead. Timing Attacks uses a similar approach as Blind Injection Attacks but instead of deducing information from the true/false responses given by the web application, time consuming queries are given to the web application. By measuring the response time needed by the web application to

20 10 2 SQL Injections and Vulnerability Scanners process the response it is possible to deduce if the attack were successful against the database or not Alternate Encodings Sometimes the application may be developed with defensive coding practices or use other prevention techniques to protect against SQL injections. One technique is to simply reject certain input patterns from the user. An attacker can circumvent this protection schemes by encoding his input in a way avoids detection by these defensive mechanisms. The attacker can to use the same range of Attacks as discussed earlier but he must encode them to syntactically change the format to avoid detection while keeping the same semantics the same. Therefore Alternative Encoding cannot be considered a separate attack. It rather is used as an evasion technique in conjunction with another attack. Alternative Encoding is still very useful and often needed as defensive coding practices often include to scan input for certain bad characters to sanitize the input data. 2.3 Scope of this study The taxonomies presented in this chapter are not fully compatible with the needs we have in the experiment. To handle this we have chosen the most important different parts we need for the experiment from theses taxonomies. In relation to the life cycle model presented above we re going to treat those categories as follows in this thesis: 1. Entry point: Only the web application is targeted. 2. Vulnerability: Only SQL injections are targeted. 3. Service (under threat): Don t care. 4. Action: Don t care. 5. Length: Don t care. 6. HTTP element: Used to describe the user input needed to trigger the vulnerability in our experiment. 7. Target: Only the web application is targeted. 8. Scope: We assume Universal scope for the experiment. 9. Privileges: Don t care. The step in the life cycle is complemented with the taxonomy for SQL injection attacks presented in sections We consider SQL injection attacks based on Tautologies, Incorrect Queries, Piggy-Backed Queries, and Union Queries. In summary, this thesis has the following scope.

21 2.4 Variables influencing vulnerability scanning effectiveness 11 Entry point Vulnerability HTTP elements Target Scope Web application Tautologies Incorrect Queries Piggy-backed Queries Union Queries GET/POST/Query parameters. Web application Universal impact 2.4 Variables influencing vulnerability scanning effectiveness A web vulnerability scanner is a tool to automatically examine web applications for security faults Fong and Okun [2007]. The tool can either search for application specific vulnerabilities using fingerprinting of the system component and compare the result against a vulnerability database or try the more aggressive method with probing the application for design and coding errors such as illegal input and buffer overflows. These tools can perform a number of steps when analyzing web applications. The vulnerability scanner maps the structure of a web application by crawling through its web pages and examining each page for input sources, links, and other relations. When the structure of the application has been explored the scanner will start a penetration test against the input vectors discovered during the exploration face. A penetration test is an active analysis of the web application that injects carefully crafted input into the various input sources that was discovered. The applications will interpret the input and produce output accordingly. The vulnerability scanner will observe the output of the application during these injections and deduce information about potential vulnerabilities based on the applications response. This makes a vulnerability scanner useful for purposes such as searching, identifying and mitigating vulnerabilities in new and existing applications. In the following sections factors believed to influence their detection capabilities are described Actions covered Web application may contain several different actions that can be performed. The taxonomy from section 2.1 introduces the following Action types: Read, Modify, Delete, Fabricate, Impersonate, Bypass, Search, Interrupt, Probe, and Other for all those types that do not fall under the first nine classes Álvarez and Petrović [2003]. If a scanner should be able to detecting vulnerabilities in web applica-

22 12 2 SQL Injections and Vulnerability Scanners tions they would need to have least one scanning method that can detect each of these cases Time, experience, and tuning of scanners It is generally assumed that the operator s previous experience and time invested in using a particular scanner will yield better scanning results Acunetix [2012]. An experienced operator should outperform an inexperienced operator when given the same task and the same tool Protections techniques used by the application If the application use some protection technique to mitigate an attack against the application this may influence the scanner s effectiveness. For example, an application can be implemented with filters that will prevent or treat certain keywords in the input parameters differently from expected input. There are two issues with this. First the scanner will not be able to detect application logic that depends on a keywords defined in the application to be triggered, as the scanner could possible not know these keywords unless the source of the application was analyzed before the scan. The other problem is that while the application now can protect itself from some injection attacks, it can still be vulnerable to other attacks since it would be impractical to predict every possible input that could trigger a vulnerability in the entire system. Therefore we assume that the exploitability (difficulty) to trigger a vulnerability in the application is related to the scanners ability to test for and analyze how input is handled by the application Scanner/Scan-method The method and techniques used by the scanner to probe the application is expected to be of some importance. The different methods include white-box testing which can utilize static analysis methods such as Context free grammars, use of secure coding practices, lexical analysis, data-flow analysis. taint analysis, white box testing with static analysis(e.g. Pixy Jovanovic et al. [2006]), and black-box testing (e.g. SecuBat Kals et al. [2006]) where the scanners searches for vulnerabilities by mapping the application and comparing response against a signature database by either using know attacks or probing unexpected behavior with different techniques (e.g. fuzzing) Huang et al. [2004] Scan barriers Web application may have application specific barriers that can be confusing and hard for the scanner to understand. Scan barriers are not a protection technique for the system, it is application logic that normal users can and will handle with ease, but could require extra supervision when used with a scanner. For example, typical scan barriers include requirements for the users to authenticate with the application before use, or answering challenges when submitting data.

23 2.5 Previous studies of web and SQL vulnerability scanning Input vector The scanner has to support the attack vectors that are required to trigger vulnerabilities in the application. This can be support for different input methods in the application like the HTTP verbs presented in the taxonomy but also less used HTTP verbs such as PUT, and DELETE. At the same time it can also be required to craft and analyze header responses from the server. All these different input vectors change the attack vector the scanner has to use to trigger the vulnerability in the system. Changing the attack vector would require the scanner to perform the scan with other input variables using a different input source. The input variables are sent and received at the endpoint of the web application. Depending on the scanners choice of data submission the application will perform different tasks Database management system The DBMS that are used in the system could influence the result of detected vulnerabilities by the scanners when test against specific vendors are conducted. For example, the language used when talking to a DBMS is SQL, and standardized portion of SQL as language is very limited in its capabilities. Therefore different vendors have extended SQL with own functionality that is specific to their product. This functionality, and scanners way of handling it, may influence the detection capabilities. 2.5 Previous studies of web and SQL vulnerability scanning Bau et al. [2010] evaluated eight black-box scanners effectiveness in detecting vulnerabilities. They found that scanners, in general, are adapted to detecting straightforward historical vulnerabilities which have been found in popular applications or textbooks. In addition they found that black-box scanners show room for improvement in other cases such as advanced and second-order forms of SQL injections. They highlight that low detection rates for the advanced types of SQL injections may be due to more systematic flaws in the scanners themselves and discussed a few ways that scanners could be improved for these vulnerability types, such as novel and non-standard keywords. They also reason about the low coverage results and false positives, have to do with active content and scripting languages on the site such as Silverlight, flash, java applets, and JavaScript. A previous study by Fonseca et al. [2007] found that scanners vary in their result and they recommended using several scanners when searching for vulnerabilities in a web application. The study also finds that the best vulnerability scanner for SQL injection was not the best vulnerability scanner for Cross-site scripting and vice versa. Furthermore, they concluded that there was a high rate of false positives in their experiment while at the same time saying that the scanners failed to detect several vulnerabilities while not presenting any results on what

24 14 2 SQL Injections and Vulnerability Scanners these vulnerabilities are. Several other studies have focused on benchmarking different vulnerability scanners to compare their effectiveness, see Bau et al. [2010], Halfond and Orso [2006], Shin and Williams [2008], Elia et al. [2010]. One issue with some of these benchmarks, is that they don t go into any details regarding the vulnerability types that are used in the tests or the detection rate for them. For example, the study performed by Bau et al. [2010] shows that the average vulnerability detection rate for SQL injections were 21.4% and relates that result to other vulnerability categories (Malware, Information leakage, configuration error, session management SQL injections, CSRF, XCS, XSS). Thus, it is largely unknown how different factors influence the detection rate you get when you search SQL injections. The tools used to test for vulnerabilities are new and are immature Curphey and Arawo [2006]. The experiment presented in this thesis provides more recent data on the effectiveness of vulnerability scanners and address some of the issues not handled in the tests described above. For example, the experiment investigates different types of SQL injection vulnerabilities. This experiment will focus on assessing the detection capabilities of vulnerability scanners when tested against a custom-build web application. In this setup we re able to vary the vulnerabilities, scanners, difficulty of exploitation, input vectors, and time investment used for each test.

25 3 Method To answer the research question presented in section 1.1, a series of experiments was performed. The focus of these experiments was to evaluate how good black box web vulnerability scanners are on detecting different types of SQL injections in web applications. 3.1 Experimental design The most effective way to test vulnerability scanners is to run them against applications that we already have analyzed Curphey and Arawo [2006]. By doing so we both test that the vulnerability scanner have the capacity to detect vulnerabilities in the application, we can make a direct comparison between the given result by the scanner against the expected result which is known in advance. In this experiment these scans are performed against a custom-built web application in which we have carefully chosen and validated the presence of all vulnerabilities so that they actually are exploitable. The dependent variable which will be measured is the number of correct alarms about detected vulnerabilities in the web application. The experiment will follow a univariate analysis procedure. Basically only the change in detection rate is observed when one of the independent variables is changed while keeping the other constant. The tests are repeated several times with different values for each of the independent variables every time. In this experiment the dependent variable is the number of correct alarms produced and the independent variables are: SQL injection vulnerability, input vector, difficulty of exploitation and scanner Keppel and Wickens [1973]. There are a number of nuisance variables that could influence the results but 15

26 16 3 Method Table 3.1: Experiment variables Variable Type Treatment Detection rate Dependent Measured by comparing reports to known vulnerabilities. SQL Injection vulnerabilities Input vector Difficulty of Exploitation Scanner Qualitative independent Qualitative independent Qualitative independent Qualitative independent Exhaustive list: piggy-backed queries, tautologies, alternative encoding and logically incorrect queries. Alternate HTTP verbs: GET, POST Three levels: Easy, Medium, Hard Selected alternatives: w3af, IBM AppScan, Netsparker, and Acunetix Time investment Nuisance The scanner is prepared according to its manual. The time required is noted. Experience Nuisance The order of scanners usage are randomly selected for each vulnerability DBMS Nuisance Kept constant (MySQL) Scan Barriers Nuisance Kept constant (no barriers in test) which are of no particular interest for this experiment. In other words, this experiment will not assess how they influence the number of correct alarms about found vulnerabilities. The known nuisance variables (experience, DBMS, barriers, exploited action, time investment) are treated by either repeatedly assigning a random value to them value or keeping them constant during the experiment. For example, we make the assumption that the user of the tools will be the development team itself we assume the user in this case can configure the tools to scan accordingly to the underlying implementation details of the tested system. All relevant variables and treatment in the experiments are summarized in table Dependent variables The experiments measure one or more dependent variables responds to the changes in the other (independent) variable. In this experiment we expect the dependent variable to pass from one form to another when we make changes to the independent variables. In our particular case the dependent variable is the number of cor-

27 3.3 Independent variables 17 rect alarms that the scanners has detected during their scans, i.e. the detection rate. This can be measured directly by comparing the reported vulnerabilities with the list of known vulnerabilities in the test bench. 3.3 Independent variables The independent variables describes variables of interest for the experiment which are assumed to have influence on the dependent variable. The independent variables in this experiment are: SQL injection vulnerability, Input vector, Difficulty of Exploitation, and Scanner. The first three are related to the system being scanned and the other two are related to the scanning procedure. How they are treated is described in the following sections. In an experiment the independent variables needs to be controlled and varied. How this is done is described below SQL Injection Vulnerabilities To better understand and distinguish between different SQL injection types we need a system for classifying the different SQL injections we are describing. The types of vulnerabilities to include in the tests are chosen according to the selected taxonomy which is based on the attackers Intent, the Input Source, and the Technical Aspect of the attack itself Sun et al. [2007]. In the experiment we can influence the vulnerabilities that are tested by the scanners. The chosen vulnerabilities are implemented as test cases in the experiment. The vulnerability scanners will try to exploit these test cases in order to trigger vulnerabilities in the application. All vulnerabilities in the experiment were complemented with tests containing working exploits that automatically could verify the presence for each of vulnerabilities in the system. These tests therefore define the baseline for positive vulnerabilities in the application when evaluating the scanners. The vulnerabilities are chosen to require the scanners to performed SQL injections by exploiting the application with the following SQL injection classes: Vulnerability type Description Piggy-Backed Queries See section Tautologies See section Alternate Encoding See section Illegal/Logically Incorrect Queries See section In addition to these for SQL injections classes the vulnerabilities are chosen to represent different functional goals and to have different injectable data types (Text, number and date fields). For complete classification of all vulnerability cases, see Table B.2.

28 18 3 Method Input vector The scanner has to probe an input source to reach the vulnerable part of the application. The input vectors can be varied according to: injections through user input, cookies, server variables, or second-order injection (injecting data into the application that are used in an insecure manner at a later execution stage). The input vectors can be controlled as a qualitative independent variable where we can vary how the exploit should be sent to reach the vulnerability. The variable is limited to only GET and POST verbs. This limitation come from the fact that data submission using cookies and second-order injections will add complexity to the application, thus adding a scan barrier to the application. Additionally, server variables can only be changed by reconfiguring the web server and such tasks are out of the scope of the experiment. Only testing GET and POST verbs is still reasonable, together they represent the most common way to submit user input from forms and links to a web application. Input type GET POST Description The GET method is used to retrieve information from a specified URL. The GET method should not have side effects that can alter the state of the application. Therefore we should not be required to roll back the application state for each test. The GET method accepts user input in the form of a query string (URLs which contains a? ). The POST method is used to submit data to the application. The origin server should perform the action requested by the POST method and redirect the client to the appropriate result Difficulty of Exploitation The exploitability (i.e. the difficulty of triggering a vulnerably with an exploit) can be introducing a layer of protection mechanisms in the application. We can expect that the number of detected vulnerabilities is dependent on how easily the application can be exploited. By adding increased protections to the vulnerabilities we see how the scanner responds to increased difficulty and limitations. When sanitization is used the following rules apply. The input is sanitized by escaping special characters (e.g. apostrophes). Characters used for comment are removed from the input. This included the character / (forward slash) and * (asterisk). A sequence of - (dashes) are replace into a single - to protect against the line comment -- (double dash). We also remove SQL keywords from the input. Keywords include EXEC, EXECUTE, SELECT, INSERT, UPDATE, DELETE, UNION, JOIN, CREATE, ALTER, DROP, RENAME, TRUNCATE, BACKUP, and RESTORE. Limiting the use of these keywords should protect the system from the most common formats of SQL Injections. This would require the scanner to perform some sort of alternative coding or fuzzing techniques that would circumvent the protection

29 3.4 Nuisance variables 19 mechanism. The protection mechanism is designed to be sufficient but not totally secure. Therefore allowing a specially crafted input to bypass the security check and still trigger the vulnerability. The tests are classified in three different levels, in an ascending order where Hard is the most difficult to exploit: Difficulty class Easy Medium Hard Description Easy difficulty means that the system does not have any additional security constraints. The medium level introduces input sanitizing in the system as described above. In addition to the input sanitization introduced by the medium difficulty, the hard difficulty also applies pattern matching to the input making which will require considerable effort from the attacker to pass SQL statements into the system without alternating the encoding first Scanner The scanners themselves should also be considers as a variable. This makes them a qualitative independent variable. The reason behind this is that there are a lot of factors that are associated with scanners. Such as: usability, scanning method and techniques, vulnerability signatures, and so on. As these factors cannot be tested independently due to issues of isolating and separating them, the entire scanner is treated as an independent single entity. A reasonable assumption is therefore that the scanners will differ in their capabilities and scanning techniques. These extra facts could give a particular scanner an advantage against the other scanners. These scanners chosen for evaluation in this experiment are: w3af, IBM App- Scan, Netsparker, Acunetix. This list of scanners may seems small, putting w3af aside these three are the only commercial scanners that we had available at full versions when the test was performed. 3.4 Nuisance variables There are factors that influence the value of the dependent variables other than those we are interested in. These factors are called nuisance variables and could cause variation in the outcome of the experiments Keppel and Wickens [1973] Experience and time investment As previously stated the results are depending on the previous experience an operator has in the scanning field. If the operators have good previous knowledge

30 20 3 Method of how vulnerability scanners work he can probably scan the application better than someone how doesn t have that prior experience. To handle this issue the order of scanners to use at is chosen at random and the vulnerabilities is divided into smaller sets and then these individually subsets is also scanned at random. These steps are repeated until all combinations have been tested and then the results are finalized. By selecting the order of scanners at random, it s assumed the operator will have roughly the same prior experience regardless of which scanner he operates. By also choosing the set of vulnerabilities to scan at random any specialized experience the operator can gain with a particular kind of vulnerabilities will be rendered useless for the next. This procedure was repeated until all scanners have been tested against all vulnerability sets, with equal amount of time investment. It is possible to improve the result of a scanner by investing more time into tuning the scanner to the scanned system and provide customized instructions to it. The strategy for controlling this variable is to configure all scanners in accordance to their manual, even if it takes a lot of time. Although the time requirement was considered a nuisance variable it was still recorded and is considered a reported as a side result of the study Choice of Database management system This variable is kept constant by using MySQL as DBMS in all tests Scan Barriers The purpose of this experiment is not to test the scope of functionality for a set of scanners but instead to test the ability to detect certain types of vulnerabilities. Therefore scan barriers will not be tested as this would only contribute to additional configuration of the tools on how to interact with the web application. The influence of Scan barriers will therefore not be addressed in this experiment. The variable is kept constant during the experiment in such sense that all possible barriers will be removed from the application. In other words, no authentication, logic or proxies are to be present between the scanner and the vulnerabilities Exploited Action The goal of an exploit is to perform some action upon the application. An Action could for example be a read operation, impersonation of a user or interruption of service. The scanner will not support all of these possible action types and knowing what all these action types are in advance is not reasonable, as every attacker will have different goals. Nevertheless we need to design test cases that cover the most typical action types, such as read operations which is a threat against confidentially and write operations which threatens the integrity. This issue was addressed by including at least one test case that will cover each typical operation.