Automated Test Generation from Vulnerability Signatures

Transcription

1 Automated Test Generation from Vulneraility Signatures Adulaki Aydin, Muath Alkhalaf, and Tevfik Bultan Computer Siene Department University of California, Santa Barara Astrat We appliations need to validate and sanitize user inputs in order to avoid attaks suh as Cross Site Sripting (XSS) and SQL Injetion. Writing string manipulation ode for input validation and sanitization is an error-prone proess leading to many vulnerailities in real-world we appliations. Automata-ased stati string analysis tehniques an e used to automatially ompute vulneraility signatures (represented as automata) that haraterize all the inputs that an exploit a vulneraility. However, there are several fators that limit the appliaility of stati string analysis tehniques in general: 1) undeidaility of stati string analysis requires the use of approximations leading to false positives, ) stati string analysis tools do not handle all string operations, ) dynami nature of the sripting languages makes stati analysis diffiult. In this paper, we show that vulneraility signatures omputed for delierately inseure we appliations (developed for demonstrating different types of vulnerailities) an e used to generate test ases for other appliations. Given a vulneraility signature represented as an automaton, we present algorithms for test ase generation ased on state, transition, and path overage. These automatially generated test ases an e used to test appliations that are not analyzale statially, and to disover attak strings that demonstrate how the vulnerailities an e exploited. I. INTRODUCTION Corretness of input validation and sanitization operations is a ruial prolem for we appliations. One of the main forms of interation etween a user and a we appliation is through text fields. The text entered y the user is parsed y the we appliation and used as the input parameter for the ation that is exeuted in response to the user s request. During ation exeution, user input an e passed as a parameter to seurity sensitive operations suh as sending a query to the ak-end dataase. If the input sent y the user inserts unintended ommands to the generated dataase query (whih is alled SQL injetion), then seurity of the appliation an e ompromised resulting in unauthorized aess to sensitive data or loss of data. In another attak senario, alled Cross Site Sripting (XSS), a user sends an input that stores maliious ode in the dataase, that an later e used for attaking other users mahines. Even for input fields whih are not entered as text fields (suh as inputs that are entered using a drop ox), a maliious user an hange the input field and insert an attak y manipulating the http request that is generated y the rowser. In order to ensure the seurity of a we appliation, the This researh is supported in part y NSF grants CCF and CNS Muath Alkhalaf is funded in part y a fellowship from the King Saud University. user inputs that flow into seurity sensitive funtions like dataases queries must e orretly validated and sanitized. Unfortunately, we appliations are notorious for seurity vulnerailities suh as SQL injetion and XSS that are due to lak of input validation and sanitization, or errors in string manipulation operations used for input validation and sanitization. In this paper, we present an automated testing framework that targets testing of input validation and sanitization operations in we appliations for disovering vulnerailities. Our framework omines automated testing tehniques with stati string analysis tehniques for vulneraility analysis [1]. We use stati string analysis to otain an over-approximation of all the input strings that an e used to exploit a ertain type of vulneraility. This set of strings is alled a vulneraility signature, whih ould e an infinite set ontaining aritrarily long strings. For speifiation of different types of vulnerailities we use attak patterns developed y seurity researhers. These are regular expressions that haraterize the strings that would ause a vulneraility when sent to a seurity sensitive funtion. Given an attak pattern and a we appliation, we use automata-ased string analysis tehniques to generate an automaton that orresponds to the vulneraility signature for that appliation for the type of vulneraility haraterized y the attak pattern. As input we appliations, we use the delierately inseure we appliations that are developed y seurity researhers to demonstrate different types of programming praties that lead to vulnerailities. Using the vulneraility signature automata generated y analyzing the delierately inseure we appliations, we automatially generate test ases ased on three overage riteria: state, transition and path overage. Eah test ase orresponds to a string suh that, when that string is given as a text field input to a we appliation, it may exploit the vulneraility that is haraterized y the given vulneraility signature. Our automated test generation algorithm tries to minimize the numer of test ases while ahieving the given overage riteria. In order to demonstrate the effetiveness of our approah we experimented on several real-world we appliations. As we report later in the paper, the automatially generated test sets were very effetive in identifying vulnerailities in these appliations. The rest of the paper is organized as follows. In Setion II we give an overview of our approah. In Setion III we review

2 Attak patterns Delierately inseure we appliations Automata-ased Stati String Analysis Forward analysis for vulneraility detetion Bakward analysis for vulneraility signature generation Vulneraility signature automaton SCC identifiation + DAG onstrution Automata-ased Test Generation Min-over paths algorithm Depth-firsttraversal + SCC entry and exit overage SCC overage Test set for state overage Test set for transition overage Test set for path overage Figure 1. Automated Test Generation from Vulneraility Signatures the vulneraility signature generation tehniques we use. In Setion V and VI we disuss the test generation algorithms we use. In Setion VII we show the experimental results of our approah. In Setion VIII we disuss the related work, and we onlude the paper in Setion IX. II. MOTIVATION AND OVERVIEW The high-level flow of our automated testing framework for input validation and sanitization funtions is shown in Figure 1. In this setion we give an overview of different aspets of our approah, efore explaining the tehnial details in the following setions. A. Automata-ased Stati String Analysis Our automated testing framework generates test ases from vulneraility signatures. A vulneraility signature is a haraterization of all user inputs that an exploit a vulneraility. In our framework we use automata-ased string analysis in whih vulneraility signatures are represented as automata. Automata-ased string analysis is a stati program analysis tehnique. Given a set of input values represented as automata, it symolially exeutes the program to ompute the set of string values that an reah to eah program point. Using a forward-analysis that propagates input values to sinks (i.e., seurity sensitive funtions), it is possile to identify attak strings that an reah to a given sink. Then, a akward analysis that propagates the attak strings ak to user input results in an automaton that orresponds to the vulneraility signature. Automata-ased stati string analysis is hallenging due to several reasons. Due to undeidaility of string verifiation prolem, string analysis tehniques use onservative approximations that over-approximate the vulneraility signatures. Due to these approximations vulneraility signatures may ontain strings that do not orrespond to attaks, leading to false positives. Moreover, string analysis tools only model a suset of availale string lirary funtions, and when an unmodeled lirary funtion is enountered, the funtion has to e over-approximated to indiate that it an return all string values, whih results in further loss of preision. Furthermore, forward and akward symoli exeution using automata an ause exponential low-up in the size of the automata when omplex string manipulation operations suh as string-replae are used extensively. Finally, dynami nature of sripting languages used in we appliation development makes stati analysis very hallenging and appliale to a restrited set of programs. Due to all these hallenges it is not possile to have a push-utton automata-ased string analysis that works for all real-world appliations. In this paper we omine stati vulneraility analysis tehniques with automated test generation. The omined approah ompensates for the weaknesses of the stati vulneraility analysis tehniques. In our approah stati vulneraility analysis is applied to a small set of programs and the results from this analysis is used for testing other appliations. Hene, programs with features that make stati vulneraility analysis infeasile an still e heked using automated testing. Moreover, the approximations that are introdued y stati vulneraility analysis that lead to false positives are eliminated during testing. B. Generating Vulneraility Signatures from Delierately Inseure Appliations Seurity researhers have developed appliations that are delierately inseure to demonstrate typial vulnerailities. These appliations are sometimes used to teah different pitfalls to avoid in developing seure appliations, and sometimes they are used as enhmarks for evaluating different vulneraility analysis tehniques. In our framework we use stati string analysis tehniques to analyze delierately inseure appliations and to ompute a haraterization of inputs that an exploit a given type of vulneraility. In order to generate the vulneraility signature for an appliation, we need an attak pattern (speified as a regular expression) that haraterizes a partiular vulneraility. An attak pattern represents the set of attak strings that an exploit a partiular vulneraility if they reah a sink (i.e., a seurity sensitive funtion). Attak patterns for different types of vulnerailities are pulily availale and an e used for vulneraility analysis. Given an attak pattern and a delierately inseure we appliation, we use automata-ased stati string analysis tehniques to generate a vulneraility signature automaton that haraterizes all the inputs for that appliation that an result in an exploit for the vulneraility haraterized y the given attak pattern. I.e., the vulneraility signature automaton only aepts the strings that are in the vulneraility signature. In the next phase of our approah we automatially generate test ases from the vulneraility signature automaton.

3 C. Automated Test Generation from Vulneraility Signatures Given a vulneraility signature automaton, any string aepted y the automaton an e used as a test ase. Hene, any path from the start state of the vulneraility signature automaton to an aepting state haraterizes a string whih an e used as a test ase. However, a vulneraility signature automaton typially aepts an infinite numer of strings sine, typially, there are an infinite ways one an exploit a vulneraility. In order to use vulneraility signature automata for testing, we need to somehow prune this infinite searh spae. Our overall goal is to minimize the numer of test ases while making sure that we over all possile ways of exploiting a vulneraility. The mehanism that allows an automaton to represent an infinite numer of strings is the loops in the automaton. So, in order to minimize the numer of test ases, we have to minimize the way the loops are traversed. We do this y identifying all the strongly-onneted omponents (SCCs) in an automaton and then ollapsing them to onstrut a direted ayli graph (DAG) that only ontains the transitions of the automaton that are not part of an SCC and represents eah SCC as a single node. Using this DAG struture, we do test generation for three overage riteria: 1) state overage where the goal is to over all states of the automaton (inluding the ones in an SCC), ) transition overage, where the goal is to over all transitions of the automaton (inluding the ones in an SCC), ) path overage, where the goal is to over all the paths in the DAG that is onstruted from the automaton, while also overing all possile ways to enter and exit from an SCC. We implement the state and transition overage using the min-over paths algorithm that we exeute on the DAG representation followed y a phase where we ensure the overage of the states and transitions inside the SCC nodes. We implement the path overage using depth-first-traversal, where, when an SCC node is enountered, we ensure that all entry and exit ominations are overed in the generated test ases. D. A Sanitization Example One of the well-known XSS attak strings is the following: <sript>alert( XSS )</sript> The sript-tag indiates exeutale ode and a maliious user might e trying to store a maliious sript to e exeuted on another user s mahine later on. Now, onsider the example ode in Figure extrated from a delierately inseure we appliation. This ode is sanitizing the input provided y the user for the name field in line 7 y deleting all appearanes of the string <sript> (it deletes it y replaing eah appearane of the string <sript> with the empty string). Later on in the program, the variale $html is used as an input for a seurity sensitive funtion, so if the sanitization is not done properly this appliation would have a vulneraility. We an try to hek if the appliation is vulnerale y testing it with the aove attak string. As expeted the sanitization ode will orretly remove the sript-tag and sanitized input will e alert( XSS )</sript>. So, this test input does not detet a vulneraility. However, this appliation has a vulneraility and the sanitization used in Figure is inorret. 1 <?php if(!array_key_exists ("name", $_GET) $_GET["name"] == NULL $_GET["name"] == ""){ $isempty = true; 4 } else { 5 $html.= "<pre>"; 6 $html.= "Hello "; 7 $html.= str_replae( "<sript>", "", $_GET["name"]); 8 $html.= "</pre>"; 9 } 10?> Figure. A Sanitization Example One an generalize the attak strings for the XSS vulneraility as an attak pattern using the following regular expression: /.*<sript.*>.*/ When we run the automata-ased string analysis on the example shown in Figure, we find out that the intersetion of the set of strings that an reah the sink and the aove attak pattern is not empty, i.e., there are some inputs that will ause a string ontaining the sript-tag reah the sink. So, we generate the vulneraility signature for this appliation whih results in an automaton that ontains 59 states and 850 transitions. Note that, this vulneraility signature automaton aptures the fat that the string-replae operation in line 7 will delete all appearanes of the string <sript> from the input. The reason that there are thousands of transitions is due to the fat that there is a transition for eah ASCII harater from eah state. When we use our automated test generation tehnique to generate a test string from the vulneraility signature automaton, we otain the following test input: <srip<sript>t> When we run the appliation with this input we disover an attak, i.e., the sink funtion reeives an input that ontains the string <sript>. This is due to the fat that the inorret sanitization funtion in Figure deletes the sustring <sript> from the aove test input and reates the attak string. In our framework, we use the test strings generated from vulneraility signatures of delierately inseure we appliations to test other appliations. If the appliations we test ontain sanitization errors similar to the errors in delierately inseure we appliations or if they do not use proper sanitization, then the generated test ases an disover their vulnerailities without analyzing them statially. Note that the test inputs generated from vulneraility signatures an also e used for appliations that are statially analyzale in order to eliminate false positives and onstrut exploits (i.e., to generate onrete inputs that demonstrate how a vulneraility an e exploited). III. VULNERABILITY SIGNATURE GENERATION We use an automata-ased string analysis to generate the vulneraility signature from an appliation [], [1]. This analysis takes as input a dependeny graph for the input program. A dependeny graph is a direted graph that speifies how the values of user inputs flow to the seurity sensitive funtions

4 (sinks). The analysis onsists of two phases. In the first phase, we perform a forward symoli reahaility analysis starting from nodes assoiated with input to ompute all possile values that eah node in the dependeny graph an take. We use this information to ollet vulnerale program points, as well as the reahale attak strings for those vulnerale program points. If the program is vulnerale, i.e., if there exists some vulnerale program points, we proeed to the seond phase. In the seond phase, we perform a akward symoli reahaility analysis from the vulnerale program points to ompute all possile values of their predeessors that will result in attak strings at these vulnerale program points. Figure shows the algorithm used in our analysis. The algorithm takes three inputs: a dependeny graph (denoted as G), a set of sink nodes (denoted as Sink), and an attak pattern (denoted as Attk). G is a direted dependeny graph that speifies how the values of user inputs flow to the seurity sensitive funtions. Sink denotes the nodes that are assoiated with seurity sensitive funtions that might lead to vulnerailities. Attk is a regular expression represented as an automaton that aepts the set of attak strings. At eah node, the set of reahale string values is approximated as a regular language and represented symolially as an automaton that aepts the language. To assoiate eah node with its automaton, we reate two automata vetors POST and PRE. The size of oth is ounded y the numer of nodes in G. POST[n] is the automaton aepting all possile string values that an reah node n. PRE[n] is the automaton aepting all possile string values that node n an take to exploit the vulneraility. Initially, all these automata aept nothing, i.e., their language is empty. Vul Sink is the set of vulnerale program points, and initially it is set to an empty set. At line 4, we first ompute POST y alling the forward analysis. At line 5, for eah node n Sink, we generate an automaton tmp y interseting the attak pattern and the possile values of n. If the language of tmp, i.e., L(tmp), is not empty, we identify that n is a vulnerale program point and add it to Vul at line 8. In fat, tmp aepts the set of reahale attak strings at node n that an e used to exploit the vulneraility. Hene, we assign tmp to PRE[n] at line 9. If Vul is not empty, we ompute PRE y alling our akward analysis at line 1. Note that for n Vul, PRE[n] has een assigned. We report vulneraility signatures for eah input node ased on PRE at line If Vul is an empty set, we report that the program is seure with respet to the attak pattern. The forward symoli reahaility analysis is ased on a standard work queue algorithm. We iteratively update the automata vetor POST until a fixpoint is reahed []. Bakward analysis uses the results of the forward analysis. Partiularly, it omputes all possile values of eah node n that an exploit the identified vulneraility. The hallenge in oth forward and akward analyses is omputing pre and post-onditions of string manipulation funtions suh as onatenation, stringreplae et., where the inputs and outputs of the pre and post-ondition operations are automata. We use the tehniques desried in [] for pre and post-ondition operations and the details of the symoli automata-ased forward and akward analyses an e found in [1]. The output of the vulneraility signature generation algo- 1: proedure VULSIGGENERATION(G, Sink, Attk) : INIT(POST, PRE) : Vul {} 4: FWDANALYSIS(G, POST) 5: for all n Sink do 6: tmp POST[n] Attk 7: if L(tmp) then 8: Vul Vul {n} 9: PRE[n] tmp 10: end if 11: end for 1: if Vul then 1: BWDANALYSIS(G, POST, PRE, Vul) 14: for all n Input do 15: REPORTVULNERABILITYSIGNATURE(PRE[n]) 16: end for 17: return Vulnerale 18: else 19: return Seure 0: end if 1: end proedure Figure. Figure 4. Vulneraility Signature Generation u 0 1 a... X Y Z... Large Numer of Paths rithm is a set of vulneraility signature automata. A vulneraility signature automaton is a tuple V = (Q, Σ, δ, q 0, F ), where Q is the set of states, Σ is the input alphaet, δ Q Σ Q is the transition relation, q 0 Q is the initial state, and F Q is the set of final states. The alphaet Σ is the set of ASCII haraters. Eah transition t δ is a tuple t = (q,, q ) where q = soure(t), q = target(t) and Σ. The vulneraility signature automata are deterministi, i.e., there is a single transition for eah soure state and alphaet symol. IV. a... X Y Z... CONVERTING VULNERABILITY SIGNATURE AUTOMATA TO DAGS Some features of the vulneraility signature automata make test generation diffiult. One feature is that there are large numer of transitions in δ where soure(t 0 ) = soure(t 1 ) = soure(t ) =... = soure(t n ) and target(t 0 ) = target(t 1 ) = target(t ) =... = target(t n ). Suh transitions ause an exponential low up in the numer of aepting paths in the automaton, and this leads to a large searh spae for test generation. As an example onsider state q in Figure 4. For this relatively small automaton there are aepting paths. Our solution to this prolem is to ollapse the transitions that have the same soure and target states into one transition as shown in Figure 5. The lael of the ollapsed transition is a range of haraters orresponding to eah transition that it represents. During test generation we only pik one harater from the range representing the all orresponding transitions. 4

5 u 0 1 {a,,,..,x,y,z...} {a,,,..,x,y,z...} 4 SCC 1 Figure 5. Collapsed Transitions SCC 0 a 1 g d e f h SCC o 6 5 i 4 m k n f h o 6 0 SCC 4 m 5 i k n e d g 1 Figure 7. High Level DAG Representation Figure 6. 0 Cyles in Automata a This allows us to avoid exponential low up in the numer of aepting paths. For the rest of the paper we assume that all transitions with the same soure and target states are ollapsed. Another feature of vulneraility signature automata is that they an ontain yles whih results in an infinite numer of aepting paths, i.e., an infinite searh spae for test generation. As an example, in Figure 6, states {q 1, q, q } and {q 4, q 5 } form yles. In order to ound the numer of aepting paths and, therefore the searh spae for test generation, we extrat a high level representation of the given vulneraility signature automaton y identifying its strongly onneted omponents (SCC). The high level representation we otain is a direted ayli graph DAG = (N, E) where N is the set of SCCs and E is the set of edges etween SCCs. At the automaton level eah edge e E is a transition suh that soure(e) s x, target(e) s y and s x s y. We use Tarjan s strongly onneted omponents algorithm to identify the yles in the vulneraility signature automata []. The worst ase time omplexity of this algorithm is O( Q + δ ) for a given vulneraility signature automaton V = (Q, Σ, δ, q 0, F ). High-level DAG representation for the automaton in Figure 6 is shown in Figure 7. It onsists of four strongly onneted omponents N = {SCC 0, SCC 1, SCC, SCC }, and six edges among them E = {e a, e, e k, e n, e f, e h }. V. STATE AND TRANSITION COVERAGE FOR VULNERABILITY SIGNATURE AUTOMATA USING MIN-COVER PATHS ALGORITHM In this setion we disuss generating test ases from vulneraility signature automata ased on state and transition overage riteria. Given a vulneraility signature automaton V = (Q, Σ, δ, q 0, F ), let L(V ) denote the set of strings aepted y V. Our aim is to find two sets of strings S s, S t L(V ) that ahieve state and transition overage, respetively. The state and transition overage definitions are as follows: For eah state in q Q there must e at least one string in S s suh that the aepting path for that strings visits q. For eah (ollapsed) transition t δ there must e at least one string in S t suh that the aepting path for that string inludes t. Finally, we want to generate the sets S s and S t in suh a way that S s and S t are minimized. The prolem of finding minimum numer of strings ased on state and transition overage riteria is very similar to a well-known graph prolem alled minimum over paths. Given a direted ayli graph, minimum over paths is the least numer of paths that visits eah edge of the graph at least one. Minimum over paths prolem has een studied in different researh areas and there are well known solutions to this prolem [4], [5]. One known solution is to redue minimum over paths prolem to the minimum flow prolem [4], [6], [5]. We follow this asi approah with some modifiations. We an divide the state and transition overage algorithms into five main steps: 1) Initialization of DAG, ) Converting DAG into a flow network, ) Minimum flow algorithm, 4) Finding minimum overing paths, 5) Extending paths with SCC Coverage. A. Initialization of DAG Vulneraility signature automaton V = (Q, Σ, δ, q 0, F ) has one start state q 0 and a set of final states F. In order to apply flow algorithms and minimum overing paths algorithm, one virtual final state q v is added to Q, for eah q F, a virtual transition t v = (q, λ, q ) is added to the transition relation δ where λ is a new symol added to the alphaet Σ. The modified automaton has one start state q 0 and one final state q v. A DAG representation DAG = (N, E) is onstruted from the modified automaton as desried in the previous setion. We use n 0 N to denote the start node of the DAG where n 0 = SCC 0 and q 0 SCC 0. Similarly, we use n v N to denote the as final node of the DAG suh that n v = SCC v and q v SCC v. A vulneraility signature automaton always has a sink state that terminates non-aepting paths orresponding to nonaepting strings. As a result, orresponding DAG representation has a sink node that does not have any outgoing edges. We generate only the strings that are aepted y vulneraility signature automaton. To do so we remove the sink node and

6 1: proedure PREPROCESSRIGTHSC(node, queue) : updated False : for all edge outgoingedges(node) do 4: nextnode targetnode(edge) 5: if flow(edge) = 0 then 6: if #inomingedges(nextnode) = 1 or #outgoingedges(nextnode) = 1 then 7: flow(edge) 1 8: updated True 9: else 10: REMOVEFROMDAG(edge) 11: end if 1: end if 1: end for 14: if not updated or alaned(node) = 0 then 15: return 16: end if 17: if updated and alaned(node) < 0 then 18: queue.enqueue(node) 19: else if updated and alaned(node) > 0 then 0: DISTRIBUTEFLOWSEVENLY(node) 1: end if : for all edge outgoingedges(node) do : nextnode targetnode(edge) 4: PREPROCESSRIGTH(nextNode, queue) 5: end for 6: end proedure Figure 8. Phase 1 for Pre-Proessing of State Coverage all inoming edges to the sink node from the DAG using a depth first traversal with a worst ase omplexity of O( E ). edges at different runs. However, this does not affet the state overage. We an define the flow funtion flow(e) as numer of visits for an edge e E. The alaned() funtion ompares the total input flow and total output flow for a node n N ased on flows for eah inoming and outgoing edges. A positive alane means that the total input flow is larger than the total output flow. In that ase line 0 distriutes the input flows to the the output flows y updating the flow values of outgoing edges. For the ase of a negative alane value, distriution is done in the reverse diretion after Phase 1 finishes as desried in [4]. Figure 9 also shows the initial flow values that are assigned to the example DAG. For the example shown in Figure 9, reverse pre-proessing (Phase ) is not neessary sine in the first phase flows are already distriuted orretly. SCC 0 0 a(1) (1) 1 SCC SCC 1 g d 4 m 5 i e n(1) h(1) SCC o 6 () SCC V v B. Converting DAG into a Flow Network A flow network is a DAG where eah edge has a apaity and eah edge reeives a flow. Capaity for eah edge e E is a non-negative real value (e) 0. Flow is a funtion f : E R that satisfies the following properties: For all e E, f(e) (e). For all e E, e E where, soure(e) = target(e ) and target(e) = soure(e ), f(e) = f(e ). For all n N, f(e) + e inoming(n) e outgoing(n) f(e ) = 0. Min-over paths algorithm does not require an upper ound for the apaity of an edge, and we assume that eah edge has infinite apaity. We define a flow as the numer of required visits to an edge in order to take eah path from the start node to the final node. To apply the min-flow algorithm, we need an initial flow assignment for eah edge in the DAG. We use a pre-proessing algorithm [4] to assign an initial flow to eah edge ased on the numer of input and output edges for eah node. This is a two phase algorithm that onsists of a depth first traversal starting from start node (Phase 1) followed y a reverse depth first traversal (Phase ) if neessary. The first phase of the initialization for state overage is shown in Figure 8. The statement at line 6 heks for the edges that an e removed safely. For example edges laeled with f and k an e safely removed from Figure 7. The resulting high level DAG is shown in Figure 9. Depending on the order that for loop retrieves the edges at line, algorithm may remove different Figure 9. Initialized DAG for State Coverage Phase 1 of the pre-proessing algorithm for transition overage is shown in Figure 10. The only modifiation ompared to the algorithm shown in Figure 8 is inside the if lok at line 5. The resulting flows for transition overage are shown in Figure 11. Starting from the initial node, the algorithm first assigns a flow value of 1 to the edges a and. When it omes to SCC during depth first traversal, it first assigns a flow of 1 to the edges k and n. As a result alane value of SCC eomes 1 and that SCC is queued for reverse pre-proessing. Similarly when algorithm first visits the SCC 1 using edges a or k, alane value for SCC 1 eomes negative and SCC 1 is also queued for reverse pre-proessing. However, when the algorithm visits SCC 1 for the seond time, alane value eomes 0 and reverse pre-proessing on SCC 1 does not have any effet. C. Minimum Flow Algorithm After we have initial flows alulated, Ford-Fulkerson algorithm is applied to the flow network with some modifiations [7], [4]. Modified Ford-Fulkerson algorithm omputes the minimum flows to visit eah transition at least one. The algorithm finds paths from the start node to the final node and removes the maximum amount of flow from eah path without reahing 0. Assume that our initialization phase alulated the flow for the path kh in Figure 11 as (4)k()h() instead of ()k(1)h(1). We an take away flows from all the edges in the path kh. Time omplexity of the algorithm for a DAG is O( p max (f 0 f min )) where p max is the maximum length path from start node to final node, f 0 is initial flow set and f min is the minimum flow [4].

7 1: proedure PREPROCESSRIGTHTC(node, queue) : updated False : for all edge outgoingedges(node) do 4: nextnode targetnode(edge) 5: if flow(edge) = 0 then 6: flow(edge) 1 7: updated True 8: end if 9: end for 10: if not updated or alaned(node) = 0 then 11: return 1: end if 1: if updated and alaned(node) < 0 then 14: queue.enqueue(node) 15: else if updated and alaned(node) > 0 then 16: DISTRIBUTEFLOWSEVENLY(node) 17: end if 18: for all edge outgoingedges(node) do 19: nextnode targetnode(edge) 0: PREPROCESSRIGTH(nextNode, queue) 1: end for : end proedure Figure 10. Phase 1 for Pre-Proessing of Transition Coverage SCC 1 e f(1) SCC o SCC V 1: list minp aths NULL : loop : path FINDMINPATH(node start) 4: if path = NULL then 5: reak 6: else 7: minp aths.add(path) 8: end if 9: end loop 10: proedure FINDMINPATH(node) 11: if node = node final then 1: path {} 1: return path 14: end if 15: for all edge outgoingedges(node) do 16: if flow(edge) = 0 then 17: ontinue 18: end if 19: DECREASEFLOWBYONE(edge) 0: nextnode targetnode(edge) 1: path = FINDMINPATH(nextNode) : if path = NULL then : ontinue 4: end if 5: path.add(edge) 6: return path 7: end for 8: return NULL 9: end proedure Figure 1. Minimum Covering Paths Algorithm Figure 11. SCC 0 0 a(1) () 1 SCC g d 4 m 5 i k(1) n(1) Initialized DAG for Transition Coverage D. Finding Minimum Covering Paths After running Minimum Flow Algorithm we an start looking for minimum overing paths. Minimum Covering Paths algorithm finds the edges that have flow(e) > 0 and forms a path that ends at the final node (i.e., the virtual node). Figure 1 shows the general loop and the reursive path finding funtion. For example, given the DAG shown in Figure 11, the minimum overing paths for transition overage are omputed as: afe v, khe v, and ne v where e v is the virtual edge. Let N k e the set of nodes that are k edges away from the start node. Let E k e the set of edges etween N k and N k+1. Let E max e the edge set with maximum size among the sets E 0, E 1, E,...E n. Finally, let P max e the maximum length path from start node to final node. Then, worst ase time omplexity for state and transition overage is O( P max E max ) and the maximum size test set size for oth overage riteria is O( E max ) whih is equal to the numer of minimum overing paths. For the DAGs that are extrated from the same vulneraility signature automaton let E max s denote the size of E max for the DAG generated for state overage and E max t denote the size of E max for the DAG generated for transition overage. Then, we have E max s E max t. For the sets of test ases generated for state and transition overage (S s and S t, respetively) we h(1) 6 () v have S s S t. E. Extending Paths with SCC Coverage One we have the results for minimum overing paths we do a pass on eah path and extend the SCC nodes n N that represent yles. We an define a strongly onneted omponent as SCC = (Q SCC, Σ, δ SCC ) where Q SCC Q and δ SCC δ. Assume there is a state q x Q SCC and a transition t δ. If q ( x) = target(t) and soure(t) / Q SCC, we say state q x is an entry point. Similarly, assume there is an edge q y Q SCC and a transition t δ. If q ( x) = soure(t) and target(t) / Q SCC, we say state q x is an exit point. There are two different strategies for SCC overage ased on DAG overage algorithm in progress. Strategy for the state overage algorithm is the following: Starting from an entry point visit all states q Q SCC at least one and end up in an exit point. Similarly, for transition overage starting from an entry point visit all transitions t δ SCC at least one and end up at an exit point. If δ SCC is greater than zero, then SCC must ontain a yle like SCC 1, SCC, and SCC in Figure 7. To terminate the algorithm we keep a queue for unvisited states or unvisited transitions and use depth first searh whenever neessary. Figure 1 shows the algorithm we use for state overage. DF S funtion at line 7 starts a depth first searh from the state given as its first argument and searhes for the state given as its seond argument without eing trapped in a yle. One it finds the state given as its seond argument, it returns a path that inludes all the states it visited. Algorithm for visiting all transitions t δ SCC is the same exept we keep a queue for unvisited transitions instead of unvisited states. Both algorithms have a worst ase omplexity of O( δ SCC ) whih depends on the overlapping yles within a SCC. Worst ase omplexity of length of the returned path is also the same as the time omplexity.

8 1: proedure VISITSTATES(SCC, q entry, q exit ) : list path NULL : queue notv isited getallstates(scc) 4: q q entry 5: notv isited.remove(q) 6: while size(notv isited) 0 do 7: visited DFS(q, notv isited.dequeue()) 8: notv isited.removeall(visited) 9: path.addall(visited) 10: q visited.last() 11: end while 1: if q q exit then 1: path.addall(dfs(q, q exit )) 14: end if 15: return path 16: end proedure Figure 1. SCC Coverage Consider the example vulneraility signature automaton shown in Figure 9. Based on state overage algorithm it an produe a path.a.h. where eah dot orresponds to a node in the DAG. Starting from the first dot whih is atually SCC 0 we extend the path. SCC 0 returns an empty path and algorithm ontinues with next SCC in the path a.h.. SCC 1 returns e for entry point q 1 and exit point q and algorithm extends the path as aeh.. At the end the algorithm returns the extended path aeh. VI. PATH COVERAGE FOR FOR VULNERABILITY SIGNATURE AUTOMATA USING DEPTH FIRST TRAVERSAL A straight forward definition of path overage would result in an infinite set of test ases due to loops in automata. So, given a vulneraility signature automaton V, we define S p L(V ) as follows: For eah path p in the DAG generated from V there must e a set of strings in S p suh that the aepting paths for those strings must orrespond to p (i.e. they must visit the same set of SCCs in the same order), and there must e an aepting path for eah omination of entry and exit nodes for all the SCCs in the path p. Path Coverage algorithm traverses DAG representation of vulneraility signature automata using a depth-first traversal (DFT). It does not have any initialization phase. It handles SCC entry-exit point overage during path exploration. Assume urrent node in the DFT is n and n orresponds to a SCC. Again assume q x is the entry point for the SCC orresponding to node n. Path overage algorithm alulates paths for all possile ominations of q x with all exit points using the SCC overage algorithm we have for transition overage. Then, it ontinues to explore paths in the high level DAG representation y following exit points in a DFT manner. By doing so, path overage algorithm alulates all possile ominations of all entry and exit points of a SCC. The path overage algorithm generates 5 paths for the example shown in Figure 11. Based on definitions we have in previous setion the time omplexity for path overage is O( E kmax Pmax ). Test size omplexity is the same as the time omplexity whih is asially all paths from start node to final nodes. As a result we have the following test set size omparison for the three overage riteria for the same vulneraility signature S s S t S p. VII. IMPLEMENTATION AND EXPERIMENTS In order to evaluate our automated testing framework, we used a delierately inseure we appliation alled Damn Vulnerale We Appliation (DVWA) to generate vulneraility signatures. DVWA is listed in OWASP Broken We Appliations Projet whih lists delierately inseure we appliations. DVWA has several SQL injetion, stored XSS and refleted XSS attaks with different seurity levels provided y the appliation. Seurity levels are no sanitization, ustom sanitization, and inorret use of uilt-in sanitization funtions. We generated vulneraility signatures for eah attak type onsidering different seurity levels. We used the Stranger stati string analysis tool [8] to generate vulneraility signatures. We ran all the experiments on an Intel I5 mahine with.5ghz X 4 proessors and GB of memory running Uuntu Tale I shows the properties of 5 vulneraility signatures generated from DVWA. We used the following well known attak patterns for vulneraility signature generation. Attak pattern /.*<sript.*>.*/ is used for vulneraility signatures XSS 1, XSS, and XSS. Attak pattern /.* or 1 = 1.*/ is used for vulneraility signature SQLI 1 and attak pattern /.* or 1 = 1.*/ is used for vulneraility signature SQLI. The sizes of the vulneraility signature automata depend on the omplexity and numer of string operations that appliation has on user inputs. We an see that vulneraility signatures SQLI 1 and XSS 1 are larger than the other three vulneraility signature automata. That is eause the orresponding appliation ode has more sanitization on user input. The appliation ode that orresponds to vulneraility signature SQLI has no sanitization at all and the generated vulneraility signature is similar to the attak pattern. For eah vulneraility signature, we an see that there is a ig differene etween the atual numer of transitions that an automaton has and the orresponding numer of ollapsed transitions whih allows us to redue the sizes of the generated test sets. For a given vulneraility signature, the relation etween the sizes of the test sets for different overage riteria follows the ordering we expet where S s S t S p. For larger vulneraility signatures, path overage algorithm produes a large numer of strings as expeted. For a given vulneraility signature, average length of the strings generated for state overage is the smallest. Sine the numer of states are smaller than the numer of transitions this is not surprising. The SCC overage algorithm for state overage produes strings with smaller lengths for most of the ases. In order to evaluate the effetiveness of our automated test generation tehniques we experimented on five open-soure appliations 1) PHP-Fusion v (ontent management system), ) RuuikCMS v1.1.1 (wesite ontent management tool), ) UL Forum v1.1.7 (forum appliation), 4) Snipe Gallery v.1.5 (image management system), 5) PHP Server Monitor v.0.1 (server management sript). We implemented a we appliation driver to automatially exeute the appliations with the automatially generated test strings. We exeuted eah appliation y assigning the automatially generated test strings to the seleted vulnerale input fields. We enaled xdeug tool to reord the server-side funtion all traes for eah request that our we appliation driver sends. After eah request, the we appliation driver extrats the sink funtion

9 Tale I. VULNERABILITY SIGNATURE AUTOMATA Vulneraility Signature Automaton Size Avr. Len. for Coverage # of Strings # of States # of Transitions # of Collapsed Transitions # of SCCs Generated Strings SQLI SQLI XSS XSS XSS State 8 9 Transition Path Cov 1 47 State Cov 1 15 Transition Cov 1 10 Path Cov 1 10 State Cov 8 1 Transition Cov 44 1 Path Cov 9 99 State Cov Transition Cov 8 1,717 Path Cov 8 1,68 State Cov 1 10 Transition Cov 7 Path Cov 7 alls with values of parameters from the trae file. For the SQL injetion attaks, eah all to mysql_query funtion is treated as a sink funtion all. For the XSS attaks, eah all to mysql_query funtion that exeutes INSERT or UPDATE statements is treated as sink funtion all. If the we appliation driver finds a sink funtion all, it heks the value of the query parameter of the sink funtion to onfirm if it ontains any type of attak. Tale II shows the effetiveness of the test sets generated using different overage riteria on different appliations. The sum of the third olumn and the fourth olumn shows the total numer of test strings in a test set generated from all vulneraility signatures for a given overage riteria. For example, there are a total of 19 test strings in the test set generated from all vulneraility signatures using the state overage riteria. Third olumn shows the numer of test strings that deteted the vulneraility in the given appliation (stated in the first olumn), and the fourth olumn shows the numer of test strings that missed the vulneraility. We an learly say that path overage and transition overage have etter detetion rates than state overage. The vulneraility detetion rates for the appliations php fusion and ruuik are lower ompared to other three appliations for eah overage riteria. This is due to the fat that these appliations have more string manipulation operations than the other three. For the fields seleted from other three appliations we oserve the same detetion rates. This is due to the fat that these appliations all have the same type of vulneraility. Tale III shows the vulneraility detetion rates of test sets generated using different overage riteria for eah vulneraility signature. It shows the distriution of the test sets in tale II to different vulneraility signatures and different overage riteria. Path overage riteria has etter detetion rates for vulneraility signatures XSS 1 and SQLI 1 whih are the larger vulneraility signature. For relatively small vulneraility signatures, path overage and transition overage detetion rates are the same. Vulneraility signature SQLI has the worst detetion rate. As we desried previously in this setion, that vulneraility signature is generated from a ode that has no sanitization operations, whih is not good enough for deteting attaks for appliations that have some string operations. One interesting result is that state overage for all XSS vulneraility signatures has a detetion rate 0%. The appliation that we used to generate the vulneraility sig- Tale II. VULNERABILITY DETECTION PERFORMANCE PER APPLICATION Appliation Coverage # Deteted # Missed Detetion Type Rate ulforum ruuik php fusion snipe phpservermon State % Transition % Path % State % Transition % Path % State 17 11% Transition % Path 5 6 4% State % Transition % Path % State % Transition % Path % natures onatenates HTML tags to the user inputs. Resulting vulneraility signature may inlude attak strings that has no losing tag >. State overage generates only strings that do not have losing tags, ut path and transition overage riteria are ale to handle that situation y visiting more transitions. Tale III. VULNERABILITY DETECTION PERFORMANCE PER VULNERABILITY SIGNATURE Appliation Coverage # Deteted # Missed Detetion Type Rate SQLI 1 SQLI XSS 1 XSS XSS State % Transition % Path % State 0 5 0% Transition 0 5 0% Path 0 5 0% State % Transition % Path % State 0 5 0% Transition % Path % State 0 5 0% Transition % Path % Overall, path overage has etter detetion rates as expeted. Transition overage detetion rates are very lose to path overage detetion rates, and transition overage generates smaller test sets. State overage is not effetive in generating attak strings for the vulneraility signatures we used.

10 VIII. RELATED WORK Stati string analysis has een an ative researh area, with the goal of finding and eliminating seurity vulnerailities aused y misuse of string manipulation operations [9], [10], [11], [1], [], [1]. String analysis fouses on statially identifying all possile values of a string expression at a program point, and this knowledge an e leveraged to eliminate vulnerailities suh as SQL injetion and XSS attaks. Due to undeidaility of string analysis prolem stati string analysis approahes use onservative approximations suh as widening [14], [15], [], that an result in false positives. Moreover stati modeling of all string manipulation funtions is hallenging and typially limits the appliaility of stati string analysis tehniques. We are not aware of any prior work that omines stati string analysis and vulneraility signatures with automated test generation. In [16], [17], [18] dynami symoli exeution has een used for automati testing of a we appliation. First, string onstraints are generated using symoli exeution. Then, these onstraints are solved to generate vulnerale input strings. In [17], [18] authors use a ounded string onstraint solver that ounds the length of the strings efore solving the onstraint. In [16] string onstraints are represented using finite state transduers. Unlike dynami symoli exeution, whih is a white ox testing approah, our approah is a lakox speifiation-ased testing approah. Dynami symoli exeution tries to inrease exeution path overage while in our ase we try to inrease overage of the vulneraility signature automaton that we use as a speifiation. In [19] a lak ox SQLI/XSS we vulneraility sanner is developed utilizing manually written attak strings with no speifi riteria. In XSS Analyzer [0], a lak ox testing approah is used where a very large dataase of attak strings is utilized to attak a we appliation. A learning algorithm is used to pik only a suset of this dataase. We use stati analysis to automatially generate vulneraility signatures from whih the attak strings are generated. Also, sine we generate attak strings from an automaton, the original size of the attak string dataase ould e infinite whereas in XSS analyzer the size of the attak string dataase is finite. In [1] state mahine ased test generation using UML state harts is disussed. They define overage riteria suh as single UML transition overage, full prediate overage, transition-pair overage, and omplete sequene overage. These overage riteria are speifi for UML diagrams. In [] authors generate test ases from finite state mahines that orrespond to a software system speifiation. State mahine ased test generation has een used for different areas suh as ontrol systems, protools, iruit design, data proessing, navigation analyses. Minimum over paths algorithm has een studied for program testing [5] in order to generate minimum numer of paths for ertain features and to generate test data for those paths. IX. CONCLUSION We presented an automated testing framework for testing input validation and sanitization operations in we appliations. In our framework the tests are generated from vulneraility signatures that are haraterized as automata. Our experiments show that vulneraility signatures generated from delierately inseure we appliations an e used to generate effetive tests for identifying vulnerailities in other appliations. REFERENCES [1] F. Yu, M. Alkhalaf, and T. Bultan, Generating vulneraility signatures for string manipulating programs using automata-ased forward and akward symoli analyses, in ASE, 009, pp [] F. Yu, T. Bultan, M. Cova, and O. H. Iarra, Symoli string verifiation: An automata-ased approah, in Pro. of SPIN, 008, pp [] R. E. Tarjan, Depth-first searh and linear graph algorithms, SIAM J. Comput., vol. 1, no., pp , 197. [4] M. Brandizi, N. Kuratova, U. Sarkans, and P. Roa-Serra, graphta, a lirary to onvert experimental workflow graphs into taular formats, Bioinformatis, vol. 8, no. 1, pp , 01. [5] S. C. Ntafos and S. L. Hakimi, On path over prolems in digraphs and appliations to program testing, IEEE Trans. Software Eng., vol. 5, no. 5, pp , [6] E. Ciurea and L. Ciupal, Sequential and parallel algorithms for minimum flows, Journal of Applied Mathematis and Computing, vol. 15, no. 1-, pp. 5 75, 004. [7] L. Ford Jr and D. Fulkerson, Maximal flow through a network, in Classi papers in ominatoris. Springer, 1987, pp [8] F. Yu, M. Alkhalaf, and T. Bultan, Stranger: An automata-ased string analysis tool for php, in TACAS, 010, pp [9] A. S. Christensen, A. Møller, and M. I. Shwartzah, Preise analysis of string expressions, in Pro. 10th International Stati Analysis Symposium, SAS 0, ser. LNCS, vol Springer-Verlag, June 00, pp [10] Y. Minamide, Stati approximation of dynamially generated we pages, in Proeedings of the 14th International World Wide We Conferene, 005, pp [11] G. Wassermann and Z. Su, Sound and preise analysis of we appliations for injetion vulnerailities, in Proeedings of the ACM SIGPLAN 007 Conferene on Programming Language Design and Implementation, 007, pp. 41. [1] G.Wassermann and Z. Su, Stati detetion of ross-site sripting vulnerailities, in ICSE, 008, pp [1] P. Hooimeijer, B. Livshits, D. Molnar, P. Saxena, and M. Veanes, Fast and Preise Sanitizer Analysis with Bek, in Usenix Seurity Symposium, 011. [14] T.-H. Choi, O. Lee, H. Kim, and K.-G. Doh, A pratial string analyzer y the widening approah, in APLAS, 006, pp [15] C. Bartzis and T. Bultan, Widening arithmeti automata, in Proeedings of the 16th International Conferene on Computer Aided Verifiation, 004, pp. 1. [16] G. Wassermann, D. Yu, A. Chander, D. Dhurjati, H. Inamura, and Z. Su, Dynami test input generation for we appliations, in Proeedings of the ACM/SIGSOFT International Symposium on Software Testing and Analysis (ISSTA 008), 008, pp [17] A. Kiezun, V. Ganesh, P. J. Guo, P. Hooimeijer, and M. D. Ernst, Hampi: a solver for string onstraints, in ISSTA, 009, pp [18] P. Saxena, D. Akhawe, S. Hanna, F. Mao, S. MCamant, and D. Song, A symoli exeution framework for javasript, in Pro. of the 1st IEEE Symposium on Seurity and Privay (Oakland 010), 010. [19] S. Kals, E. Kirda, C. Krügel, and N. Jovanovi, Seuat: a we vulneraility sanner, in WWW, 006, pp [0] O. Tripp, O. Weisman, and L. Guy, Finding your way in the testing jungle: a learning approah to we seurity testing, in ISSTA, 01, pp [1] A. J. Offutt and A. Adurazik, Generating tests from uml speifiations, in UML, 1999, pp [] G. Friedman, A. Hartman, K. Nagin, and T. Shiran, Projeted state mahine overage for software testing, in ISSTA, 00, pp