Bypassing Space Explosion in Regular Expression Matching for Network Intrusion Detection and Prevention Systems

Transcription

1 Bypassing Spae Explosion in Regular Expression Mathing for Network Intrusion Detetion and Prevention Systems Jignesh Patel Alex X. Liu Eri Torng Department of Computer Siene and Engineering Mihigan State University East Lansing, MI 48823, U.S.A. {patelji1, alexliu, Astrat 1 Introdution 1.1 Bakground and Prolem Statement Network intrusion detetion and prevention systems ommonly use regular expression (RE) signatures to represent individual seurity threats. While the orresponding DFA for any one RE is typially small, the DFA that orresponds to the entire set of REs is usually too large to e onstruted or deployed. To address this issue, a variety of alternative automata implementations that ompress the size of the final automaton have een proposed suh as XFA and D 2 FA. The resulting final automata are typially muh smaller than the orresponding DFA. However, the previously proposed automata onstrution algorithms do suffer from some drawaks. First, most employ a Union then Minimize framework where the automata for eah RE are first joined efore minimization ours. This leads to an expensive NFA to DFA suset onstrution on a relatively large NFA. Seond, most onstrut the orresponding large DFA as an intermediate step. In some ases, this DFA is so large that the final automaton annot e onstruted even though the final automaton is small enough to e deployed. In this paper, we propose a Minimize then Union framework for onstruting ompat alternative automata fousing on the D 2 FA. We show that we an onstrut an almost optimal final D 2 FA with small intermediate parsers. The key to our approah is a spae and time effiient routine for merging two ompat D 2 FA into a ompat D 2 FA. In our experiments, our algorithm runs up to 302 times faster and uses 1390 times less memory than previous algorithms. For example, we are ale to onstrut a D 2 FA with over 80,000,000 states using only 1GB of main memory in only 77 minutes. The ore omponent of today s network seurity devies suh as Network Intrusion Detetion and Prevention Systems is signature ased deep paket inspetion. The ontents of every paket need to e ompared against a set of signatures. Appliation level signature analysis an also e used for deteting peer-to-peer traffi, providing advaned QoS mehanisms. In the past, the signatures were speified as simple strings. Today, most deep paket inspetion engines suh as Snort [1, 24], Bro [23], TippingPoint X505 and Ciso seurity applianes use regular expressions (REs) to define the signatures. REs are used instead of simple string patterns eause REs are fundamentally more expressive and thus are ale to desrie a wider variety of attak signatures [28]. As a result, there has een a lot of reent work on implementing high speed RE parsers for network appliations. Most RE parsers use some variant of the Deterministi Finite State Automata (DFA) representation of REs. A DFA is defined as a 5-tuple (Q,Σ,δ,q 0,A), where Q is a set of states, Σ is an alphaet, δ: Q Σ Q is the transition funtion,q 0 Q is the start, anda Q is the set of aepting states. Any set of REs an e onverted into an equivalent DFA with the minimum numer of states [13, 14]. DFAs have the property of needing onstant memory aess per input symol, and hene result in preditale and fast andwidth. The main prolem with DFAs is spae explosion: a huge amount of memory is needed to store the transition funtion whih has Q Σ entries. Speifially, the numer of states an e very large (state explosion), and the numer of transitions per state is large ( Σ ). To address the DFA spae explosion prolem, a variety of DFA variants have een proposed that require muh less memory than DFAs to store. For example, there is the De-

2 layed Input DFA (D 2 FA) proposed y Kumar et al. [17]. The asi idea of D 2 FA is that in a typial DFA for real world RE set, given two states u and v, δ(u,) = δ(v,) for many symols Σ. We an remove all the transitions for v from δ for whih δ(u,) = δ(v,) and make a note that v s transitions were removed ased on u s transitions. When the D 2 FA is later proessing input and is in state v and enounters input symol x, if δ(v,x) is missing, the D 2 FA an use δ(u,x) to determine the next state. We an do the same thing for most states in the DFA, and it results in tremendous transition ompression. Kumar et al. oserve an average derease of 97.6% in the amount of memory required to store a D 2 FA when ompared to its orresponding DFA. In more detail, to uild a D 2 FA from a DFA, just do the following two steps. First, for eah state u Q, pik a deferred state, denoted y F(u). (We an have F(u) = u.) Seond, for eah state u Q for whih F(u) u, remove all the transitions for u for whih δ(u, x) = δ(f(u), x). When traversing the D 2 FA, if on urrent state u and urrent input symol x, δ(u,x) is missing (i.e. has een removed), we an use δ(f(u),x) to get the next state. Of ourse, δ(f(u), x) might e missing too, in whih ase we then useδ(f(f(u)),x) to get the next state, and so on. The only restrition on seleting deferred states is that the funtion F annot reate a yle other than a self-loop on the states; otherwise all states on that yle might have their transitions on some x Σ removed and there would no way of finding the next state. Figure 1(a) shows a DFA for the REs set {.*a.*,.*.*}, and Figure 1() shows the D 2 FA uilt from the DFA. The dashed lines represent deferred states. The DFA has = 3328 transitions, whereas the D 2 FA only has 1030 atual transitions and 9 deferred transitions. D 2 FA are very effetive at dealing with the DFA spae explosion prolem. In partiular, D 2 FA exhiit tremendous transition ompression reduing the size of the DFA y a huge fator; this makes D 2 FA muh more pratial for a software implementation of RE mathing than DFAs. D 2 FAs are also used as starting point for advaned tehniques like those in [18, 20]. This leads us to the fundamental prolem we address in this paper. Given as input a set of REs R, uild a ompat D 2 FA as effiiently as possile that also supports frequent updates. Effiieny is important as urrent methods for onstruting D 2 FA may e so expensive in oth time and spae that they may not e ale to onstrut the final D 2 FA even if the D 2 FA is small enough to e deployed in networking devies that have limited omputing resoures. Suh issues eome douly important when we onsider the issue of the frequent updates (typially additions) to R that our as new seurity threats are identified. The limited resoure networking devie must e ale to effiiently ompute the new D 2 FA. One sutle ut important point aout this prolem is that the resulting D 2 FA must report whih RE (or REs) from R mathed a given input; this applies eause eah RE typially orresponds to a unique seurity threat. Finally, while we fous on D 2 FA in this paper, we elieve that our tehniques an e generalized to other ompat RE mathing automata solutions [5, 8, 16, 26, 27]. 1.2 Summary of Prior Art Given the input RE set R, any solution that uilds a D 2 FA for R will have to do the following two operations: (a) union the automata orresponding to eah RE in R and () minimize the automata, oth in terms of the numer of states and the numer of edges. Previous solutions [6, 17] employ a Union then Minimize framework where they first uild automata for eah RE within R, then perform union operations on these automata to arrive at one omined automaton for all the REs in R, and only then minimize the resulting omined automaton. In partiular, previous solutions typially perform a omputationally expensive NFA to DFA suset onstrution followed y or omposed with DFA minimization (for states) and D 2 FA minimization (for edges). Consider the D 2 FA onstrution algorithm proposed y Kumar et al. [17]. They first apply the Union then Minimize framework to produe a DFA that orresponds to R and then onstrut the orresponding minimum state DFA. Next, in order to maximum transition ompression, they essentially solve a maximum weight spanning tree prolem on the following weighted graph whih they all a Spae Redution Graph (SRG). The SRG has DFA states as its verties. The SRG is a omplete graph with the weight of an edgew(u,v) equal to the numer of ommon transitions etween DFA states u and v. One the spanning tree is seleted, a root state is piked and all edges are direted towards the root. These direted edges give the deferred state for eah state. Figure 1() shows the SRG uilt for the DFA in Figure 1(a). Using the resulting maximum weight spanning tree on the SRG, they then set the deferment state for eah state eliminating redundant transitions. Behi and Crowley also use the Union then Minimize Framework to arrive at a minimum state DFA [6]. At this point, rather than using an SRG to set deferment states for eah state, Behi and Crowley use state levels where the level of a DFA stateuis the length of the shortest string that takes the DFA from the start state to state u. Behi and Crowley oserved that if all states defer to a state that is at a lower level than itself, then the deferment funtionf an never produe a yle. Furthermore, when proessing any input string of length n, at most n 1 deferred transitions will e proessed. Thus, for eah state u, among all the states at a lower level thanu, Behi and Crowley set F(u)

3 from 1,3 {,} 1 3 {a,} a from from 4,6,7,9,10,12 4,6,10 {,} /1,2 a a from 5,8,11 {a,,} from 2,5,8, /1 11/ {a,} 0 a {,} a 2 {a,} (a) () () /1 12/1,2 11/2 Figure 1. (a) DFA for RE set {.*a.*,.*.*}. () SRG for the DFA. Edges with weight 1 are not shown. Unlaeled edges have weight 255. () The D 2 FA. Dashed edges represent deferment. to e the state whih shares the most transitions withu. The resulting D 2 FA is typially a it larger in size than the D 2 FA uilt using the SRG whih does not have the deferring to lower level state restrition. 1.3 Limitations of Prior Art Prior methods have three fundamental limitations. First, they follow the Union then Minimize framework whih means they reate large automata and only minimize them at the end. This also means they must employ the expensive NFA to DFA suset onstrution. Seond, prior methods uild the orresponding minimum state DFA efore onstruting the final D 2 FA. This is very ostly in oth spae and time. The D 2 FA is typially 50 to 100 times smaller than the DFA, so even if the D 2 FA would fit in availale memory, the intermediate DFA might e too large, making it impratial to uild the D 2 FA. This is exaerated in the ase of the Kumar et al. algorithm whih needs the SRG whih ranges from aout the size of the DFA itself to over 50 times the size of the DFA. The resulting spae and time required to uild the DFA and SRG impose serious limits on the D 2 FA that an e pratially onstruted. We do oserve that the method proposed in [6] does not need to reate the SRG. Furthermore, as the authors have noted, there is a way to go from the NFA diretly to the D 2 FA, ut implementing suh an approah is still very ostly in time as many transition tales need to e repeatedly rereated in order to realize these spae savings. Third, none of the previous methods provide effiient algorithms for updating the D 2 FA when a new RE is added tor. 1.4 Our Approah To address these limitations, we propose a Minimize then Union framework. Speifially, we first minimize the small automata orresponding to eah RE fromrand then union the minimized automata together. A key property of our method is that our union algorithm automatially produes a minimum state D 2 FA for the regular expressions involved without expliit state minimization. Likewise, we reate deferment states effiiently while performing the union operation using deferment information from the input D 2 FAs. Together, these optimizations lead to a vastly more effiient D 2 FA onstrution algorithm in oth time and spae. In more detail, givenr, we first uild a DFA and D 2 FA for eah individual RE in R. The heart of our tehnique is the D 2 FA merge algorithm that performs the union. It merges two smaller D 2 FAs into one larger D 2 FA suh that the merged D 2 FA is equivalent to the union of REs that the D 2 FAs eing merged were equivalent to. Starting from the the initial D 2 FAs for eah RE, using this D 2 FA merge suroutine, we merge two D 2 FAs at a time until we are left with just one final D 2 FA. The initial D 2 FAs are eah equivalent to their respetive REs, so the final D 2 FA will e equivalent to the union of all the REs in R. Figures 2(a) and 2() show the initial D 2 FAs for the RE set {.*a.*,.*.*}. The resulting D 2 FA from merging these two D 2 FAs using the D 2 FA merge algorithm is shown in Figure 2(). Advantages of our algorithm One of the main advantages of our algorithm is a dramati inrease in time and spae effiieny. These effiieny gains are partly due to our use of the Minimize then Union framework instead of the Union then Minimize framework. More speifially, our improved effiieny omes aout from the following four fators. First, other than for the initial DFAs that orrespond to individual REs in R, we uild D 2 FA ypassing DFAs. Those initial DFAs are very small (typially < 50 states), so the memory and time required to uild the ini-

4 a a /1 (a) /2 () {a,} {a,} 0,0 0 0,1 2 0,2 5 0,3 8 0,4 11/2 a a {,} 1,0 1 1,1 4 () 2,0 3 2,2 7 3,1 6 3,3 10 4,2 9/1 4,4 12/1, Deferment for 5= 0, Deferment for 7= 2, Deferment for 9= 4, Deferment for 12= 4,4 (d) Figure 2. (a) D 1, the D 2 FA for RE.*a.*. () D 2, the D 2 FA for RE.*.*. () D 3, the merged D 2 FA. (d) Illustration of setting deferment for some states in D 3. tial DFAs and D 2 FAs is negligile. The D 2 FA merge algorithm diretly merges the two input D 2 FAs to get the output D 2 FA without reating the DFA first. Seond, other than for the initial DFAs, we never have to perform the NFA to DFA suset onstrution. Third, other than for the initial DFAs, we never have to perform DFA state minimization. Fourth, when setting deferment states in the D 2 FA merge algorithm, we use deferment information from the two input D 2 FA. This typially involves performing only a onstant numer of omparisons per state rather than a linear in the numer of states omparison per state as is required y previous tehniques. All told, our algorithm has a pratial time omplexity of O(n Σ ) where n is the numer of states in the final D 2 FA and Σ is the size of the input alphaet. In ontrast, Kumar et al. s algorithm [17] has a time omplexity of O(n 2 (log(n)+ Σ )) and Behi and Crowley s algorithm [6] has a time omplexity of O(n 2 Σ ) just for setting the deferment state for eah state and ignoring the ost of the NFA suset onstrution and DFA state minimization. See Setion 5.4 for a more detailed omplexity analysis. These effiieny advantages allow us to uild muh larger D 2 FAs than are possile with previous methods. For the syntheti RE set that we onsider in Setion 6, given a maximum working memory size of 1GB, we an uild a D 2 FA with 80,216,064 states with our D 2 FA merge algorithm whereas the Kumar et al. algorithm an only uild a D 2 FA with 397,312 states. Also from Setion 6, our algorithm is typially 25 to 300 times faster than previous algorithms on our RE sets. Besides eing muh more effiient in onstruting D 2 FA from srath, our algorithm is very well suited for frequent RE updates. When an RE needs to e added to the urrent set, we just need to merge the D 2 FA for the RE to the urrent D 2 FA using our merge routine whih is a very fast operation. Tehnial Challenges For our approah to work, the main hallenge is to figure out how to effiiently union two minimum state D 2 FAs D 1 and D 2 so that the resulting D 2 FAD 3 is also a minimum state D 2 FA. There are two aspets to this hallenge. First, we need to make sure that D 2 FAD 3 has the minimum numer of states. More speifially, supposed 1 andd 2 are equivalent to RE sets R 1 and R 2, respetively. We need to ensure that D 3 has the minimum numer of states of any D 2 FA equivalent tor 1 R 2. We use an existing union ross produt onstrution for this and prove that it results in a minimum state D 2 FA for our purpose. We emphasize that this is not true in general ut holds for appliations where D 3 must identify whih REs from R 1 R 2 math a given input string. Many seurity appliations meet this riteria. Our seond hallenge is uilding the D 2 FA D 3 without uilding the entire DFA equivalent to D 3 while ensuring that D 3 ahieves signifiant transition ompression; that is, the numer of atual edges stored in D 2 FA D 3 must e small. More onretely, as eah state in D 3 is reated, we need to immediately set a deferred state for it; otherwise, we would e storing the entire DFA. Furthermore, we need to hoose a good deferment state that eliminates as many

5 edges as possile. We address this hallenge y effiiently hoosing a good deferment state for D 3 y using the deferment information from D 1 and D 2. Typially, the algorithm only needs to ompare a handful of andidate deferment states. Key Contriutions In summary, we make the following ontriutions: (1) We propose a novel Minimize then Union framework for onstruing D 2 FA for network seurity RE sets that we elieve will generalize to other RE mathing automata. Note, Smith et al. used the Minimize then Union Framework when onstruting XFA [26, 27], though they did not prove any properties aout their union algorithms. (2) To implement this framework, we propose a very effiient D 2 FA merge algorithm for performing the union of two D 2 FAs. (3) To prove the orretness of this D 2 FA merge algorithm, we prove a fundamental property aout the standard union ross produt onstrution and minimum state DFAs when applied to network seurity RE sets that an e applied to other RE mathing automata. We implemented our algorithms and onduted experiments on realworld and syntheti RE sets. Our experiments indiate that: (a) Our algorithm generates D 2 FA with a fration of the memory required y existing algorithms making it feasile to uild D 2 FA with many more states. For the real world RE sets we onsider in the experimental setion, our algorithm requires an average of 1390 times less memory than the algorithm proposed in [17] and 31 times less memory than the algorithm proposed in [6]. () Our algorithm runs muh faster then existing algorithms. For the real world RE sets we onsider in the experimental setion, our algorithm runs an average of 302 times faster than the algorithm proposed in [17] and 28 times faster than the algorithm proposed in [6]. () Even with the huge spae and time effiieny gains, our algorithm generates D 2 FA only slightly larger than existing algorithms in the worst ase. The rest of the paper is organized as follows. We review related work in Setion 2. In Setion 3 we present our onstrution for effiiently uilding DFA for pattern mathing RE sets. In Setion 4 we present our D 2 FA merge algorithm and the inrementally uilding approah. Setion 5 presents some properties of our D 2 FA merge algorithm, and some variations of it. We present experimental results and onlusions in Setions 6 and 7 respetively. For spae reasons, the proofs of Theorems and Lemmas and algorithm pseudoode are given in the Appendix. 2 Related Work Initially network intrusion detetion and prevention systems used string patterns to speify attak signatures [2, 3, 29, 31 33, 36]. Sommer and Paxson [28] first proposed using REs instead of strings to speify attak signatures. Today network intrusion detetion and prevention systems mostly use REs for attak signatures. RE mathing solutions are typially software-ased or hardware-ased (FPGA or ASIC). Software-ased approahes are heap and deployale on general purpose proessors, ut their throughput may not e high. To ahieve higher throughput, software solutions an e deployed on ustomized ASIC hips at the ost of low versatility and high deployment ost. To ahieve deterministi throughput, software-ased solutions must use DFAs, whih fae a spae explosion prolem. Speifially, there an e state explosion where the numer of states inreases exponentially in the numer of REs, and the numer of transitions per state is extremely high. To address the spae explosion prolem, transition ompression and state minimization software-ased solutions have een developed. Transition ompression shemes that minimize the numer of transitions per state have mostly used one of two tehniques. One is alphaet re-enoding, whih exploits redundany within a state, [6, 7, 10, 15]. The seond is default transitions or deferment states whih exploits redundany among states [4, 6, 17, 18]. Kumar et al. [17] originally proposed the use of default transitions. Behi and Crowley [6] proposed a more effiient way of using default transitions. Our work falls into the ategory of transition ompression via default transitions. Our algorithms are muh more effiient than those of [6, 17] and thus an e applied to muh larger RE sets. For example, if we are limited to 1GB of memory to work with, we show that Kumar et al. s original algorithm an only uild a D 2 FA with less than 400,000 states whereas our algorithm an uild a D 2 FA with over 80,000,000 states. Two asi approahes have een proposed for state minimization. One is to partition the given RE set and uild a DFA for eah partition [35]. When inspeting paket payload, eah input symol needs to e sanned against eah partition s DFA. Our work is orthogonal to this tehnique and an e used in omination with this tehnique. The seond approah is to modify the automata struture and/or use extra memory to rememer history and thus avoid state dupliation [5,8,16,26,27]. We elieve our merge tehnique an e adopted to work with some of these approahes. For example, Smith et al. also use the Minimize then Union framework when onstruting XFA [26, 27]. One potential drawak with XFA is that there is no fully automated proedure to onstrut XFAs from a set of regular expressions. Paraphrasing Yang, Karim, Ganapathy, and Smith [34], onstruting an XFA from a set of REs requires manual analysis of the REs to identify and eliminate amiguity. FPGA-ased solutions typially exploit the parallel proessing apailities of FPGAs to implement a Nondeterministi Finite Automata (NFA) [5,7,11,12,21,25,30] or to implement multiple parallel DFAs [22]. TCAM ased solu-

6 tions have een proposed for string mathing in [3,9,31,36] and for REs in [20]. Our work an potentially e applied to these solutions as well. Reently and independently, Liu et al. proposed to onstrut DFA y hierarhial merging [19]. That is, they essentially propose the Minimize then Union framework for DFA onstrution. They onsider merging multiple DFAs at a time rather than just two. However, they do not onsider D 2 FA, and they do not prove any properties aout their merge algorithm inluding that it results in minimum state DFAs. 3 Pattern Mathing DFAs 3.1 Pattern Mathing DFA Definition In a standard DFA, defined as a 5-tuple (Q,Σ,δ,q 0,A), eah aepting state is equivalent to any other aepting state. However, in many pattern mathing appliations where we are given a set of REs R, we must keep trak of whih REs have een mathed. For example, eah RE may orrespond to a unique seurity threat that requires its own proessing routine. This leads us to define Pattern Mathing Deterministi Finite State Automata (PMDFA). The key differene etween a PMDFA and a DFA is that for eah stateq in a PMDFA, we annot simply mark it as aepting or rejeting; instead, we must reord whih REs fromrare mathed when we reah q. More formally, given as input a set of REsR, a PMDFA is a5-tuple(q,σ,δ,q 0,M) where the last termm is now defined asm: Q 2 R. 3.2 Minimum State PMDFA onstrution Given a set of REs R, we an uild the orresponding minimum state PMDFA using the standard Union then Minimize framework: first uild an NFA for the RE that orresponds to an OR of all the REsr R, then onvert the NFA to a DFA, and finally minimize the DFA treating aepting states as equivalent if and only if they orrespond to the same set of regular expressions. This method an e very slow, mainly due to the NFA to DFA onversion, whih often results in an exponential growth in the numer of states. Instead, we propose a more effiient Minimize then Union framwork. Let R 1 and R 2 denote any two disjoint susets of R, and let D 1 and D 2 e their orresponding minimum state PMDFAs. We use the standard union ross produt onstrution to onstrut a minimum state PMDFA D 3 that orresponds to R 3 = R 1 R 2. Speifially, suppose we are given the two PMDFAs D 1 = (Q 1,Σ,δ 1,q 01,M 1 ) and D 2 = (Q 2,Σ,δ 2,q 02,M 2 ). The union ross produt PMDFA of D 1 and D 2, denoted as UCP(D 1,D 2 ), is given y D 3 = UCP(D 1,D 2 ) = (Q 3,Σ,δ 3,q 03,M 3 ) where Q 3 = Q 1 Q 2, δ 3 ( q i,q j,x) = δ 1 (q i,x),δ 2 (q j,x), q 03 = q 01,q 02, andm 3 ( q i,q j ) = M 1 (q i ) M 2 (q j ). Eah state ind 3 orresponds to a pair of states, one from D 1 and one fromd 2. For notational larity, we use and to enlose an ordered pair of states. Transition funtion δ 3 just simulates othδ 1 andδ 2 in parallel. Many states in Q 3 might not e reahale from the start state q 03. Thus, while onstruting D 3, we only reate states that are reahale fromq 03. We now argue that this onstrution is orret. This is a standard onstrution, so the fat that D 3 is a PMDFA for R 3 = R 1 R 2 is straightforward and overed in standard automata theory textooks (e.g. [14]). We now show that D 3 is also a minimum state PMDFA forr 3 assumingr 1 R 2 =, a result that does not follow for standard DFAs. Theorem 3.1. Given two RE sets, R 1 andr 2, and equivalent minimum state PMDFAs, D 1 and D 2, the union ross produt DFA D 3 = UCP(D 1,D 2 ), with only reahale states onstruted, is the minimum state PMDFA equivalent tor 3 = R 1 R 2 if R 1 R 2 =. Proof. First sine only reahale states are onstruted,d 3 annot e trivially redued. Now assume D 3 is not minimum. That would mean there are two states in D 3, say p 1,p 2 and q 1,q 2, that are indistinguishale. This implies that x Σ, M 3 (δ 3 ( p 1,p 2,x)) = M 3 (δ 3 ( q 1,q 2,x)). Working on oth sides of this equality, we get x Σ, M 3 (δ 3 ( p 1,p 2,x)) = M 3 ( δ 1 (p 1,x),δ 2 (p 2,x) ) as well as x Σ, = M 1 (δ 1 (p 1,x)) M 2 (δ 2 (p 2,x)) M 3 (δ 3 ( q 1,q 2,x)) = M 3 ( δ 1 (q 1,x),δ 2 (q 2,x) ) This implies that = M 1 (δ 1 (q 1,x)) M 2 (δ 2 (q 2,x)) x Σ M 1 (δ 1 (p 1,x)) M 2 (δ 2 (p 2,x)) = M 1 (δ 1 (q 1,x)) M 2 (δ 2 (q 2,x)). Now siner 1 R 2 =, this gives us x Σ, M 1 (δ 1 (p 1,x)) = M 1 (δ 1 (q 1,x)) and x Σ, M 2 (δ 1 (p 2,x)) = M 2 (δ 1 (q 2,x)) This implies that p 1 and q 1 are indistinguishale in D 1 and p 2 and q 2 are indistinguishale in D 2, implying that othd 1 andd 2 are not minimum state PMDFAs, whih is a ontradition and the result follows.

7 Our effiient onstrution algorithm works as follows. First, for eah RE r R, we uild an equivalent minimum state PMDFA D for r using the standard method, resulting in a set of PMDFAsD. Then we merge two PMDFAs from D at a time using the aove UCP onstrution until there is just one PMDFA left in D. The merging in done in a greedy manner: in eah step, the two PMDFAs with the fewest states are merged together. Note the onditionr 1 R 2 is always satisfied in all the merges. In our experiments, our Minimize then Union tehnique runs exponentially faster than the standard Union then Minimize tehnique eause we only apply the NFA to DFA step to the NFAs that orrespond to eah individual regular expression rather than the omposite regular expression. This makes a signifiant differene even when we have a relatively small numer of regular expressions. For example, for our C7 RE set whih ontains 7 REs, the standard tehnique requires seonds to uild the PMDFA, ut our tehnique uilds the PMDFA in only 0.66 seonds. For the remainder of this paper, we use DFA to stand for minimum state PMDFA. 4 Effiient D 2 FA Constrution In this setion, we first formally define what a D 2 FA is and then desrie how we an extend the Minimize then Union tehnique to D 2 FA ypassing DFA onstrution. 4.1 D 2 FA Definition Let D = (Q,Σ,δ,q 0,M) e a DFA. A orresponding D 2 FA D is defined as a 6-tuple (Q,Σ,ρ,q 0,M,F). Together, funtion F: Q Q and partial funtion ρ: Q Σ Q are equivalent to DFA transition funtionδ. Speifially, F defines a unique deferred state for eah state in Q, and ρ is a partially defined transition funtion. We use dom(ρ) to denote the domain of ρ, i.e. the values for whih ρ is defined. The key property of a D 2 FA D that orresponds to DFA D is that q, Q Σ, q, dom(ρ) (F(q) = q δ(q,) δ(f(q),)); that is for eah state, ρ only has those transitions that are different from that of its deferred state in the underlying DFA. When defined, ρ(q, ) = δ(q, ). States that defer to themselves must have all their transitions defined. We only onsider D 2 FA that orrespond to minimum state DFA, though the definition applies to all DFA. The funtion F defines a direted graph on the states of Q. A D 2 FA is well defined if and only if there are no yles of length > 1 in this direted graph whih we all a deferment forest. We usep q to denotef(p) = q, i.e. p diretly defers to q. We use p q to denote that there is a path from p toq in the deferment forest defined yf. We usep q to denote the numer of transitions in ommon etween states p andq; i.e. p q = { Σ δ(p,) = δ(q,)}. The total transition funtion for a D 2 FA is defined as { δ ρ(u,) if u, dom(ρ) (u,) = δ (F(u),) else It is easy to see that δ is well defined and equal to δ if the D 2 FA is well defined. 4.2 D 2 FA Merge Algorithm The UCP onstrution merges two DFAs together. We extend the UCP onstrution to merge two D 2 FAs together as follows. During the UCP onstrution, as eah new state u is reated, we define F(u) at that time. We then define ρ to only inlude transitions for u that differ from F(u). To help explain our algorithm, Figure 2 shows an example exeution of the D 2 FA merge algorithm. Figures 2(a) and 2() show the D 2 FA for the REs.*a.* and.*.*, respetively. Figure 2() shows the merged D 2 FA for the D 2 FAs in figures 2(a) and 2(). We use the following onventions when depiting a D 2 FA. The dashed lines orrespond to the deferred state for a given state. For eah state in the merged D 2 FA, the pair of numers aove the line refer to the states in the original D 2 FAs that orrespond to the state in the merged D 2 FA. The numer elow the line is the state in the merged D 2 FA. The numer(s) after the / in aepting states give the id(s) of the pattern(s) mathed. Figure 2(d) shows how the deferred state is set for a few states in the merged D 2 FAs D 3. We explain the notation in this figure as we give our algorithm desription. For eah state u D 3, we set the deferred state F(u) as follows. While merging D 2 FAs D 1 and D 2, let state u = p 0,q 0 e the new state urrently eing added to the merged D 2 FAD 3. Letp 0 p 1 p l e the maximal deferment haindc 1 (i.e. p l defers to itself) ind 1 starting at p 0, and q 0 q 1 q m e the maximal deferment hain DC 2 in D 2 starting at q 0. For example, in Figure 2 (d), we see the maximal deferment hains for u = 5 = 0,2, u = 7 = 2,2, u = 9 = 4,2, and u = 12 = 4,4. For u = 9 = 4,2, the top row is the deferment hain of state 4 ind 1 and the ottom row is the deferment hain of state2 in D 2. We will hoose some state p i,q j where 0 i l and 0 j m to e F(u). In Figure 2(d), we represent these andidate F(u) pairs with edges etween the nodes of the deferment hains. For eah andidate pair, the numer on the top is the orresponding state numer in D 3 and the numer on the ottom is the numer of ommon transitions in D 3 etween that pair and state u. For example, for u = 9 = 4, 2, the two andidate pairs represented are state 7 ( 2,2 ) whih shares 256 transitions in ommon with state 9 and state 4 ( 1,1 ) whih shares 255 transitions in ommon with state 9. Note that a andidate pair

8 is only onsidered if it is reahale in D 3. In Figure 2(d) with u = 9 = 4, 2, three of the andidate pairs orresponding to 4,1, 2,1, and 1,2 are not reahale, so no edge is inluded for these andidate pairs. Ideally, we want i and j to e as small as possile though not oth 0. For example, our est hoies are typially p 0,q 1 or p 1,q 0. In the first ase, p 0 p 1 = p 0,q 0 p 1,q 0, and we already have p 0 p 1 in D 1. In the seond ase, q 0 q 1 = p 0,q 0 p 0,q 1, and we already have q 0 q 1 in D 2. In Figure 2 (d), we set F(u) to e p 0,q 1 for u = 5 = 0,2 and u = 12 = 4,4, and we use p 1,q 0 for u = 9 = 4,2. However, it is possile that oth states are not reahale from the start state in D 3. This leads us to onsider other possile p i,q j. For example, in Figure 2 (d), oth 2,1 and 1,2 are not reahale in D 3, so we use reahale state 1,1 asf(u) foru = 7 = 2,2. We onsider a few different algorithms for hoosing p i,q j. The first algorithm whih we all the first math method is to find a pair of states (p i,q j ) for whih p i,q j Q 3 and i+j is minimum. Stated another way, we find the minimumz 1 suh that the set of statesz = { p i,q z i (max(0,z m) i min(l,z)) ( p i,q z i Q 3 )}. From the set of states Z, of whih there are at most two hoies, we hoose the state that has the most transitions in ommon with p 0,q 0 reaking ties aritrarily. If Z is empty for all z > 1, then we just pik p 0,q 0, i.e. the deferment pointer is not set (or the state defers to itself). The idea ehind the first math method is that p 0,q 0 p i,q j dereases as i + j inreases. In Figure 2(d), all the seleted F(u) orrespond to the first math method. A seond more omplete algorithm for setting F(u) is the est math method where we always onsider all (l + 1) (m + 1) 1 pairs and pik the pair that is in Q 3 and has the most transitions in ommon with p 0,q 0. The idea ehind the est math method is that it is not always true that p 0,q 0 p x,q y p 0,q 0 p x+i,q y+j fori+j > 0. For instane we an havep 0 p 2 < p 0 p 3, whih would mean p 0,q 0 p 2,q 0 < p 0,q 0 p 3,q 0. In suh ases, the first math method will not find the pair along the deferment hains with the most transitions in ommon with p 0,q 0. In Figure 2(d), all the seletedf(u) also orrespond to the est math method. It is diffiult to reate a small example where first math and est math differ. When adding the new state u to D 3, it is possile that some state pairs along the deferment hains that were not in Q 3 while finding the deferred state for u will later on e added to Q 3. This means that after all the states have een added to Q 3, the deferment for u an potentially e improved. Thus, after all the states have een added, for eah state we again find a deferred state. If the new deferred state is etter than the old one, we reset the deferment to the new deferred state. Algorithm 1 shows the pseudoode for the D 2 FA merge algorithm with the first math method for hoosing a deferred state. Note that we use u and u 1,u 2 interhangealy to indiate a state in the merged D 2 FA D 3 whereuis a state in Q 3, andu 1 andu 2 are the states in Q 1 andq 2, respetively, that state u orresponds to. Algorithm 1: D2FAMerge(D 1,D 2 ) Input: A pair of D 2 FAs,D 1 = (Q 1,Σ,ρ 1,q 0 1,M1,F1) and D 2 = (Q 2,Σ,ρ 2,q 0 2,M2,F2), orresponding to RE sets, say R 1 and R 2, withr 1 R 2 =. Output: A D 2 FA orresponding to the RE set R 1 R 2 1 Initialize D 3 to an empty D 2 FA; 2 Initialize queue as an empty queue; 3 queue.push( q 01,q 02 ); 4 while queue not empty do 5 u = u 1, u 2 := queue.pop(); 6 Q 3 := Q 3 {u}; 7 foreah Σ do 8 nxt := δ 1(u 1, ),δ 2(u 2, ) ; 9 if nxt / Q 3 nxt / queue then queue.push (nxt); 10 Add (u, ) nxt transition toρ 3; 11 M 3(u) := M 1(u 1) M 2(u 2); 12 F 3(u) :=FindDefState(u); 13 Remove transitions for u from ρ 3 that are in ommon withf 3(u); 14 foreah u Q 3 do 15 newdptr := FindDefState(u); 16 if (newdptr F 3(u)) (newdptr u > F 3(u) u) then 17 F 3(u) := newdptr; 18 Reset all transitions for u inρ 3 and then remove ones that are in ommon withf 3(u); 19 returnd 3; 20 FindDefState( v 1,v 2 ) 21 Let p 0 = v 1,p 1,...,p l e the list of states on the deferment hain from v 1 to the root ind 1; 22 Let q 0 = v 2,q 1,...,q m e the list of states on the deferment hain from v 2 to the root ind 2; 23 for z = 1 to(l+m) do 24 S := { p i,q z i (max(0,z m) i min(l,z)) ( p i,q z i Q 3)}; 25 ifs then returnargmax v S ( v 1,v 2 v); 26 return v 1,v 2 ; 4.3 Diret D 2 FA onstrution for RE set Similar to effiient DFA onstrution, we first uild the D 2 FA for eah RE inrusing the method desried in [20]. We then merge the D 2 FAs together using a alaned inary tree struture to minimize the worst-ase numer of merges that any RE experienes.

9 5 D 2 FA Merge Algorithm Properties 5.1 Proof of Corretness The D 2 FA merge algorithm exatly follows the UCP onstrution to reate the states. So the orretness of the underlying DFA follows from the the orretness of the UCP onstrution. Theorem 5.1 shows that the merged D 2 FA is also well defined (no yles in deferment forest). Lemma 5.1. In the D 2 FA D 3 = D2FAMerge(D 1,D 2 ), u 1,u 2 v 1,v 2 u 1 v 1 u 2 v 2. Proof. If u 1,u 2 = v 1,v 2 then the lemma is trivially true. Otherwise, let u 1,u 2 w 1,w 2 v 1,v 2 e the deferment hain in D 3. When seleting the deferred state for u 1,u 2, D2FA Merge always hoose a state that orresponds to a pair of states along deferment hains for u 1 andu 2 ind 1 andd 2, respetively. Therefore, we have that u 1,u 2 w 1,w 2 u 1 w 1 u 2 w 2. By indution on the length of the deferment hain and the fat that the relation is transitive, we get our result. Theorem 5.1. If D 2 FAs D 1 and D 2 are well defined, then the D 2 FA D 3 = D2FAMerge(D 1,D 2 ) is also well defined. Proof. Sine D 1 and D 2 are well defined, there are no yles in their deferment forests. Now assume that D 3 is not well defined, i.e. there is a yle in its deferment forest. Let u 1,u 2 and v 1,v 2 e two distint states on the yle. Then, we have that u 1,u 2 v 1,v 2 v 1,v 2 u 1,u 2 Using Lemma 5.1 we get (u 1 v 1 u 2 v 2 ) (v 1 u 1 v 2 u 2 ) i.e. (u 1 v 1 v 1 u 1 ) (u 2 v 2 v 2 u 2 ) Sine u 1,u 2 v 1,v 2, we have u 1 v 1 u 2 v 2 whih implies that at least one of D 1 or D 2 has a yle in their deferment forest whih is a ontradition. 5.2 Limiting Deferment Depth Sine no input is onsumed while traversing a deferred transition, in the worst ase, the numer of lookups needed to proess one input harater is given y the depth of the deferment forest. As previously proposed, we an guarantee a worst ase performane y limiting the depth of the deferment forest. For a state u 1 of a D 2 FA D 1, the deferment depth of u 1, denoted as ψ(u 1 ), is the length of the maximal deferment hain in D 1 from u 1 to the root. Ψ(D 1 ) = max v Q1 ψ(v) denotes the deferment depth of D 1 (i.e. the depth of the deferment forest ind 1 ). Lemma 5.2. In the D 2 FA D 3 = D2FAMerge(D 1,D 2 ), u 1,u 2 Q 3, ψ( u 1,u 2 ) ψ(u 1 )+ψ(u 2 ). Proof. Let ψ( u 1,u 2 ) = d. If ψ( u 1,u 2 ) = 0, then u 1,u 2 is a root and the lemma is trivially true. So, we onsider d 1 and assume the lemma is true for all states with ψ < d. Let u 1,u 2 w 1,w 2 v 1,v 2 e the deferment hain in D 3. Using the indutive hypothesis, we have ψ( w 1,w 2 ) ψ(w 1 )+ψ(w 2 ) Given u 1,u 2 w 1,w 2, we assume without loss of generality thatu 1 w 1. Using Lemma 5.1 we get thatu 1 w 1. Therefore ψ(w 1 ) ψ(u 1 ) 1. Comining the aove, we getψ( u 1,u 2 ) = ψ( w 1,w 2 )+1 ψ(w 1 )+ψ(w 2 )+1 (ψ(u 1 ) 1)+ψ(u 2 )+1 ψ(u 1 )+ψ(u 2 ). Lemma 5.2 diretly gives us the following Theorem. Theorem 5.2. If D 3 = D2FAMerge(D 1,D 2 ), then Ψ(D 3 ) Ψ(D 1 )+Ψ(D 2 ). For an RE set R, if the initial D 2 FAs haveψ = d, in the worst ase, the final merged D 2 FA orresponding to R an haveψ = d R. Although Theorem 5.2 gives the value of Ψ in the worst ase, in pratial ases, Ψ(D 3 ) is very lose to max(ψ(d 1 ),Ψ(D 2 )). Thus the deferment depth of the final merged D 2 FA is usually not muh higher thand. Let Ω denote the desired upper ound on Ψ. To guarantee Ψ(D 3 ) Ω, we modify thefinddefstate suroutine in Algorithm 1 as follows: When seleting andidate pairs for the deferred state, we only onsider states with ψ < Ω. Speifially, we replae line 24 with the following S := { p i,q z i (max(0,z m) i min(l,z)) p i,q z i Q 3) (ψ( p i,q z i ) < Ω)} When we do the seond pass (lines 14-19), we may inrease the deferment depth of nodes that defer to nodes that we readjust. We reord the affeted nodes and then do a third pass to reset their deferment states so that the maximum depth ound is satisfied. In pratie, this happens very rarely. When onstruting a D 2 FA with a given ound Ω, we first uild D 2 FAs without this ound. We only apply the oundω when performing the final merge of two D 2 FAs to reate the final D 2 FA. 5.3 Deferment to a Lower Level In [6], the authors propose a tehnique to guarantee an amortized ost of 2 lookups per input harater without limiting the depth of the deferment tree. They ahieve this y having states only defer to lower level states where the level of any state u in a DFA (or D 2 FA), denoted level(u), is defined as the length of the shortest string that ends in that

10 state (from the start state). More formally, they ensure that for all states u, level(u) > level(f(u)) if u F(u). We all this property the ak-pointer property. If the akpointer property holds, then every deferred transition taken dereases the level of the urrent state y at least 1. Sine a regular transition on an input harater an only inrease the level of the urrent state y at most 1, there have to e fewer deferred transitions taken on the entire input string than regular transitions. This gives an amortized ost of at most2transitions taken per input harater. In order to guarantee the D 2 FA D 3 has the akpointer property, we perform a similar modifiation to the FindDefState suroutine in Algorithm 1 as we performed when we wanted to limit the maximum deferment depth. When seleting andidate pairs for the deferred state, we only onsider states with a lower level. Speifially, we replae line 24 with the following: S := { p i,q z i (max(0,z m) i min(l,z)) ( p i,q z i Q 3) (level( v 1,v 2 ) > level( p i,q z i ))} For states for whih no andidate pairs are found, we just searh through all states in Q 3 that are at a lower level for the deferred state. In pratie, this searh through all the states needs to e done for very few states eause if D 2 FAs D 1 and D 2 have the ak-pointer property, then almost all the states in D 2 FAs D 3 have the ak-pointer property. As with limiting maximum deferment depth, we only apply this restrition when performing the final merge of two D 2 FAs to reate the final D 2 FA. 5.4 Algorithmi Complexity The time omplexity of the original D 2 FA algorithm proposed in [17] iso(n 2 (log(n)+ Σ )). The SRG haso(n 2 ) edges, and O( Σ ) time is required to add eah edge to the SRG and O(log(n)) time is required to proess eah edge in the SRG during the maximum spanning tree routine. The time omplexity of the D 2 FA algorithm proposed in [6] is O(n 2 Σ ). Eah state is ompared with O(n) other states, and eah omparison requires O( Σ ) time. The time omplexity of our new D2FAMerge algorithm to merge two D 2 FAs iso(nψ 1 Ψ 2 Σ ), wherenis the numer of states in the merged D 2 FA, and Ψ 1 and Ψ 2 are the maximum deferment depths of the two input D 2 FAs. When setting the deferment for any state u = u 1,u 2, in the worst ase the algorithm ompares u 1,u 2 with all the pairs along the deferment hains of u 1 and u 2, whih are at most Ψ 1 and Ψ 2 in length, respetively. Eah omparison requires O( Σ ) time. In pratie, the time omplexity is O(n Σ ) as eah state needs to e ompared with very few states for the following three reasons. First, the maximum deferment depth Ψ is usually very small. The largest value of Ψ among our 8 primary RE sets in Setion 6 is 7. Seond, the length of the deferment hains for most states is muh smaller than Ψ. The largest value of average deferment depth ψ among our 8 RE sets is Finally, many of the state pairs along the deferment hains are not reahale in the merged D 2 FA. Among our 8 RE sets, the largest value of the average numer of omparisons needed is When merging all the D 2 FAs together for an RE set R, the total time required in the worst ase would e O(nΨ 1 Ψ 2 Σ log( R )). The worst ase would happen when the RE set ontains strings and there is no state explosion. In this ase, eah merged D 2 FA would have a numer of states roughly equal to the sum of the sizes of the D 2 FAs eing merged. When there is state explosion, the last D 2 FA merge would e the dominating fator, and the total time would just eo(nψ 1 Ψ 2 Σ ). When modifying the D2FAMerge algorithm to maintain ak-pointers, the worst ase time would e O(n 2 Σ ) eause we would have to ompare eah state witho(n) other states if none of the andidate pairs are found at a lower level than the state. In pratie, this searh needs to e done for very few states, typially less than1%. 6 Experimental Results In this setion, we evaluate the effetiveness of our algorithm (D 2 FAMERGE) on real-world and syntheti RE sets. We ompare D 2 FAMERGE with the original D 2 FA algorithm proposed in [17] (ORIGINAL) that optimizes transition ompression and the D 2 FA algorithm proposed in [6] (BACKPTR) that enfores the ak-pointer property desried in Setion Methodology Data Sets Our main results are ased on eight real RE sets, four proprietary RE sets C7, C8, C10, and C613 from a large networking vendor and four puli RE sets Bro217, Snort 24, Snort31, and Snort 34, that we partition into three groups, STRING, WILDCARD, and SNORT, ased upon their RE omposition. For eah RE set, the numer indiates the numer of REs in the RE set. The STRING RE sets, C613 and Bro217, ontain mostly string mathing REs. The WILDCARD RE sets, C7, C8 and C10, ontain mostly REs with multiple wildard losures.*. The SNORT RE sets, Snort24, Snort31, and Snort34, ontain a more diverse set of REs, roughly 40% of whih have wildard losures. To test salaility, we use Sale, a syntheti RE set onsisting of26 REs of the form /.* u * l 789!#%&/, where u and l are the 26 upperase and lowerase alphaet letters. Even though all the REs are nearly idential differing only in the harater after the two.* s, we still get

11 the full multipliative effet where the numer of states in the orresponding minimum state DFA roughly doules for every RE added Metris We use the following metris to evaluate the algorithms. First, we measure the resulting D 2 FA size (# transitions) to assess transition ompression performane. Our D 2 FAMERGE algorithm typially performs almost as well as the other algorithms even though it uilds up the D 2 FA inrementally rather than ompressing the final minimum state DFA. Seond, we measure the the maximum deferment depth (Ψ) and average deferment depth (ψ) in the D 2 FA to assess how quikly the resulting D 2 FA an e used to perform regular expression mathing. Smaller Ψ and ψ mean that fewer deferment transitions that proess no input haraters need to e traversed when proessing an input string. Our D 2 FAMERGE signifiantly outperforms the other algorithms. Finally, we measure the spae and time required y the algorithm to uild the final automaton. Again, our D 2 FAMERGE signifiantly outperforms the other algorithms. When omparing the performane of D 2 FAMERGE with another algorithm A on a given RE or RE set, we define the following quantities to ompare them: transition inrease is (D 2 FAMERGE D 2 FA size - A D 2 FA size) divided yad 2 FA size, transition derease is (A D 2 FA size - D 2 FAMERGE D 2 FA size) divided yad 2 FA size, average (maximum) deferment depth ratio is A average (maximum) deferment depth divided y D 2 FAMERGE average (maximum) deferment depth, spae ratio is A spae divided y D 2 FAMERGE spae, and time ratio isauild time divided y D 2 FAMERGE uild time Measuring Spae When measuring the required spae for an algorithm, we measure the maximum amount of memory required at any point in time during the onstrution and then final storage of the automaton. This is a diffiult quantity to measure exatly; we approximate this required spae for eah of the algorithms as follows. For D 2 FAMERGE, the dominant data struture is the D 2 FA. For a D 2 FA, the transitions for eah state an e stored as pairs of input harater and next state id, so the memory required to store a D 2 FA is alulated as = (#transitions) 5 ytes. However, the maximum amount of memory required while running D 2 FAMERGE may e higher than the final D 2 FA size eause of the following two reasons. First, when merging two D 2 FAs, we need to maintain the two input D 2 FAs as well as the output D 2 FA. Seond, we may reate an intermediate output D 2 FA that has more transitions than needed; these extra transitions will e eliminated one all D 2 FA states are added. We keep trak of the worst ase required spae for our algorithm during D 2 FA onstrution. This typially ours when merging the final two intermediate D 2 FA to form the final D 2 FA. For ORIGINAL, we measure the spae required y the minimized DFA and the SRG. For the DFA, the transitions for eah state an e stored as an array of size Σ with eah array entry requiring four ytes to hold the next state id. For the SRG, eah edge requires 17 ytes as oserved in [6]. This leads to a required memory for uilding the D 2 FA of = Q Σ 4+(#edgesin SRG) 17 ytes. For BACKPTR, we onsider two variants. The first variant uilds the minimized DFA diretly from the NFA and then sets the deferment for eah state. For this variant, no SRG is needed, so the spae required is the spae needed for the minimized DFA whih is Q Σ 4 ytes. The seond variant goes diretly from the NFA to the final D 2 FA; this variant uses less spae ut is muh slower as it stores inomplete transition tales for most states. Thus, when omputing the deferment state for a new state, the algorithm must rereate the omplete transition tales for eah state to determine whih has the most ommon transitions with the new state. For this variant, we assume the only spae required is the spae to store the final D 2 FA whih is = (#transitions) 5 ytes even though more memory is definitely needed at various points during the omputation. We also note that oth implementations must perform the NFA to DFA suset onstrution on a large NFA whih means even the faster variant runs muh more slowly than D 2 FAMERGE Corretness We tested orretness of our algorithms y verifying the final D 2 FA is equivalent to the orresponding DFA. Note, we an only do this hek for our RE sets where we were ale to ompute the orresponding DFA. Thus, we only verified orretness of the final D 2 FA for our eight real RE sets and the smaller Sale RE sets. 6.2 D 2 FAMERGE versus ORIGINAL We first ompare D 2 FAMERGE with ORIGINAL that optimizes transition ompression when oth algorithms have unlimited maximum deferment depth. These results are shown in Tale 1 for our 8 primary RE sets. Tale 2 summarizes these results y RE group. We make the following oservations. (1) D 2 FAMERGE uses muh less spae than ORIGINAL. On average, D 2 FAMERGE uses 1390 times less memory than ORIGINAL to uild the resulting D 2 FA. This differene is most extreme when the SRG is large, whih is true for the two STRING RE sets and Snort24 and Snort34. For these RE sets, D 2 FAMERGE uses etween 347 and 3613

12 RE set ORIGINAL D 2 FAMERGE # Def. depth RAM Time Def. depth RAM Time States # Trans # Trans Avg. Max. (MB) (s) Avg. Max. (MB) (s) Bro C C C C Snort Snort Snort Tale 1. The D 2 FA size, D 2 FA average ψ and maximum Ψ deferment depths, spae estimate and time required to uild the D 2 FA for ORIGINAL and D 2 FAMERGE on our eight primary RE sets. RE set group Trans Def. depth ratio Spae Time inrease Avg. Max. ratio ratio All 20.1% STRING 44.0% WILDCARD 3.0% SNORT 21.3% Tale 2. Average values of transition inrease, deferment depth ratios, spae ratios, and time ratios for D 2 FAMERGE ompared with ORIGINAL for our RE set groups. times less memory than ORIGINAL. For the RE sets with relatively small SRGs suh as those in the WILDCARD and Snort31, D 2 FAMERGE uses etween 35 and 49 times less spae than ORIGINAL. (2) D 2 FAMERGE is muh faster than ORIGINAL. On average, D 2 FAMERGE uilds the D 2 FA 300 times faster than ORIGINAL. This time differene is maximized when the deferment hains are shortest. For example, D 2 FAMERGE only requires an average of 0.12 mse and 0.19 mse per state for the WILDCARD and SNORT RE sets, respetively, so D 2 FAMERGE is, on average, 338 and 399 times faster than ORIGINAL for these RE sets, respetively. For the STRING RE sets, the deferment hains are longer, so D 2 FAMERGE requires an average of 1.23 mse per state, and is, on average,100 times faster than ORIGINAL. (3) D 2 FAMERGE produes D 2 FA with muh smaller average and maximum deferment depths than ORIGINAL. On average, D 2 FAMERGE produes D 2 FA that have average deferment depths that are 7.3 times smaller than ORIGI- NAL and maximum deferment depths that are 4.8 times smaller than ORIGINAL. In partiular, the average deferment depth for D 2 FAMERGE is less than 2 for all ut the two STRING RE sets, where the average deferment depths are 2.16 and Thus, the expeted numer of deferment transitions to e traversed when proessing a length n string is less than n. One reason D 2 FAMERGE works so well is that it eliminates low weight edges from the SRG so that the deferment forest has many shallow deferment trees instead of one deep tree. This is partiularly effetive for the WILDCARD RE sets and, to a lesser extent, the SNORT RE sets. For the STRING RE sets, the SRG is fairly dense, so D 2 FAMERGE has a smaller advantage relative to ORIG- INAL. (4) D 2 FAMERGE produes D 2 FA with only slightly more transitions than ORIGINAL, partiularly on the RE sets that need transition ompression the most. On average, D 2 FAMERGE produes D 2 FA with roughly 20% more transitions than ORIGINAL does. D 2 FAMERGE works est when state explosion from wildard losures reates DFA omposed of many similar repeating sustrutures. This is preisely when transition ompression is most needed. For example, for the WILDCARD RE sets that experiene the greatest state explosion, D 2 FAMERGE only has 3% more transitions than ORIGINAL. On the other hand, for the STRING RE sets, D 2 FAMERGE has, on average, 44% more transitions. For this group, ORIGINAL needed to uild a very large SRG and thus used muh more spae and time to ahieve the improved transition ompression. Furthermore, transition ompression is typially not needed for suh RE sets as all string mathing REs an e plaed into a single group and the resulting DFA an e uilt. In summary, D 2 FAMERGE ahieves its est performane relative to ORIGINAL on the WILDCARD RE sets (exept for spae used for onstrution of the D 2 FA) and its worst performane relative to ORIGINAL on the STRING RE sets (exept for spae used to onstrut the D 2 FA). This is desirale as the spae and time effiient D 2 FAMERGE is most needed on RE sets like those in the WILDCARD eause those RE sets experiene the greatest state explosion. 6.3 D 2 FAMERGE versus ORIGINAL with Bounded Maximum Deferment Depth We now ompare D 2 FAMERGE and ORIGINAL when they impose a maximum deferment depth ound Ω of 1, 2, and 4. Beause time and spae do not hange signifiantly, we fous only on numer of transitions and average defer-

13 ORIGINAL D 2 FAMERGE RE # Trans Avg. def. depth # Trans Avg. def. depth set Ω = 1 Ω = 2 Ω = 4 Ω=1 Ω=2 Ω=4 Ω = 1 Ω = 2 Ω = 4 Ω=1 Ω=2 Ω=4 B C C C C S S S Tale 3. The D 2 FA size and D 2 FA average ψ deferment depth for ORIGINAL and D 2 FAMERGE on our eight primary RE sets given maximum deferment depth ounds of 1, 2 and 4. ment depth. These results are shown in Tale 3. Note that for these data sets, the resulting maximum depth Ψ typially is idential to the maximum depth ound Ω; the only exeption is for D 2 FAMERGE andω = 4; thus we omit the maximum deferment depth from Tale 3. Tale 4 summarizes the results y RE group highlighting how muh etter or worse D 2 FAMERGE does than ORIGINAL on the two metris of numer of transitions and average deferment depth ψ. Overall, D 2 FAMERGE performs very well when presented a ound Ω. In partiular, the average inrease in the numer of transitions for D 2 FAMERGE with Ω equal to 1, 2 and 4, is only 108%, 14% and 0.7% respetively, ompared to D 2 FAMERGE with unounded maximum deferment depth. Stated another way, when D 2 FAMERGE is required to have a maximum deferment depth of 1, this only results in slightly more than twie the numer of transitions in the resulting D 2 FA. The orresponding values for ORIG- INAL are 3095%, 1099% and 119%. Ω = 1 Ω = 2 Ω = 4 RE set group Trans Avg. def. Trans Avg. def. Trans Avg. dptr der. depth ratio der. depth ratio der. len ratio All 91.3% % % 1.5 STRING 90.0% % % 1.0 WILDCARD 89.3% % % 2.1 SNORT 94.0% % % 1.4 Tale 4. Average values of transition derease and average deferment depth ratios for D 2 FAMERGE ompared with ORIGINAL for our RE set groups given maximum deferment depth ounds of 1, 2 and 4. These results an e partially explained y examining the average deferment depth data. Unlike in the unounded maximum deferment depth senario, here we see that D 2 FAMERGE has a larger average deferment depth ψ than ORIGINAL exept for the WILDCARD when Ω is 1 or 2. What this means is that D 2 FAMERGE has more states that defer to at least one other state than ORIGINAL does. This leads to the lower numer of transitions in the final D 2 FA. Overall, for Ω = 1, D 2 FAMERGE produes D 2 FA with roughly 90% fewer transitions than ORIGINAL for all RE set groups. For Ω = 2, D 2 FAMERGE produes D 2 FA with roughly 50% fewer transitions than ORIGINAL for the WILDCARD RE sets and roughly 90% fewer transitions than ORIGINAL for the other RE sets. 6.4 D 2 FAMERGE versus BACKPTR We now ompare D 2 FAMERGE with BACKPTR whih enfores the ak-pointer property desried in Setion 5.3. We adapt D 2 FAMERGE to also enfore this ak-pointer property. The results for all our metris are shown in Tale 5 for our 8 primary RE sets. We onsider the two variants of BACKPTR desried in Setion 6.1.3, one whih onstruts the minimum state DFA orresponding to the given NFA and one whih ypasses the minimum state DFA and goes diretly to the D 2 FA from the given NFA. We note the seond variant appears to use less spae than D 2 FAMERGE. This is partially true sine BACKPTR reates a smaller D 2 FA than D 2 FAMERGE. However, we underestimate the atual spae used y this BACKPTR variant y simply assuming its required spae is the final D 2 FA size. We ignore, for instane, the spae required to store intermediate omplete tales or to perform the NFA to DFA suset onstrution. Tale 6 summarizes these results y RE group displaying ratios for many of our metris that highlight how muh etter or worse D 2 FAMERGE does than BACKPTR. Similar to D 2 FAMERGE versus ORIGINAL, we find that D 2 FAMERGE with the akpointer property performs well when ompared with oth variants of BACKPTR. Speifially, with an average inrease in the numer of transitions of roughly 23%, D 2 FAMERGE runs on average 27 times faster than the fast variant of BACKPTR and 120 times faster than the slow variant of BACKPTR. For spae, D 2 FAMERGE uses on average almost 31 times less spae than the first variant of BACKPTR and on average roughly 42% more spae than the seond variant of BACKPTR. Furthermore, D 2 FAMERGE reates D 2 FA with average deferment depth 2.9 times smaller than BACKPTR and maximum deferment depth 1.9 times smaller than BACKPTR. As was the ase with ORIGINAL, D 2 FAMERGE ahieves its est performane relative to BACKPTR on the WILD-

14 BACKPTR D 2 FAMERGE with ak-pointer RE Def. depth RAM Time RAM2 Time2 Def. depth RAM Time set # Trans # Trans Avg. Max. (MB) (s) (MB) (s) Avg. Max. (MB) (s) B C C C C S S S Tale 5. The D 2 FA size, D 2 FA average ψ and maximum Ψ deferment depths, spae estimate and time required to uild the D 2 FA for oth variants of BACKPTR and D 2 FAMERGE with the ak pointer property on our eight primary RE sets. Memory required to uild CARD RE sets and its worst performane relative to BACKPTR on the STRING RE sets. This is desirale as the spae and time effiient D 2 FAMERGE is most needed on RE sets like those in the WILDCARD eause those RE sets experiene the greatest state explosion. RAM (MB) ORIGINAL D 2 FAMERGE RE set group Trans Def. depth ratio Spae Time Spae2 Time2 inrease Avg. Max. ratio ratio ratio ratio All 23.4% STRING 45.0% WILDCARD 1.3% SNORT 31.0% Tale 6. Average values of transition inrease, deferment depth ratios, spae ratios, and time ratios for D 2 FAMERGE ompared with oth variants of BACKPTR for RE set groups. 6.5 Salaility results Finally, we assess the improved salaility of D 2 FAMERGE relative to ORIGINAL using the Sale RE set assuming that we have a maximum memory size of 1 GB. For oth ORIGINAL and D 2 FAMERGE, we add one RE at a time from Sale until the spae estimate to uild the D 2 FA goes over the 1GB limit. For ORIGINAL, we are ale only ale to add 12 REs; the final D 2 FA has 397, 312 states and requires over 71 hours to ompute. As explained earlier, we inlude the SRG edges in the RAM size estimate. If we exlude the SRG edges and only inlude the DFA size in the RAM size estimate, we would only e ale to add one more RE efore we reah the 1GB limit. For D 2 FAMERGE, we are ale to add 19 REs; the final D 2 FA has 80,216,064 states and requires only 77 minutes to ompute. This data set highlights the quadrati versus linear running time of ORIGINAL and D 2 FAMERGE, respetively. Figure 3 shows how the spae and time requirements grow for ORIGINAL and D 2 FAMERGE as RE s from Sale are added one y one until 19 have een added. Time (s) e #REs Time required to uild #REs ORIGINAL D 2 FAMERGE Figure 3. Memory and time required to uild D 2 FA versus numer of Sale REs used for ORIGINAL s D 2 FA and D 2 FAMERGE s D 2 FA. 7 Conlusions In this paper, we propose a novel Minimize then Union framework for onstruting D 2 FAs using D 2 FA merging. Our approah requires a fration of memory and time ompared to urrent algorithms. This allows us to uild muh larger D 2 FAs than was possile with previous tehniques. Our algorithm naturally supports frequent RE set updates. We onduted experiments on real-world and syntheti RE sets that verify our laims. For example, our algorithm requires an average of1400 times less memory and300 times less time than the original D 2 FA onstrution algorithm of Kumar et al.. We elieve our Minimize then Union framework an e inorporated with other alternative automata for RE mathing.

15 Referenes [1] Snort. [2] A. V. Aho and M. J. Corasik. Effiient string mathing: an aid to iliographi searh. Communiations of the ACM, 18(6): , [3] M. Aliherry, M. Muthuprasanna, and V. Kumar. High speed pattern mathing for network ids/ips. In Pro IEEE International Conferene on Network Protools, pages Ieee, [4] M. Behi and S. Cadami. Memory-effiient regular expression searh using state merging. In Pro. INFOCOM. IEEE, [5] M. Behi and P. Crowley. A hyrid finite automaton for pratial deep paket inspetion. In Pro. CoNext, [6] M. Behi and P. Crowley. An improved algorithm to aelerate regular expression evaluation. In Pro. ACM/IEEE ANCS, [7] M. Behi and P. Crowley. Effiient regular expression evaluation: Theory to pratie. In Pro. ACM/IEEE ANCS, [8] M. Behi and P. Crowley. Extending finite automata to effiiently math perl-ompatile regular expressions. In Pro. ACM CoNEXT, Artile Numer 25. [9] A. Bremler-Barr, D. Hay, and Y. Koral. Compatdfa: Generi state mahine ompression for salale pattern mathing. In Pro. IEEE INFOCOM, pages 1 9. Ieee, [10] B. C. Brodie, D. E. Taylor, and R. K. Cytron. A salale arhiteture for high-throughput regular-expression pattern mathing. SIGARCH Computer Arhiteture News, [11] C. R. Clark and D. E. Shimmel. Effiient reonfigurale logi iruits for mathing omplex network intrusion detetion patterns. In Pro. Field-Programmale Logi and Appliations, pages , [12] C. R. Clark and D. E. Shimmel. Salale pattern mathing for high speed networks. In Pro. 12th Annual IEEE Symposium on Field-Programmale Custom Computing Mahines (FCCM), Washington, DC, [13] J. E. Hoproft. The Theory of Mahines and Computations, hapter An nlogn algorithm for minimizing the states in a finite automaton, pages Aademi Press, [14] J. E. Hoproft, R. Motwani, and J. D. Ullman. Introdution to Automata Theory, Languages, and Computation. Addison-Wesley, [15] S. Kong, R. Smith, and C. Estan. Effiient signature mathing with multiple alphaet ompression tales. In Pro. 4th Int. Conf. on Seurity and privay in ommuniation netowrks (SeureComm), page 1. ACM Press, [16] S. Kumar, B. Chandrasekaran, J. Turner, and G. Varghese. Curing regular expressions mathing algorithms from insomnia, amnesia, and aalulia. In Pro. ACM/IEEE ANCS, pages , [17] S. Kumar, S. Dharmapurikar, F. Yu, P. Crowley, and J. Turner. Algorithms to aelerate multiple regular expressions mathing for deep paket inspetion. In Pro. SIG- COMM, pages , [18] S. Kumar, J. Turner, and J. Williams. Advaned algorithms for fast and salale deep paket inspetion. In Pro. IEEE/ACM ANCS, pages 81 92, [19] Y. Liu, L. Guo, M. Guo, and P. Liu. Aelerating DFA onstrution y hierarhial merging. In Pro. IEEE Ninth International Symposium on Parallel and Distriuted Proessing with Appliations, [20] C. R. Meiners, J. Patel, E. Norige, E. Torng, and A. X. Liu. Fast regular expression mathing using small tams for network intrusion detetion and prevention systems. In Pro. 19th USENIX Seurity Symposium, Washington, DC, August [21] A. Mitra, W. Najjar, and L. Bhuyan. Compiling PCRE to FPGA for aelerating SNORT IDS. In Pro. ACM/IEEE ANCS, [22] J. Mosola, J. Lokwood, R. P. Loui, and M. Pahos. Implementation of a ontent-sanning module for an internet firewall. In Pro. IEEE Field Programmale Custom Computing Mahines, [23] V. Paxson. Bro: a system for deteting network intruders in real-time. Computer Networks, 31(23-24): , [24] M. Roesh. Snort: Lightweight intrusion detetion for networks. In Pro. 13th Systems Administration Conferene (LISA), USENIX Assoiation, pages , [25] R. Sidhu and V. K. Prasanna. Fast regular expression mathing using fpgas. In Pro. FCCM, pages , [26] R. Smith, C. Estan, and S. Jha. Xfa: Faster signature mathing with extended automata. In Pro. IEEE Symposium on Seurity and Privay, pages , [27] R. Smith, C. Estan, S. Jha, and S. Kong. Deflating the ig ang: fast and salale deep paket inspetion with extended finite automata. In Pro. SIGCOMM, pages , [28] R. Sommer and V. Paxson. Enhaning ytelevel network intrusion detetion signatures with ontext. In Pro. ACM CCS, pages , [29] I. Sourdis and D. Pnevmatikatos. Pnevmatikatos: Fast, large-sale string math for a 10gps fpga-ased network intrusion detetion system. In Pro. Int. on Field Programmale Logi and Appliations, pages , [30] I. Sourdis and D. Pnevmatikatos. Pre-deoded ams for effiient and high-speed nids pattern mathing. In Pro. Field- Programmale Custom Computing Mahines, [31] J.-S. Sung, S.-M. Kang, Y. Lee, T.-G. Kwon, and B.-T. Kim. A multi-gigait rate deep paket inspetion algorithm using tam. In Pro. IEEE GLOBECOM, [32] L. Tan and T. Sherwood. A high throughput string mathing arhiteture for intrusion detetion and prevention. In Pro. 32nd Annual Int. Symposium on Computer Arhiteture (ISCA), pages , [33] N. Tuk, T. Sherwood, B. Calder, and G. Varghese. Deterministi memory-effiient string mathing algorithms for intrusion detetion. In Pro. Infoom, pages , [34] L. Yang, R. Karim, V. Ganapathy, and R. Smith. Fast, memory-effiient regular expression mathing with NFA- OBDDs. Computer Networks, 55(55): , [35] F. Yu, Z. Chen, Y. Diao, T. V. Lakshman, and R. H. Katz. Fast and memory-effiient regular expression mathing for deep paket inspetion. In Pro. ANCS, pages , [36] F. Yu, R. H. Katz, and T. V. Lakshman. Gigait rate paket pattern-mathing using TCAM. In Pro. 12th IEEE Int. Conf. on Network Protools (ICNP), pages , 2004.