DISTRIBUTED DATA PARALLEL TECHNIQUES FOR CONTENT-MATCHING INTRUSION DETECTION SYSTEMS G. Chapman J. Cleee E. Idle ABSTRACT Content matching i a neceary component of any ignature-baed network Intruion Detection Sytem (IDS). Thee packet inpection typically require coniderable delay often conuming more than 70% of the IDS proceing time. Unfortunately, thi delay become more ignificant a ecurity policie and network peed continue to increae. Thi paper introduce a new parallel IDS content matching technique that provide initial packet inpection with le delay. The technique ditribute portion of a packet payload acro an array of n proceor, each reponible for canning a maller amount of original payload. Given thi deign, each proceor ha le data to inpect thu reducing the overall delay. Unlike imilar parallel approache, our technique enure that ecurity i maintained (no fale negative). Furthermore, the propoed parallel technique i hown to reult in an initial match peed-up of approximately.25n uing Snort (an open ource IDS), actual IDS policie, and traffic trace a ignificant improvement over current parallel technique. KEYWORDS Aho-Coraick, Data Parallel, Intruion Detection, Packet, Parallel, Signature Matching, Snort, Wu-Manber INTRODUCTION Intruion Detection Sytem (IDS) inpect arriving packet for maliciou content (ignature) a defined by a ecurity policy. Unfortunately, comparing packet header and payload againt a policy can be complex and time-conuming. For example, it ha been found that content matching (canning for ignature) account for more than 70% of the packet proceing time [2], [7]. Therefore, given the riing number [5] and ophitication of network threat, thee ytem will be forced to operate at much fater peed in the near future. One approach for reducing the content matching time i the ue of better earching algorithm. Three tring -4244-53-06/07/$25.00 c 2007 IEEE earching algorithm are commonly ued for IDS content matching, Aho-Coraick [], Boyer-Moore [4], and Wu-Manber [3]. The Aho-Coraick algorithm ue a finite tate machine for tring matching and provide the bet wort cae performance, but require a ignificant amount of memory for the tate machine. In contrat, Boyer-Moore provide the bet performance when earching for a ingle ignature [6], but cale poorly. The Wu-Manber algorithm can be viewed a a multi-pattern verion of Boyer-Moore, which require le memory than Aho-Coraick and provide a better average cae performance. Although the incluion of thee multi-pattern earch algorithm and the development of new IDS pecific earch algorithm have greatly improved performance, it may not be ufficient for the next generation of high peed network. Parallel content matching offer a calable method for inpecting packet in a high peed environment [3], [8], [4]. Thee ytem typically conit of an array of proceor that are ued to proce packet in parallel. For example, a imple parallel approach would ditribute the arriving packet evenly acro the array of proceor, each having a copy of the complete policy [8]. Uing terminology borrowed from parallel computing, thi i conidered a data parallel approach. While thi technique reduce the amount of packet per proceor, load balancing and maintaining tate (ending packet of one connection to the ame proceor) i difficult [8]. In contrat, another parallel approach divide the packet payload acro the array uch that each proceor inpect a maller piece of the original packet payload [3], [4]. Although potentially fater, if a ignature pan multiple proceor it will not be found [3]. Thi ecurity iue can be reolved if the data aigned to proceor overlap uch that each proceor can oberve the entire ignature [4]. However, a the overlapped data mut be canned more than once thi greatly increae the content earch time. Thi paper decribe a new parallel content matching approach, called Divided Data Parallel (DDP), that divide the payload of a packet acro an array of proceor a decribed in the preceding paragraph [3], of7
[4]. However, the propoed method incorporate a lightweight ynchronization ytem that mitigate the impact of data overlap. The match-bit allow one proceor to quickly indicate to other proceor that a match ha been found for a given packet. Once the notification ha been received, the remaining proceor can tart inpecting another packet. Thi modification caue DDP to perform ignificantly better than other data parallel method. Experimental reult were taken uing Snort (an open ource IDS). Snort policie for web-traffic, and actual traffic trace indicate a traditional data parallel approach uing n proceor, for content matching, reduce the content matching time by approximately 0.75n. In contrat, the DDP algorithm reduce the content matching phae by.25n. The remainder of thi paper i organized a follow. The econd ection give a thorough overview of Network Intruion Detection Sytem (NIDS), Snort rule yntax, and Snort detection. The third ection cover previou parallel technique, while the fourth ection introduce the new DDP approach.the fifth ection how reult of the DDP algorithm compared to other algorithm, and the lat two ection ummarize the contribution of thi paper, and introduce the area of future reearch. NETWORK INTRUSION DETECTION USING SNORT A decribed in the introduction, the purpoe of a ignature-baed IDS i to detect maliciou behavior on a computer ytem or network. Thi i accomplihed by canning packet payload for known pattern or ignature. While thi type of IDS require ignature be know a priori, it remain an important component for ecuring computer ytem. Snort, one of the mot popular IDS, i an open-ource project developed and maintained by Sourcefire[9]. It i commonly ued by both reearch project and commercial product becaue of it eae of ue and veratility. Snort can perform realtime traffic analyi, content matching, and it can detect multiple type of attack. Figure diagram the five primary proceing tage in Snort. During the firt tage Snort receive packet via libpcap from either the network or a uer defined trace file. Once the packet are captured, they are ent through an immediate decoding proce, which fill a tructure baed off of the packet protocol. After packet are decoded they are ent through a preproceing tage, which perform packet analyi and reaembly. For example, if a TCP packet i captured but ha a malformed header, the preproceor can drop the packet from the ytem [0]. After preproceing, content normalization occur. For example, telnet and HTTP are two type of traffic that are normalized. Once normalization i complete, the data move to the mot time-conuming tage of Snort, the content matching tage. In thi tage Snort detection engine ue a robut tring earching algorithm (decribed in the introduction) to compare each packet payload with rule from a ignature file. The ignature file contain a lit of known maliciou ignature, and upon canning, if a ignature i matched the alert engine i notified. A Snort rule expree the action to perform on matching packet/tream. For example, conider the following rule in Figure 3. Each Snort rule conit of three component. The firt identifie the action that mut be taken if there i a match. The econd denote the primary match criterion. In Figure 3, the match criterion identifie any TCP packet detined for the 0...0/24 addre pace and port 222. The third component contain rule option and decribe any additional match criteria (for example, pattern in the payload) and parameter for executing the action. In Figure 3, Snort would earch for the hexadecimal pattern "00 22 33 aa" in the payload. If it i alo a match, the meage "rpcd requet" i generated. Snort allow the pecification of packet header and payload match criteria. It i important to note that the rule option may contain multiple pattern and/or pecification. Snort will firt earch for the longet pattern for each rule, called the initial match. If there i an initial match it i then verified by earching for any additional content and/or verifying pecification decribed by the rule option. If maliciou data i found in the payload the alert engine i notified. The multi-tage approach of Snort i effective in detecting maliciou packet and with parallel technique Snort can be ued on network with high line peed demand. PARALLEL SIGNATURE MATCHING TECHNIQUES The ignature matching tage of Snort account for more than 70% of the proceing time [2], [7]. The ue of fater earching algorithm ha reduced the ignature matching delay; however thee olution are not ufficient for the increaing number of policy rule and network peed. One calable olution for reducing the ignature matching delay i the ue of parallelization. A parallel IDS conit of an array of n proceor (thi 2of7
packet capture pre-proce content normalization ignature matching alert tate info Figure. High-level proceing tage of Snort IDS. alert udp any any -> 0...0/24 222 (content:" 00 22 33 aa "; mg:"rpcd requet";) action primary match rule option Figure 2. An example Snort rule that conit of three part: action, primary match, and rule option. may be an array of computer, proceor, or proceor core). Uing concept developed for parallel computing, the array can be configured two way, function parallel or data parallel. In a function-parallel ytem the policy rule are ditributed acro the array of proceor, therefore each proceor ha a maller local policy. The data (packet payload) i then duplicated acro the array of proceor and every proceor earche the data for a maller number of ignature (defined by the local policy). Thi approach typically reduce the delay ince all the proceor are ued to proce the data. However a decribed in [2], thi form of parallelim doe not reduce the ignature matching delay. Thi i primarily due to the ue of the multi-pattern earch algorithm, uch a Wu-Manber and Aho-Coraick. The performance of thee algorithm i ub-linear with repect to the number of pattern. The earch delay for 00 pattern i not ubtantially more than the delay for 0 pattern [2]. Therefore, ditributing the rule acro each proceor only minimally reduce proceing delay. In a data parallel configuration, each proceor in the array ha the ame policy (ame ignature). The data i then ent to one proceor, uch that each proceor ha n of the original load (load balancing i the objective) []. A peed-up of.94 for 2 proceor and 3.48 for 4 proceor wa oberved uing a imple data parallel ignature matching technique [], [2]. However, the previou experiment ditributed packet in a round robin fahion, which i difficult to perform in real-time at high peed becaue the tate information required per packet flow. Although the actual peed-up may be maller than oberved experimentally, thee reult indicate that a imple data parallel approach can improve ytem peed in a calable fahion. Divided Data Parallel Signature Matching Another form of data parallelim divide the payload of each packet acro the array of proceor. Each proceor inpect a different portion, or fragment, of the ame packet, but collectively the entire packet payload i inpected [3], [4]. There are everal advantage to thi approach. Firt, many of the earching algorithm, uch a Aho-Coraick, are bounded by the amount of data to be inpected. Therefore, reducing the amount of data per proceor hould reduce the inpection time. Another important advantage of the divided data parallel method i the ability to maintain tate. Since a packet i inpected by every proceor, tate information can reide on any proceor. Uing thee technique, the performance i potentially better than the previou data parallel deign. Unfortunately, ignature can be found only if they completely exit within a packet fragment. If a ignature pan multiple fragment then it will not be found, reulting in a fale negative [3]. A een in Figure 3, thi can be avoided by duplicating data acro the proceor, called overlap, uch that a proceor can oberve the ignature [4]. Let T be the packet payload coniting of an array of m byte (or character). Furthermore, let p be the number of byte in the larget pattern, or ignature, to earch for in T. Each proceor i aigned byte to earch in T, where p ince each proceor mut be able to oberve the larget ignature. How the fragment overlap i critical to the performance and ha not been directly addreed by previou work [3], [4]. Given the larget pattern ize i p, then conecutive fragment mut overlap by, or have in common, (p ) byte. Given the value of and p, the firt fragment conit of byte T 0 through T, while the econd fragment would conit of byte T (p ) through T 2+p. The econd fragment tart at byte T (p ) ince the lat byte of the pattern may be located at + ; therefore, thi preent the need of a (p ) byte overlap with the previou fragment. Thi i depicted in Figure 3, which conit of an 8 byte packet, a fragment ize of 6 byte, and maximum pattern of 3 byte. If the ignature (p = 3) tart at byte 3 then it will be 3of7
f ragment0 f ragment2 (p ) 0 2 3 4 5 6 7 8 9 0 2 3 4 5 6 7 (p ) (p ) f ragment Figure 3. Packet payload coniting of 8 byte (m = 8), fragment ize of 6 byte ( = 6), and maximum pattern length of 3 byte (p = 3). Uing Equation, the minimum number of proceor required to proce the packet in time unit i 4. f ragment3 found in fragment 0; however, if the ignature tart at byte 4 it will be found in fragment. Therefore the (p ) overlap enure the ignature can be detected anywhere between conecutive fragment. In general, the i th fragment conit of byte T i( (p )) through T i( (p ))+. It i poible that the lat fragment i horter than the other. If it i horter than p, then thee remaining byte are added to the previou fragment. A imple analyi of the amount of time required to proce a packet payload can be done with the following aumption. Aume that every fragment contain byte and each byte require one time unit to proce. Furthermore, aume all proceor mut inpect every byte of their fragment and all proceor mut tart at the ame time. Given thee aumption the total amount of time required to proce the entire packet payload i. Given m byte in the packet payload then the minimum number of proceor required to proce the packet in unit of time i n = + m (p ) () A een in Figure 3, the lat fragment i not overlapped. The remaining m byte of the packet are divided acro other proceor uch that each proceor ha (p ) unique (non-overlapped) byte a compared to it leftmot neighboring fragment. Thi equation can be olved for to determine the appropriate fragment ize given a fixed number of proceor which typically the cae. A een in Equation, the amount of time required to proce the packet payload decreae a decreae which alo require more proceor. The hortet amount of time required to proce the packet occur when = p. Therefore the minimum number of proceor required to proce the packet payload in the hortet amount of time i n p = +(m p) (2) Note when = p every byte, except for the firt and lat, i inpected by more than proceor ince it i contained in more than one fragment. Having more proceor than defined by equation or 2 will not decreae the proceing time. A explained in the next ection, if there are more proceor than required it i poible to ue them to inpect other packet. Although overlapping doe eliminate fale negative, it alo increae the earch time. For example an 8 proceor ytem only decreaed the earch time by 60% a compared to a ingle proceor machine [4]. Overlap portion of the payload are inpected multiple time, while a ingle proceor would only inpect each byte of the payload once. A a reult the imple data parallel approach decribed in [], [2] provide better gain than the current divided data parallel method. ANEW DIVIDED DATA PARALLEL SIGNATURE MATCHING APPROACH A previouly decribed, the divided data parallel technique ha everal advantage but the earch time i increaed when overlap i ued. Thi ection introduce a new divided data parallel approach that ignificantly reduce earch time while till eliminating fale negative. A een in Figure 4, the new propoed divided data parallel ytem conit of an array of n proceor, each implementing the ame policy. A decribed in the previou ection, the payload of an arriving packet i divided acro the proceor uch that each proceor inpect only a maller portion, or fragment, of the original payload. Note, the fragment do contain overlap data to enure a ignature can be detected at any location within the packet. Fragment are queued at the proceor where they are independently inpected for ignature. Thu thi new deign allow proceor to inpect different packet fragment imultaneouly, which improve performance ince the proceor need not be ynchronized. For 4of7
packet divider packet 3 packet 2 packet 2 packet 2 packet 3 fragment 0 fragment fragment 0 packet packet packet packet fragment 0 fragment fragment 2 packet 0 packet 0 packet 0 packet 0 fragment 0 fragment fragment 2 proceor 0 proceor proceor 2 Figure 4. A new propoed divided data parallel ytem where a packet i divided into fragment then forwarded to an array of proceor. A i aociated with each fragment to indicate weather a match ha been found and allow the proceor operate independently. example, in Figure 4 proceor 0 and may inpect fragment from packet 2, while proceor 3 i inpecting a fragment from packet 3. Thi form of pipelining ignificantly improve performance. If an initial match i only conidered, a done in previou divided data parallel technique, then the performance can be further improved by allowing fragment to be ignored if an initial match ha already been found. To provide thi functionality, a i aociated with each packet and i teted by the proceor before it fragment i inpected. Initially et to fale, a match-bit for a packet i et to true if a proceor find a pattern match with an aociated fragment. If the match-bit aociated with a packet i true, then the proceor can ignore any fragment aociated with that packet. Thi alo help the proceor to operate more aynchronouly ince they can quickly ignore certain fragment. Fragment ditribution alo impact the ytem performance. Conider a imple round robin approach, where each i th portion of a packet i aigned to the i th proceor. Since packet have different length, the number of fragment may be le than the number of proceor. A imple round robin ditribution enure the firt proceor will alway have a fragment while the lat proceor may not; therefore ome form of load balancing i needed. Aigning a fragment to the next unued proceor provide load balancing, but experimental reult indicate a random ditribution provide imilar performance with minimal overhead. The propoed improvement, aynchronou earch and firt match notification, provide ignificantly fater inpection time while eliminating fale negative. EXPERIMENTS AND RESULTS The performance of the parallel content-matching approache were evaluated experimentally uing an eight core, hared memory, Linux-baed computer. The packet ignature match component of Snort 2.6.0 wa changed to perform either a data parallel or a divided data parallel earch, and wa changed to meaure the earch time. The number of proceor ued for the divided data parallel method were determined uing Equation. A done in the previou divided data parallel reearch, all experiment (data parallel and divided data parallel) only performed an initial match, which earche for the longet pattern in each rule. Each experiment ued the web and HTTP content rule upplied at the Snort web-ite, which conited of 344 rule total. The maximum pattern length, p, wa 80 byte. The packet ued for inpection conited of 3 day of actual web-traffic ent to a web-erver located at a major reearch univerity. Experiment meaured the peed-up a compared to a ingle proceor machine. In addition, the reult of the inpection were compared to a ingle proceor to enure no fale poitive or negative occurred. The performance of different match algorithm for the parallel approache wa meaured a well a the impact of different packet ize. The firt experiment compared the performance of the divided data parallel approach uing: no overlap (which reult in fale negative) [3], overlap [4], and overlap with the match-bit (the new approach propoed in thi paper). The Aho-Coraick algorithm wa ued for content matching in each experiment. Figure 5 how the peed-up a the number of proceor increaed for the three different divided data parallel approache. When neither the match-bit or overlap were ued the reult are lightly better than a normal data parallel 5of7
approach. Unfortunately a previouly dicued, thi approach reult in fale negative (number of matche were fewer than when uing a ingle proceor). A een in the figure, the performance drop when overlap i ued to eliminate fale negative. At 4 proceor the peed-up i only 3., while at 8 the peed-up i 4.9. Therefore the incluion of overlap had a negative impact on performance. The ue of overlap and the match-bit provided the bet performance. Thi divided data parallel approach reulted in a 2.66 peed-up with 2 proceor and 0.65 with 8 proceor. On average thi correpond to a.25n peed-up, where n i the number of proceor. The peed-up i greater than n becaue of pipelining, ince proceor can inpect fragment from different packet in parallel. Thi additional form of parallelim provide a ignificant increae in performance. Speed Up 0 9 8 7 6 5 4 3 2 Speed Up a Number of Proceor Increae DDP Initial Match with overlap and match bit DDP Initial Match no overlap DDP Initial Match with overlap Data Parallel Initial Match 2 4 6 8 Number of Proceor Figure 5. Speed-up of the different parallel approache, each uing the Aho-Coraick content matching algorithm. The next experiment compared the performance of different earch algorithm (Wu-Manber and Aho-Coraick) uing the propoed divided data parallel approach (overlap and ) and tandard data parallel. In general, Wu-Manber provide reaonably fat content matching with a mall memory requirement, while Aho-Coraick provide fater earch time but require more memory to tore neceary data tructure. Both are available in the current verion of Snort. A hown Figure 6, data parallel uing the Wu-Manber algorithm ha a peed gain of 4.48 time when 8 proceor are ued while DDP uing Wu-Manber ha a peed gain of 9.05 when 8 proceor are ued. Data parallel uing Aho-Coraick ha a peed gain of 3.4 with 8 proceor while DDP uing Aho-Coraick ha a peed gain of 0.65. Thee reult indicate the new propoed divided data parallel method outperform data parallel uing any ignature matching algorithm. In addition, like the traditional data parallel method, the performance of the propoed divided data parallel method i independent of the ignature matching algorithm. Thi i expected ince the data parallel paradigm divide the data tream to be proceed, not the proceing method for the data tream. Figure 6. Speed Up 0 9 8 7 6 5 4 3 2 Speed Up a Number of Proceor Increae DDP Initial Match uing Aho Coraick DDP Initial Match uing Wu Manber DP Initial Match uing Wu Manber DP Initial Match uing Aho Coraick 2 4 6 8 Number of Proceor Graph howing the peed-up of the parallel algorithm The lat experiment meaured the effect of different packet ize on the divided data parallel approach. The larget pattern length wa 20 byte and the packet payload were 360, 680, 340, and 70 byte. Thee payload amount give equal length fragment. Note that for each doubling in packet ize the number of packet ued i halved. Thi i to maintain a conitent amount of data throughout the experiment. Auming every byte of data mut be proceed, the total amount of byte inpected per group of proceor i computed by uing the following equation, where m i the packet ize in byte, n i the number of proceor, p i the ize of the larget pattern in byte, and k i the total number of packet. (m +(n ) (p )) k (3) Figure 7 i a graph of the peed-up a the packet ize change; note that thee peed-up rate mirror the value computed uing Equation 3. The larger packet perform the bet, becaue a maller portion of the packet i overlap, wherea mall packet perform wort becaue they contain a larger portion of overlap. CONCLUSIONS Thi paper introduced a new divided data parallel method that build upon the work in [3], [4] and how peed gain greater than data parallel [], [2]. The new method till divide the packet payload into fragment and ue overlapping to prevent fale negative. However, unlike other divided data parallel approache thee fragment can be proceed independently. The 6of7
Speed Up Figure 7. 8 7 6 5 4 3 2 360 680 340 70 Speed Up a Number of Proceor Increae 2 4 6 8 Number of Proceor Time to proce packet a packet ize change new method aociate a with each packet. Initially et to fale, the indicate if a match ha been found in the packet. If a proceor find a match, then the aociated i et to true. Before a proceor inpect a fragment the i checked, if it i already true then the fragment i not inpected and the proceor move to the next fragment in it queue. Thi allow proceor to operate independently ince certain fragment can be ignored. A a reult the ytem permit pipelining ince proceor may proce fragment from different packet. Experimental reult uing Snort (an open ource IDS) and actual traffic trace indicate the new divided data parallel method reult in a peed-up of approximately.25n where n i the number of proceor; wherea previou work wa only able to achieve a peed-up of 0.75n. Furthermore, the new approach i independent of the content matching algorithm. The new divided data parallel method i a calable technique that perform better than current method. ditribution will affect the peed of the ytem, reearch need to determine how detrimental thi cot i. REFERENCES [] Alfred V. Aho and Margaret J. Coraick. Efficient tring matching: An aid to bibliographic earch. Communication of the ACM, 8(6), June 975. [2] S. Antonato K.G. Anagnotaki and E. P. Markato. Generating realitic workload for network intruion detection ytem. In Proceeding ACM Workhop on Software and Performance., 2004. [3] Herbert Bo and Kaiming Huang. A network intruion detection ytem on ixp200 network proceor with upport for large rule et. http://citeeer.cail.mit.edu/703003.html. [4] Robert S. Boyer and J. Strother Moore. A fat tring earching algorithm. Commun. ACM, 20(0):762 772, 977. [5] Cert. Cert/cc tatitic 988-2006. http://www.cert.org/tat. [6] Richard Cole. Tight bound on the complexity of the boyermoore tring matching algorithm. Sympoium on Dicrete Algorithm, page 224 233, 99. [7] Mike Fik and George Varghee. Fat content-baed packet handling for intruion detection. Technical report, Univerity of California at San Diego, 200. [8] C. Kruegel, F. Valeur, G. Vigna, and R. Kemmerer. Stateful intruion detection for high-peed network, 2002. [9] Sourcefire. Sourcefire network ecurity. http://www.ourcefire.com/. [0] Sourcefire. Snort Uer Manual, December 2006. [] Patrick Wheeler and Errin W. Fulp. A taxonomy of parallel technique for intruion detection. ACMSE 2007, 2007. [2] Patrick Stuart Wheeler. Technique for improving the performance of ignature baed intruion detection ytem. Mater thei, Univerity of California Davi, 2006. [3] Sun Wu and Udi Manber. A fat algorithm for multi-pattern earching. Technical report, Univerity of Arizona, May 994. [4] Jianming Yu and Jun Li. A parallel nid pattern matching engine and it implementation on network proceor. In 2005 International Conference on Security and Management (SAM 2005), 2005. FUTURE WORK Although the new divided data parallel method ha hown great promie, there are everal area for future work. One area important for all divided parallel technique i the upport of match verification. To provide thi functionality a two tiered ytem could be ued, where the divided data parallel ytem end all initial matche to a econdary IDS for verification. Further, a dynamic metric for determining the packet plit at runtime can be developed. The packet can be plit in varying way baed upon the packet ize and the maximum pattern length. Thi reearch focued on the content matching phae, future reearch can focu on the entire ytem a a whole. The cot of fragmentation and 7of7