An Immunological Approach to Change Detection: Algorithms, Analysis and Implications

Transcription

1 An Immunological Appoach to Change Detection: Algoithms, Analysis and Implications Patik D haeselee Dept. of Compute Science Univesity of New Mexico Albuqueque, NM, patik@cs.unm.edu Stephanie Foest Dept. of Compute Science Univesity of New Mexico Albuqueque, NM, foest@cs.unm.edu Paul Helman Dept. of Compute Science Univesity of New Mexico Albuqueque, NM, helman@cs.unm.edu Abstact We pesent new esults on a distibutable changedetection method inspied by the natual immune system. A weakness in the oiginal algoithm was the exponential cost of geneating detectos. Two detecto-geneating algoithms ae intoduced which un in linea time. The algoithms ae analyzed, heuistics ae given fo setting paametes based on the analysis, and the pesence of holes in detecto space is examined. The analysis povides a basis fo assessing the pacticality of the algoithms in specific settings, and some of the implications ae discussed. 1. Intoduction It is impactical to find and patch evey secuity hole in a lage compute system. Thus, the need fo a moe compehensive appoach to secuity is inceasing. Any single potection mechanism is likely vulneable to some class of intusions. Fo example, elying on a potection mechanism that is designed fo known types of intusion implies vulneability to novel intusion methods. It is ou belief that a multi-faceted appoach is most appopiate, in which, simila to natual immune systems, both specific and non-specific potection mechanisms play a ole. This pape is concened with one aspect of ou oveall stategy: the vey geneal poblem of change detection. The method discussed is non-specific, in the sense that it is not specifically aimed towads cetain wellknown attacks, as opposed to, fo example, one using known signatues. It is also geneal in the sense that it could be used fo a wide vaiety of change-detection poblems, including those equiing some toleance of noise, o involving dynamic steams of data (such as activity pattens in unning pocesses [7]). On the othe hand, it might not always be as efficient as some of the knowledge-intensive special-pupose mechanisms fo detecting specific kinds of changes o known attacks. Its stength, howeve, is its geneality; it potentially could To appea at the 1996 IEEE Symposium on Secuity and Pivacy. be applied in many settings as a safety net to catch changes that might othewise go undetected. The change-detection method we ae studying was inspied by the geneation of T-cells in the immune system. In the thymus, T-cells with essentially andom eceptos ae geneated, but befoe they ae eleased to the est of the body, those T-cells that match self poteins ae deleted [9, 10]. Similaly, ou method distinguishes self stings (the potected data o activities) fom nonself stings (foeign o malicious data o activities) by geneating detectos fo anything that is not in the set of self stings. This pinciple of tying to match anything that has not peviously been encounteed we call "negative detection." Many methods fo change detection ely on a centalized detection potocol, i.e., each object has to be checked in its entiety, and the monito has to contain all the infomation about the oiginal objects. Ou method on the othe hand is inheently distibutable: Small sections of an object can be checked fo change independently. Diffeent independently geneated detecto sets (unning on diffeent machines fo example) can be used to achieve a highe detection ate fo a single object. The failue ate deceases exponentially with the numbe of independent detecto sets used. The individual detectos in the detecto set can be un independently as well, fo instance in a scheme with autonomous agents (such as the one pesented in [1]), whee each agent would contain one o a few detectos. We think this distibutability popety is cucial because it allows each copy of the algoithm to use a unique set of detectos. Having identical potection algoithms can be a majo vulneability in lage netwoks of computes because an intusion at one site implies that all sites ae vulneable. The negative detection method was intoduced in [6]. Ou cuent emphasis is on extending the theoetical basis of the method and addessing the impotant question of pacticality, including (i) the feasibility of geneating detectos, (ii) detemining how to choose paametes fo the algoithm, and (iii) discussing the

2 implications fo eal-wold poblems. This pape pesents algoithmic and analytical esults that enable us to addess these questions. The oiginal algoithm simply geneated andom detectos and then censoed the ones that matched self stings (accoding to a pedefined matching ule). We pesent two new algoithms, both of which ae moe efficient at geneating detectos. Section 2 gives an oveview of these algoithms, thei time and space complexity and some of the fomulas and paamete bounds deived fom them. Section 3 compaes esults obtained with the diffeent algoithms and fomulas, both on andomly geneated files and on a eal binay file. Section 4 touches on some of the pactical issues, including guidelines on applying the method, how to choose the paametes, and the fomulas involved. Section 5 pesents conclusions and outlines aeas fo futhe study. Cuently we estict ouselves to the case whee both self stings and detectos ae stings of length l ove an alphabet of size m. In this pape, the alphabet is usually binay (m=2). Self consists of an unodeed set of these stings (a multiset, because stings can occu moe than once). Figue 1 shows the elevant sets of stings and how they elate to each othe. The goal of ou method is to find a detecto set R that matches as many of the nonself stings in N as possible, without matching any of the self stings in S. We define the failue pobability as the pobability that a andom nonself sting will not be matched by any of the detectos in R. We futhe U detected undetected holes self stings D S N' N Matches Matches MatchedBy C R C Ud Figue 1: Sets of stings and thei elations. Sting space U and detecto space U d ae dawn sepaately fo claity, even though U=U d fo this pape. P is MatchedBy Q iff Q contains all the detectos matching any sting in P. Q Matches P iff P contains all the stings matched by any detecto in Q. S: self stings; N: nonself stings; C: candidate detectos; R: detecto epetoie chosen fom C; N : detectable nonself stings; D: detected nonself stings. Not indicated ae the set of holes H=N-N, and the set of undetected nonself stings F=N -D. define the matching pobability P m as the pobability that a andomly chosen sting and detecto match accoding to the specified matching ule. To simplify the notation, we will wite N X fo the size (i.e. cadinality) X of a set of stings X. In paticula, N S is the size of the self set and is the detecto set size. 2. Detecto geneating algoithms This section descibes thee diffeent algoithms fo geneating detecto sets: the oiginal exhaustive geneating algoithm and two new algoithms, based on dynamic pogamming, which un in linea time with espect to the size of the input. See [6] fo an exposition of the exhaustive detecto geneating algoithm (2.1). Fo moe details on the linea time algoithm (2.2), see [8] and [3]. This last epot also coves the geedy algoithm (2.3) and the algoithm fo counting the holes (2.4), including some examples and a deivation of the time and space complexities Exhaustive detecto geneating algoithm This algoithm mios most closely the geneation of T-cells in the immune system. Candidate detectos ae dawn at andom fom U d and checked against all stings in S. If they fail to match any of the self stings, they ae kept as valid detectos. This pocess of andom geneation and checking against S is epeated until the equied numbe of detectos is geneated. This algoithm equies geneating a numbe of candidate detectos ( 0 : initial detecto epetoie size, befoe negative selection). that is exponential in the size of self (fo a fixed matching pobability P m ) [2]: ln( Pf ) NR 0 = N Pm 1 P S. ( m) Fo independent detectos, we can appoximate the failue pobability achieved by detectos by: ( ) Pf 1 Pm. (1) Fo P m sufficiently small and sufficiently lage, this gives: ln ( ) P m. (2) The assumption that the detectos ae independent is not entiely valid. As N S o P m inceases, the candidate detecto set (C in Figue 1) will shink., so the detectos chosen become less independent. Ovelap among the detectos deceases the amount of sting space coveed, esulting in a highe failue pobability than (1) would indicate. The time complexity of this algoithm is popotional to 0, the numbe of candidate detectos that need to be examined, and N S (because each sting may have to be compaed against all self stings). Space complexity is detemined by N S :

3 ln( Pf ) time : O Pm ( 1 Pm) space : O( l N S ). NS NS, 2.2. Linea time detecto geneating algoithm The geneate-and-test algoithm descibed above is inefficient because most of the candidate detecto stings ae ejected. Howeve it does wok fo abitay matching ules. Fo specific matching ules we might be able to find a moe efficient detecto geneating algoithm. Hee, we descibe a two-phase algoithm fo the -contiguous-bits matching ule (two l-bit stings match each othe if they ae identical in at least contiguous positions) that uns in linea time with espect to the size of the input (fo fixed matching paametes l and ). In Phase I, we solve a counting ecuence fo the numbe of stings unmatched by stings in S (candidate detectos, set C in Figue 1). In Phase II, we use the enumeation imposed by the counting ecuence to pick detectos andomly fom this set of candidate detectos. We will adopt the following notation: s denotes a bit sting. ŝ denotes s stipped of its fist (leftmost) bit. s b, whee b {0,1}, denotes s appended with b. In paticula, ŝ b is s stipped of its fist bit and appended with b. A template of ode is a size l sting consisting of l- blank symbols (epesented by asteices hee) and fully specified contiguous bits. In paticula, a template t i,s, is that template in which the specified bits stat at position i and ae given by the -bit sting s. Fo example, with l=6, =3, s=010: t 2,s =*010**. A sting (o template) matches a sting if they ae identical (no blanks) in at least contiguous positions. A ight (left) completion of a template t is that template with all the blanks to the ight (left) eplaced by bits. Fo example: *01011 is a valid ight completion fo *010**. a, b ( ] stands fo the intege inteval (a+1)...b. Phase I: Solving the counting ecuence Fo bit stings s of length and fo 1 i (l + 1), let C i [s] = the numbe of ight completions of t i,s unmatched by any sting in S. Each enty C i [s] in the aay coesponds to an ode template t i,s, i In essence, these templates enumeate all the possible ways two stings can match each othe ove contiguous bits. In paticula, fo i=l-+1, t i,s consists of l- blanks, followed by consecutive bits. Thee ae no blanks to the ight, so the only ight completion of such a template is the template itself. Theefoe C l-+1 [s] will be zeo if the template t l-+1,s is matched in S, one othewise: C l +1 [s] = 0,if t l +1,s is matched in S 1, othewise Fo 1 i < (l + 1), we can calculate the numbe of unmatched ight completions based on the numbe of unmatched ight completions at i+1. If t i,s is diectly matched in S, C i [s] is zeo. Othewise, we can subdivide the ight completions of t i,s into those with a 0 bit diectly following the significant bits of s, and those with a 1 bit thee. These ae exactly the numbe of ight completions fo ŝ 0 and ŝ 1 espectively: C i [s] = 0, if t i,s is matched in S C i+1 [ŝ.0]+ C i+1 [ŝ.1], othewise Fo example, with l=6, =3, suppose s 1 = is one of the stings in S, then the following templates ae diectly matched by this sting: 110***, *101**, **010* and ***100. Theefoe, C 1 [110] = C 2 [101] = C 3 [010] = C 4 [100] = 0. Suppose sting s 2 = is also in S. The template **110* is not diectly matched by s 1 no s 2. Howeve, because both ***100 and ***101 ae matched in S (by s 1 and s 2 espectively), **110* will not have any unmatched ight completions eithe: C 3 [110] = C 4 [100] + C 4 [101] = 0. This can easily be veified: **110* has two ight completions: **1100 and **1101. The fist one is matched by s 1, the second by s 2. Phase II: Geneating stings unmatched by S Note that as the ecuence pogesses fom column C l-+1 [.] of the C aay to column C 2 [.], the emaining blanks in the ight completions ae gadually filled up, and fo C 1 [.], the ight completions ae fully specified l- bit stings. Theefoe, C 1 [s] denotes the numbe of unmatched l-bit binay stings stating with the -bit binay sting s. The total numbe of stings unmatched by S is T = C 1 [s]. s C 1 [.] can be viewed as a patitioning of the space of unmatched stings into patitions of size C 1 [s] fo each initial -bit sting s. Of all the unmatched stings stating with s, we know that C 2 [ŝ.0] have a 0 bit next, while C 2 [ŝ.1] have a 1 bit next, so C 2 [.] can be viewed as a futhe patitioning of this space. Similaly fo C 3 [.] to C l-+1 [.]. Afte patitioning accoding to C l-+1 [.], each patition consists of one single l-bit sting. We can theefoe impose an explicit numbeing fom 1 to T on the unmatched stings, based on the natual ode of bit stings. Given this explicit numbeing, we can geneate andom integes in {1..T} and etieve the coesponding stings. Fo a numbe k {1.. T}, we find the k th unmatched sting u k in the following way: Fist, do a binay seach on C 1 [.] to find s 1 such that P 1 = C 1 [s] < k Q 1 = C 1 [s]. s<s 1 All unmatched stings in (P 1,Q 1 ] have s 1 as thei leading bits. The sting u k we ae looking fo is in the s s 1

4 patition of unmatched stings numbeed (P 1 +1)...Q 1, theefoe the fist bits of u k ae given by s 1. Now we can detemine fo each i = 2...(l + 1) the bit at position (+i-1) of u k, by checking in which patition k falls. Fo example, to detemine the bit at position +1, we can patition the inteval futhe into (P 1, P 1 + C 2 [ŝ 1.0]] and (P 1 + C 2 [ŝ 1.0],Q 1 ], coesponding to the stings with eithe a 0 o 1 bit coming next. We add a bit b 1 =0 if k is in the fist inteval, b 1 =1 bit if k is in the second one. We then set P 2 and Q 2 using: P i = P, if b = 0 i-1 i 1 P i 1 + C i [ŝ i 1.0], if b i 1 = 1 and Q i = P + C [ŝ.0], if b = 0 i 1 i i 1 i 1 Q i-1, if b i 1 = 1 Let s 2 = ŝ 1 b 1. k is now in the inteval (P 2,Q 2 ], which we can split up into intevals (P 2, P 2 + C 3 [ŝ 2.0]] and (P 2 + C 3 [ŝ 2.0],Q 2 ]. Bit b 2, will be detemined by whethe k falls in the fist o second inteval. Futhe bit positions of u k ae detemined similaly. The pinciple data stuctue used in this algoithm consists of the lage (l ) 2 C aay epesenting all the possible ways two stings can match ove contiguous bits. This has an impact on the time and space complexity of the algoithm: time : O( ( l ) NS )+ O( ( l ) 2 )+ O( l NR), (( ) 2 ) space : O l 2. The above algoithm uns in time linea in the size of the self set and detecto set (fo given paametes l and ). This is in contast to the exhaustive detecto geneating algoithm, which an in time exponential to the size of the self set, but equied essentially only constant space. The linea time algoithm still equies time and space exponential in the length of the matching egion, which may pesent a poblem if we need to choose long stings (l) and matching egions () Geedy detecto geneating algoithm We can achieve a bette coveage of the sting space with the same numbe of detectos (o a smalle detecto set fo the same amount of coveage) by not selecting the detectos at andom, but placing them as fa apat as possible. The geedy algoithm we developed ties to do exactly that. At each step it picks one of the detectos that will match as many as possible of the as yet unmatched nonself stings. To constuct the C aay in Phase I of the pevious algoithm we chose to examine the stings fom ight to left (fom C l-+1 [.] to C 1 [.]). We can also go though the stings fom left to ight, constucting a second aay C stating at C 1[.] and calculating the following levels using a simila ecuence elation as fo C. Because C i [s] epesents the numbe of nonmatching ight completions fo template t i,s and C i[s] the numbe of nonmatching left completions, D i [s] = C i [s] C i [s] epesents the numbe of unmatched fully specified bit stings coesponding to this template. If a given template has a zeo enty in D, we know that all stings containing that template will match some sting in S. Convesely, if we estict ouselves to picking templates with nonzeo enties in D when constucting bit stings, we know those stings will not be matched by any sting in S. Phase I: Geneate D aays: D S and D R The algoithm uses two diffeent D aays, called D S and D R, the fist one based on the self set S and the second one based on the cuent state of the detecto set R. The D S aay tells us which templates we ae allowed to choose fom when constucting detectos. We will call the templates that have nonzeo enties in this D S aay valid detecto templates. Phase II: Geneating stings unmatched by S The second aay D R, based on the cuent detecto set R, will indicate how many stings fo each template ae not yet matched by the peviously geneated detectos. Fo each new detecto to be geneated, we then ty to select the templates matching the most unmatched stings. We have to update this aay D R each time a new detecto is geneated, so it will geneally be cheape to just keep the C R and C R aays aound and update these incementally. Because we begin with R being empty, we can initialize C R and C R with thei ( l + 1 i) ( i ) maximum values: C []= s 2 and C [] s = 2 1 Ri, ( l ) (coesponding to DRi, []= s 2 ). Fo each new detecto, we seach though D R fo the valid detecto template with the lagest enty. If thee is a tie between templates, we choose one at andom. Stating fom this template, we then tavese the D R aay to the left and to the ight, each time choosing to add a 0 o 1 bit to the stating template depending on which epesents the template with the highest numbe of stings not yet matched by R (while still esticting ouselves to valid detecto templates). Next, we have to update the C R and C R aays to eflect that a new detecto has been added to R. We can do this incementally, by setting those enties in C R and C R to zeo that diectly match the detecto, and ecalculating the appopiate enties. We epeat this pocess of picking a detecto and updating C R, C R and D R until all valid detecto templates have zeo enties in D R. At that point, fo any template that is not in S thee ae no moe stings that have not yet been matched by a detecto, i.e., we have coveed all stings that can possibly be coveed by detectos. We call this a complete detecto epetoie. This algoithm also has the attactive popety that we can keep a unning count of the numbe of nonself stings that ae still unmatched by any detecto. Fo a Ri,

5 given acceptable failue pobability we can simply keep geneating detectos until we each the coesponding numbe of unmatched nonself stings (o until we un out of candidate detectos, which would mean that thee ae too many holes to be able to each the desied ). At best, we can spead the detectos apat such that no two detectos match the same nonself sting. This gives us an absolute lowe bound on the numbe of detectos needed [12]: ( ) N 1 P P. (3) R f m Looking at the stuctue of the template aay we can get anothe estimate fo. Each detecto geneated matches one of the 2 templates in each of the (l-) columns of the template aay, and sets the coesponding count thee to zeo. We can zeo out all the enties in a column with at most 2 detectos: 2. This fomula does not take into account the enties in the template aay aleady zeoed out by matching self stings. If we assume the self stings ae independent, each template has a chance (1 2 ) N S of not matching any of the N S self stings. The estimate fo then becomes: ( ) N S (4) Because we need to update the template aay each time a new detecto is geneated, the time complexity is quite a bit highe than fo the pevious algoithm, although the space complexity is of the same ode: time : O ( l ) 2 N, ( R) (( ) 2 2 ). space : O l 2.4. Counting the holes Even though the above algoithm is capable of constucting a complete detecto epetoie, this does not necessaily mean it can constuct a detecto set capable of ecognizing all non-self stings, i.e., all stings not in S. Depending on the matching ule used and the stings in S, thee may be some nonself stings, called holes, fo which it is impossible to geneate valid detectos. Fo example, if S contains two stings s 1 and s 2 that match each othe ove (-1) contiguous bits, they can induce two othe stings h 1 and h 2 that cannot be detected because any candidate detecto would also match eithe s 1 o s 2, as shown below: s1 : a1... akbk bk+ 1 ck+... cl s2 : a1... ak bk bk+ 1 ck +... cl h1 : a1... akbk bk+ 1 ck +... cl h2 : a1... ak bk bk+ 1 ck+... cl whee a i, a i, b i, c i, and c i ae single bits. A simila agument shows that we also can have holes using a Hamming distance matching ule (whee two stings match if thei Hamming distance is less than o equal to a fixed adius ). In fact, almost all pactical matching ules with a fixed matching pobability can be expected to exhibit holes [4, 5]. Howeve, we can eliminate holes altogethe by choosing a matching ule with a vaiable matching adius, such that potential holes ae filled by detectos with high specificity. Because holes will neve be detected by any detecto, they imposes a lowe bound on the failue pobability we can achieve, whethe we use a single set of detectos o seveal independent detecto sets geneated fo the same matching ule. It is theefoe advisable in a distibuted setting to choose a diffeent matching ule (o simply diffeent paametes) fo each machine, so each will have a diffeent set of holes which ae likely coveed by some othe machines. On the othe hand, we can take advantage of the position of these holes to povide a cetain level of noise toleance in ou detection method: because holes ae close to self stings, we might not eally cae if they go undetected. Many othe change-detection algoithms (such as checksums and message digests) ae sensitive to any change in the data and theefoe not applicable when a cetain amount of noise toleance is equied. Running the geedy algoithm until all valid detectos have been used up would tell us exactly how many nonself stings cannot be detected. In [3] we developed a moe efficient algoithm fo counting the exact numbe of holes, simila to the way the numbe of detecto stings not matched by S ae counted in the linea time algoithm. Its space and time complexities ae simila: time : O( ( l ) NS )+ O ( l ) 2, (( ) 2 ) space : O l 2. ( ) The easonably shot unning time makes this a useful tool in discoveing appopiate settings fo the l and paametes of the matching ule. At the vey minimum, we want the numbe of holes N H to be small enough to allow the desied failue pobability : N H 2 l. If we stick close to this uppe bound on the allowed numbe of holes, almost all valid detectos will be needed to cove the vey last nooks and cannies of the detectable nonself sting space. If the numbe of holes is much smalle than this, we may need substantially fewe detectos fo the same. The smalle the faction of non-zeo enties thee ae in the template aay, the moe holes thee will be (because holes consist solely of templates that have zeo enties in the template aay). The template aay becomes spase if N S >> 2, because each of the N S self stings can match one of the 2 templates. In ode to get only a small numbe of holes, we may want to use N S 2. (5)

6 L S (a) N S (b) l (c) (d) 500B KB KB KB P m (e) indep. =0.1 (f) geedy complete (g) estimate complete (h) geedy =0.1 (i) optimal =0.1 (j) entopy =0.1 (k) numbe of holes (l) e e e+08 Table 1: Repetoie sizes and numbe of holes fo diffeent configuations. (a): file size in bytes, (b), (c), (d): paametes chosen fo the matching ule. (e): coesponding matching pobability P m fo the - contiguous-bit matching ule. (f): calculated accoding to fomula (2), fo independent detectos. (g): size of complete detecto epetoie geneated with the geedy algoithm. (h): estimate fo (g) based on fomula (4). (i): geneated by geedy algoithm until 0.1 ( means that 0.1 could not be achieved). (j): lowe bound on (i) based on fomula (3). (k): entopy-based lowe bound on (h). 1 (l): numbe of holes pesent. 3. Results and analysis In this section we use the algoithms and analysis fom section 2 to exploe which paamete settings ae pactical. Subsection 3.1 looks at esults obtained fo elatively small andomly geneated self sets, and daws some conclusions fom compaisons between diffeent algoithms and fomulas. Subsection 3.2 looks at a much lage eal-wold example and examines how the failue pobability scales with detecto set size and matching length Repetoie sizes using diffeent algoithms Table 1 shows detecto epetoie sizes obtained with the diffeent algoithms using andomly geneated self files and a numbe of diffeent paamete sets ( N S, l and ), as well as some uppe and lowe bounds pedicted by the fomulas pesented in the pevious sections. We have abitaily chosen =0.1 as an acceptable failue ate. Because the algoithm might be able to cove moe of the total nonself sting space with a smalle set of detectos if thee is a stuctue to the self stings, independent self stings will tend to be a wost case situation fo estimating (ignoing the effect of the holes on the achievable ). When we compae the esults in columns (f) and (i), we see that the geedy algoithm geneates a detecto set that is fom 8% to 41% smalle than the size pedicted fo the independently chosen detectos. Also note that fomula (2) indicates that the desied should be eachable with a detecto set of a cetain size, although thee might be so many holes in the sting space that this is uneachable, as indicated by the enties maked in (i). Fo the linea time and the exhaustive algoithm, thee is no guaantee that a detecto set of the size indicated by (2) will achieve the specified. This is not a poblem fo the geedy algoithm, because we can continue geneating detectos until exceeds the specification. With a geedy selection of detectos, the last detectos geneated will only match a small numbe of as yet unmatched nonself stings, and will theefoe not have a significant effect on. This means that when going 1 Refeences [4, 5] show that fo a set of independent self stings, the following is a lowe bound on the numbe of detectos needed: N S log 2 ( 1 ) l log 2 (m) (6). This bound is based on the amount of infomation that needs to be stoed in R about S.

7 fom = 0.1 to = min NR ( ), quite a lage numbe of exta detectos may have to be added to match all of the tiny unmatched egions of nonself sting space. This explains the lage diffeence in detecto set size between, fo example, 90% detection ate and the maximum detection ate (complete epetoie). Both (3) and (6) (columns (j) and (k) in Table 1) ae indeed effective lowe bounds on the size of the detecto set needed fo a failue pobability of =0.1. The entopy-based lowe bound is less stict, patially because it does not take the popeties of the matching ule into account. The lowe bound in column (j) is based on optimal spacing of the detectos, which is pecisely what the geedy algoithm ties to achieve, so we could view column (j) as the optimal detecto set size fo the geedy algoithm. Inteestingly, when the geedy algoithm aims fo an optimal detecto set size that is smalle o almost equal to the size indicated by the entopy-based lowe bound (i.e., (j) (k)), it is unable to do so (enty in (i) is ) because the numbe of holes in the sting space (column (l)) is too lage with espect to the desied. This suggests that ( ) ( 1 ) P m N log 1 P S 2 f (7) l log 2 (m) is an inteesting lowe bound on the value fo P m (and theefoe ) given N S, l and. Within one set of ows with the same N S and l, we see that the epetoie size deceases with matching length. Howeve, the numbe of holes in sting space seems to incease in an exponential fashion with deceasing up to the point whee we can no longe find an adequate detecto set fo the acceptable failue pobability. Since P m inceases exponentially with (l-), a smalle matching length means that each detecto matches moe stings, so fewe detectos ae needed. Howeve, each self sting will also match moe detectos, so the space of candidate detectos becomes smalle and the numbe of holes due to inteaction between self stings gets lage. Fo the smallest fo which we can constuct an adequate detecto set R, a lage numbe of the nonself stings not matched by R ae holes. As mentioned befoe, using diffeent matching lengths at the same time would allow us to combine matching all nonself stings and coveing most of the nonself sting space by a small numbe of detectos. By looking at ows with the same L S, we can get an idea of the effect of choosing a shote o longe sting length to split the data up in self stings. Fo instance, the ows with N S =250 and l=32 ae geneated fo the same data as the ows with N S =500 and l=16. Similaly fo N S =500, l=32 and N S =1000, l=16. We see that with a longe sting length l, a smalle numbe of detecto is needed to achieve = 0.1. If we look at how much space is taken up by the detecto set ( l ), the lage sting length still comes out ahead. Note that with the lage sting length the numbe of holes in the sting space is substantially lage (fo the same values of ). This is due to the fact that the sting space itself is much lage as well. Howeve, the faction of sting space occupied by holes is smalle fo lage l because the self stings ae spaced fathe apat and theefoe inteact with each othe less. This means that fo a lage sting length l, we can choose smalle and still have an acceptable detecto set. This is exactly opposite to what we would expect if we wanted to keep the matching pobability P m constant vesus fo a eal data file Figue 2 shows how vaies with fo a binay file (GNU emacs v SGI binay, 3.2MB). The data (a) (b) Figue 2: vesus fo a binay file. (a): linea scale fo ; (b): log scale fo.

8 fo each value of was deived fom a single lage detecto set geneated with the linea time algoithm (one million detectos fo =16 and =18; 5 million detectos fo =19) nonself stings wee checked against each of these detecto sets, and fo each nonself sting we ecoded the fist detecto to match the sting. We can then deive the pobability of success (1- ) as the cumulative histogam ove these values. Note that this means that the points in each cuve ae not completely independent. Howeve, this appoach gives us a easonable appoximation fo a much smalle computational effot. The figue shows a shap dop in the failue pobability fo the fist couple of hunded thousand detectos. This is due to the pobabilistic natue of coveage of the sting space by detectos. We would expect this decline to be even moe ponounced if we wee to geneate the detectos using the geedy algoithm, because then the detectos chosen fist ae those which cove as many nonself stings as possible. The decline in is shape fo smalle values of, and theefoe fo lage matching pobabilities, because each detecto matches a lage faction of sting space, so most of the space is coveed by a small numbe of detectos. Howeve, as the detecto set size inceases, levels off at a highe level fo smalle because thee ae moe holes. Note also that fo each detecto set size thee is an optimal value fo. In geneal, if we want to have a smalle detecto set, we will have to use a smalle value fo (such that each detecto matches moe nonself stings). Similaly, fo each value of thee is an optimal value fo. As the acceptable failue ate deceases, we will have to go to lage values of (moe specific detectos) and lage detecto sets. Finally, if we want to exploit the fact that the failue pobability deceases exponentially with an inceasing numbe of machines, each of which is unning its own detecto set, we may aleady be satisfied with 0.5. Figue 2 shows that this can be achieved with a epetoie as small as 30,000 to 120,000 detectos. This is quite an achievement given that the oiginal self file contains about one million stings. 4. Summay of fomulas, pactical issues and ules of thumb This section gives a summay of the appopiate fomulas fo the -contiguous-bits matching ule, and gives step-by step instuctions on how to choose the matching ule paametes fo a eal data steam Choosing the alphabet size The lage the alphabet size used, the hade it becomes to make an optimal choice fo the matching length. This is due to the fact that fo the matching ules consideed hee, the matching pobability P m m [11, 13]. Assuming that P m has to stay within cetain bounds fo the detection algoithm to pefom efficiently, the ange of acceptable values fo becomes vey naow with inceasing alphabet size. Fo some applications, howeve, a non-binay alphabet may be moe appopiate. Fo example, [6] descibes an expeiment in which a C pogam compiled fo a RISC achitectue was checked fo changes. Each opcode was mapped into one of 104 symbols (m=104). An impotant aea of futue investigation is to study the pefomance of diffeent size alphabets, especially in those cases whee a non-binay alphabet is most natual Choosing the sting length Fist, we detemine a lowe bound fo l by equiing that the self stings should occupy only a faction of the total sting space: N S L S l 2 l. Table 2 illustates some values fo L S and the coesponding lowe bounds fo l. L S 384 2K 10K 49K 229K 1M 4.7M 21M 92M 403M l Table 2: L S vesus lowe bound fo l Note that this is not an exact lowe bound fo l, because splitting up the data into l-bit stings may cause many duplicate stings, so N S L S l. If we want to know the exact lowe bound fo l (o if the data to be checked fo changes is not a fixed size file but athe a continuous steam of data) we can explicitly count the numbe of unique l-bit stings fo diffeent values of l. The second step in selecting sting length is to detemine whethe thee exists a natual sting length imposed by the data. Fo instance, if the data to be potected is a database consisting of a seies of 4-byte ecods, then a multiple of fou bytes would be an obvious sting length to ty because it peseves the stuctue of the data. If the data does not exhibit any natual sting length, we might still be able to find by inspection cetain ecuing featues that we want to be able to captue. The sting length will have to be longe than these featues fo them to be peseved in the self stings. Finally, thee will be an uppe bound to the sting length, imposed by the geneating algoithms. Inceasing l usually also means has to be inceased (cetainly if we want the matching pobability to emain constant). Howeve, fo lage l and the algoithms descibed in this epot become computationally vey expensive Choosing the matching length To keep the numbe of holes low, we want to pick such that N S 2 (5). Again, we might want to eplace

9 N S in this fomula by the numbe of unique l-bit stings that appea in the data. As illustated by Table 1, fomula (7) gives an appoximate uppe bound on P m (and thus a lowe bound on, fo a given sting length l) in ode to be able to constuct a detecto epetoie fo the acceptable failue pobability : ( ) ( 1 ) P m N log 1 P S 2 f. (7) l log 2 (m) Finally, afte a minimum value fo has been chosen, we may want to count the exact numbe of holes pesent in the sting space. If this exceeds the acceptable numbe of holes of N H 2 l, we choose a smalle matching pobability (i.e. lage ) and ty again. In geneal, because we don t have an exact pocedue to detemine the optimal values fo l and, a numbe of valid combinations may have to be tied and weighed against each othe in tems of ease of constuction of the detecto set, size of the esulting detecto set, attainable failue pobability, etc Choosing the numbe of detectos Above we have seen a numbe of lowe bounds and estimates fo unde diffeent assumptions. We will go though these one by one, fom the least tight lowe bounds to what we expect to be the best estimates. If the paametes of the self set and matching ule ae such that geneating the detectos using the geedy algoithm is too expensive, we can use the linea time algoithm to select a set of detectos independently chosen fom the candidate detectos. If we assume the detectos chosen ae also independent of each othe with espect to the total sting space, we get the lowe bound fom (2): ln( ) P m. (2) This last independence assumption usually doesn t hold: because of the limited numbe of candidate detectos, thee will be moe ovelap between detectos than we would expect fom stings chosen independently fom the entie sting space. If this is the case, the numbe of detectos needed to achieve can be quite a bit lage than the value indicated by (2). We may have to estimate the actual a posteioi ove a sample of nonself stings. If we use an algoithm that attempts to spead out the detectos, such as the geedy algoithm, the minimum numbe of detectos needed to achieve an acceptable is given by (3): ( 1 ) P m. (3) This is an absolute lowe bound, in the sense that it is impossible to cove that much of sting space with fewe detectos. Howeve, depending on the stuctue pesent in the self data, it may be had to pick detectos that ae spead apat optimally. Note that if we have chosen accoding to fomula (7), using this lowe bound fo will automatically imply that we also satisfy the entopy lowe bound (6). Using (3) to calculate the numbe of detectos needed fo the geedy algoithm is only inteesting in tems of getting a ballpak figue fo to evaluate whethe the paametes l and have been chosen efficiently. If we ae satisfied with the estimate we simply un the geedy algoithm until it eaches exactly the desied. If we ae inteested in achieving the minimum possible failue pobability, we can constuct a complete detecto epetoie by unning the geedy algoithm till exhaustion. (4) povides a faily close estimate fo the size of the complete detecto set fo independent self stings: ( ) N S Detection scheme. (4) One aea that we have not yet examined closely is how best to implement the actual detection scheme once an appopiate detecto set has been geneated. Hee ae some possible examples: Maximum secuity: evey sting needs to be checked against the entie detecto set. Othe, moe conventional, change-detection algoithms may be moe appopiate in this case. Intemittent checking: evey so often a small numbe of stings can be checked against a small numbe of detectos chosen at andom fom the detecto set. This elies on the pobabilistic natue of this change-detection method. It assumes that if a change is undetected, it will be detected duing some othe check, o on anothe machine. This might not be acceptable if a single occuence of the change can be fatal. Weighted detection: detectos can be chosen moe fequently depending on pevious pefomance, based on known expected changes (known failue modes, vius signatues, etc.), o in ode to get a homogeneous coveage of nonself sting space (some aeas of nonself sting space may be coveed by moe detectos than othes). Distibuted detection: the detecto set is split up ove a numbe of autonomous agents (see [1]) each doing checks in paallel. This is also the scheme used in the eal immune system, whee each T-cell coesponds to a single detecto. Distibuted independent detection: each agent has a detecto set geneated independently fom all othe agents. This is simila to a population of individuals with immune systems. The advantage is that holes in one individual s immune system can be coveed by anothe s, so an infection cannot spead though the entie population. A simila setup can be used to potect a whole netwok of computes.

10 5. Conclusion The linea time algoithm has made it pactical to constuct efficient detecto sets fo lage data sets. The space equiements fo this constuction algoithm ae substantial. Howeve, this space is only needed once, to calculate the detecto set, and can be discaded aftewads. The geedy algoithm allows us to sacifice some of the speed of detecto geneation in exchange fo a moe compact detecto set and a failue pobability guaanteed to be below the acceptable. We have also made significant impovements towads quantifying the ange of acceptable values fo the diffeent paametes associated with the detection method, to the point that we can give guidelines fo setting up a detection system like this fo any given data set. The distibuted natue of this algoithm is pomising fo netwoked and distibuted computing envionments. As a vey geneal-pupose change-detection method it can supplement moe specific, and theefoe moe bittle, potection mechanisms. We imagine a layeed compute immune system, with specific potection mechanisms against well known o peviously encounteed intusions, and non-specific potection mechanisms like the one pesented hee to intecept those intusions that evade the specific mechanisms. Thee ae a numbe of emaining issues to be examined. Hee ae the most impotant ones: It might be possible to deive moe exact fomulas fo non-andom data by looking at some measues of the self stings (entopy, aveage numbe of nonzeo enties in the template aays, numbe of holes etc.) It is possible to constuct a linea time algoithm fo the Hamming distance matching ule. This matching ule might give impoved pefomance because it does not limit the length of the matching templates and should theefoe be able to captue lage stuctues in the self stings. Fom a secuity point of view, it might be useful to have a matching ule fo which it is povably had to constuct (and thus to foge) a detecto set. Can we come up with matching ules based on some NPcomplete poblems fo instance? The effect of using negative detection as opposed to positive detection is not vey well undestood. Moe eseach would be needed to claify this issue. 6. Acknowledgments: The authos thank Dipanka Dasgupta, Deek Smith, Ron Hightowe and Andew Kosoesow fo thei useful suggestions and citical comments. The idea of a compute immune system gew out of a collaboation with D. Alan Peelson though the Santa Fe Institute. This wok is suppoted by gants fom the National Science Foundation (gant IRI ), Office of Naval Reseach (gant N ), Inteval Reseach Copoation, Geneal Electic Copoate Reseach and Development, and Digital Equipment Copoation. Refeences: [1] M. Cosbie and G. Spaffod, "Defending a Compute System using Autonomous Agents", in Poceedings of the 18th National Infomation Systems Secuity Confeence, [2] R. J. De Boe and A. S. Peelson, How divese should the immune system be? in Poceedings of the Royal Society London B, v.252, London, [3] P. D haeselee, Futhe efficient algoithms fo geneating antibody stings, Technical Repot CS95-3, The Univesity of New Mexico, Albuqueque, NM, [4] P. D haeselee, A change-detection algoithm inspied by the immune system: Theoy, algoithms and techniques, Technical Repot CS95-6, The Univesity of New Mexico, Albuqueque, NM, [5] P. D haeselee, An Immunological Appoach to Change Detection: Theoetical Results, accepted to the 9th IEEE Compute Secuity Foundations Wokshop, [6] S. Foest, A. S. Peelson, L. Allen and R. Cheukui, Self-nonself discimination in a compute, in Poceedings of the 1994 IEEE Symposium on Reseach in Secuity and Pivacy, Los Alamitos, CA: IEEE Compute Society Pess, [7] S. Foest, S. A. Hofmey, A. B. Somayaji and T. A. Longstaff, A sense of self fo UNIX pocesses, submitted to the 1996 IEEE Symposium on Secuity and Pivacy, [8] P. Helman and S. Foest, An efficient algoithm fo geneating andom antibody stings, Technical Repot CS-94-07, The Univesity of New Mexico, Albuqueque, NM, [9] J.W. Kapple, N. Roehm, P. Maack, "T cell toleance by clonal elimination in the thymus." in Cell, 49: , [10] W. E. Paul, Ed., Fundamental Immunology, Raven Pess Ltd. New Yok, 88-90, [11] J. K. Pecus, O. E. Pecus and A. S. Peelson, Pobability of Self-Nonself discimination in Theoetical and Expeimental Insights into Immunology, [12] A. S. Peelson and G. F. Oste, Theoetical Studies of Clonal Selection: Minimal Antibody Repetoie Size and Reliability of self-nonself Discimination in Jounal of Theoetical Biology, [13] J. V. Uspensky, Intoduction to Mathematical Pobability, McGaw-Hill, NY, 1937.