Generating In-Line Monitors For Rabin Automata

Transcription

1 Generting In-Line Monitors For Rin Automt Hugues Chot, Rphel Khoury, nd Ndi Twi Lvl University, Deprtment of Computer Science nd Softwre Engineering, Pvillon Adrien-Pouliot, 1065, venue de l Medecine Queec City, Cnd {hugues.chot.1,rphel.khoury.1,ndi.twi}@ulvl.c Astrct. A promising solution to the prolem of securing potentilly mlicious moile code lies in the use of progrm monitors. Such monitors cn e in-lined into n untrusted progrm to produce n instrumented code tht provly stisfies the security policy. It is well known tht enforcement mechnisms sed on Schneider s security utomt only enforce sfety properties [1]. Yet susequent studies show tht wider rnge of properties thn those implemented so fr could e enforced using monitors. In this pper, we present n pproch to produce model of n instrumented progrm from security requirement represented y Rin utomton nd model of the progrm. Bsed on n priori knowledge of the progrm ehvior, this pproch llows to enforce, in some cses, more thn sfety properties. We provide theorem stting tht trunction enforcement mechnism considering only the set of possile executions of specific progrm is strictly more powerful thn mechnism considering ll the executions over n lphet of ctions. Key words: Computer Security, Dynmic Anlysis,Monitoring Softwre Sfety 1 Introduction Execution monitoring is n pproch to code sfety tht seeks to llow n untrusted code to run sfely y oserving its execution nd recting if need e to prevent potentil violtion of user-supplied security policy. This method hs mny promising pplictions, prticulrly with respect to the sfe use of moile code. Acdemic reserch on monitoring hs generlly focused on two questions. The first reltes to the set of policies tht cn e enforced y monitors nd the conditions under which this set could e extended. The second question dels with the wy to in-line monitor into n untrusted or potentilly mlicious progrm in order to produce new instrumented progrm tht provly respects the desired security policy. While studies on security policy enforcement mechnisms show tht n priori knowledge of the trget progrm s ehvior would increse the power of these mechnisms [2, 3], no further investigtions hve een pursued in order to tke full dvntge of this ide in the context of runtime monitoring. In this pper, we present n pproch to generte sfe instrumented progrm, from security policy nd n untrusted progrm in which the monitor drws on n priori knowledge of the progrm s possile ehvior. The policy is stted s deterministic Rin utomton, model which cn recognize the sme clss of lnguges s non deterministic Büchi utomt [4].

2 2 H. Chot, R. Khoury nd N. Twi In our frmework progrm execution my e of infinite length representing the executions of progrms such s demons or servers. Finite executions re mde infinite y ttching t their end n infinite repetition of void ction. The use of Rin utomton is motivted y the need for determinism in order to simplify our method nd the ssocited proofs. Our pproch drws on dvnces in discrete events system control y [5] nd on relted susequent reserch y Lngr nd Mejri [6] nd consists in comining two models vi the utomt product opertor: model representing the system s ehvior nd nother one representing the property to e enforced. In our pproch, the model representing the system s ehvior is represented y LTS nd the property to e enforced is stted s Rin utomton. The LTS representing the progrm could e uilt directly from the control flow grph fter control flow nlysis [7, 8]. To sum up, our pproch either returns n instrumented progrm, modeled s leled trnsition system, which provly respects the input security policy or termintes with n error messge. While the ltter cse sometimes occurs, it is importnt to stress tht this will never occur if the desired property is sfety property which cn e enforced using existing pproches. Our pproch is thus strictly more expressive. The rest of this pper is orgnized s follows. Section 2 presents review of relted work. In Section 3, we define some concepts tht re used throughout the pper. The elorted method is presented in Section 4. In Section 5, we discuss the theoreticl underpinnings of our method. Some concluding remrks re finlly drwn in Section 6 together with n outline of possile future work. 2 Relted Work Schneider, in his seminl work [1], ws the first to investigte the question of which security policies could e enforced y monitors. He focused on specific clsses of monitors, which oserve the execution of trget progrm with no knowledge of its possile future ehvior nd with no ility to ffect it, except y orting the execution. Under these conditions, he found tht monitor could enforce the precise security policies tht re identified in the literture s sfety properties, nd re informlly chrcterized y prohiiting certin d thing from occurring in given execution. These properties cn e modeled y security utomton nd their representtion hs formed the sis of severl prcticl s well s theoreticl monitoring frmeworks. Schneider s study lso suggested tht the set of properties enforcele y monitors could e extended under certin conditions. Building on this insight, Ligtti, Buer nd Wlker [3, 9] exmined the wy the set of policies enforcele y monitors would e extended if the monitor hd some knowledge of its trget s possile ehvior or if its ility to lter tht ehvior were incresed. The uthors modified the ove definition of monitor long three xes, nmely (1) the mens on which the monitor relies in order to respond to possile violtion of the security policy; (2) whether the monitor hs ccess to informtion out the progrm s possile ehvior; (3) nd how strictly the monitor is required to enforce the security policy. Consequently, they were le to provide rich txonomy of clsses of security policies, ssocited with the pproprite

3 Generting In-Line Monitors 3 model needed to enforce them. Severl of these models re strictly more powerful thn the security utomt developed y Schneider nd re used in prctice. Evolving long this line of inquiry, Ligtti et l. [10] gve more precise definition of the set of properties enforcele y the most powerful monitors, while Fong [11] nd Tlhi et l. [12] expounded on the cpilities of monitors operting under memory constrints. Hmlen et l. [2], on the other hnd showed tht in-lined monitors, (whose opertion is injected into the trget progrm s code, rther thn working in prllel), cn lso enforce more properties thn those modeled y security utomton. In [13], method is given to enforce oth sfety nd co-sfety properties y monitoring. The first prcticl ppliction using this frmework ws developed y Erlingsson nd Schneider in [14]. In tht project, security utomton is merged into oject code, nd sttic nlysis is used to reduce the runtime overhed incurred y the policy enforcement. Similr pproches, working on source code, were developed y Colcomet nd Frdet [15], y Lngr nd Mejri [6] nd y Kim et l. [16 19]. All these methods re limited to enforcing sfety properties, which must e included either s security utomton, or stted in custom logic developed for this ppliction. The first two focus on optimizing the instrumenttion introduced in the code. 3 Preliminries Before moving on, let us riefly strt with some preliminry definitions. We express the desired security property s Rin utomton. A Rin utomton R, over lphet A is tuple (Q, q 0, δ, C) such tht A is finite or countly infinite set of symols; Q is finite set of sttes; q 0 Q is the initil stte; δ Q A Q is trnsition function; C = {(L j, U j ) j J} is the cceptnce set. It is set of couples (L j, U j ) where L j Q nd U j Q for ll j J nd J N. Let R stnd for Rin utomton defined over lphet A. A suset Q Q is dmissile if nd only if there exists j J such tht Q L j = nd Q U j. For the ske of simplicity, we refer to the elements defining n utomton or model following formlism: the set of sttes Q of utomton R is referred to s R.Q nd we leve it s Q when it is cler in the context. A pth π, is finite (respectively infinite) sequence of sttes q 1, q 2,..., q n (respectively q 1, q 2,... ) such tht there exists finite (respectively infinite) sequence of symols 1, 2,..., n (respectively 1, 2,...) clled the lel of π such tht δ(q i, i ) = q i+1 for ll i {0,..., n} (respectively i 0). In fct, pth is sequence of sttes consisting of possile run of the utomton, nd the lel of this pth is the input sequence tht genertes this run. A pth is sid to e empty if its lel is the empty sequence ǫ. We denote y set(π) the set of sttes visited y the pth π. The first stte of π is clled the origin of π. If π is finite, the lst stte it visits is clled its end; otherwise, if it is infinite, we write inf (π) for the set of sttes tht re visited infinitely often in π. A

4 4 H. Chot, R. Khoury nd N. Twi pth π is initil if nd only if its origin is q 0, the initil stte of the utomton, nd it is finl if nd only if it is infinite nd inf (π) is dmissile. A pth is successful if nd only if it is oth initil nd finl. The property of successfulness of pth determines, in fct, the cceptnce condition of Rin utomt. A sequence is ccepted y Rin utomton iff it is the lel of successful pth. The set of ll ccepted sequences of R is the lnguge recognized y R, noted L R. Let q Q e stte of R. We sy tht q is ccessile iff there exists n initil pth (possily the empty pth) tht visits q. We sy tht q is co-ccessile iff it is the origin of finl pth. Executions re modeled s sequences of tomic ctions tken from finite or countly infinite set of ctions A. The empty sequence is noted ǫ, the set of ll finite length sequences is noted A, tht of ll infinite length sequences is noted A ω, nd the set of ll possile sequences is noted A = A ω A. Let τ A nd σ A e two sequences of ctions. We write τ; σ for the conctention of τ nd σ. We sy tht τ is prefix of σ noted τ σ iff τ A nd there exists sequence σ such tht τ; σ = σ. Let A e n ction symol. A stte q Q is n successor of q if δ(q, ) = q. Conversely, stte q is successor of q if there exists symol such tht δ(q, ) = q. Let π = q 1, q 2,..., q n e finite pth in R. This pth is cycle if q 1 = q n. The cycle π is dmissile iff set(π) is dmissile. It is ccessile iff there is stte q in set(π) such tht q is ccessile, nd likewise, it is co-ccessile iff there is stte q in set(π) such tht q is co-ccessile. 2 3 {, } 2 end end C = {({3}, {4}),(, {5})} Fig. 1. A Rin Automton with cceptnce Condition C Fig. 2. Exmple- Leled trnsition system Let us consider Figure 1. It represents Rin utomton. In this figure, ll the sttes re ccessile nd co-ccessile. The pths 3, 4, 3, 4, 3, 3, 4, 3 nd 2, 2 re indmissile cycles, while 5, 5 is n dmissile cycle nd oth infinite pths 1, 2, 3, 4, 5, 5,... nd 1, 2, 3, 4, 3, 4, 4,... re initil nd finl nd therefore oth re successful.

5 Generting In-Line Monitors 5 Finlly security property ˆP is predicte on executions. An execution σ is sid to e vlid or to respect the property if ˆP(σ). A Rin utomton R represents security policy ˆP iff L R = {σ ˆP(σ)}, the set of executions tht stisfy the security policy. Ausing the nottion, we extend the ppliction of ˆP to set of sequences, thus if Σ is set of sequences ˆP(Σ) mens tht ll the sequences of Σ stisfy ˆP. 4 Method In this section we explin our pproch in more detil nd illustrte its opertion with n exmple. The min lgorithm tkes s input Rin utomton R, which represents security Policy ˆP nd leled trnsition system (LTS) M, which models progrm. The lgorithm either returns model of n instrumented progrm tht enforces ˆP on M or returns n error messge. The ltter cse occurs when it is not possile to produce n instrumented progrm tht oth enforces the desired security property nd genertes ll vlid sequences of M. Following [20, 2, 9], we consider tht n enforcement mechnism successfully, enforces the property if the two following conditions re stisfied. First, the enforcement mechnism must e trnsprent; mening tht ll possile progrm executions tht respect the property must e emitted, i.e. the enforcement mechnism cnnot prevent the execution of sequence stisfying the property. Second, the enforcement mechnism must e sound, mening tht it must ensure tht ll oservle output respects the property. We revisit nd expnd these ides in Sections 4.3 nd 5.We illustrte ech step of our pproch using n exmple progrm nd security policy. 4.1 Property Encoding As mentioned erlier, the desired security property is stted s Rin utomton. The security property ˆP to which we seek to conform the trget progrm is modeled y the Rin utomton in Figure 1, over the lphet A { end } with A = {, }. The symol end is specil token dded to A to cpture the end of finite sequence, since the Rin utomton only ccepts infinite length sequences. The finite sequence σ is thus modeled s σ; ( end ) ω. The lnguge ccepted y this utomton is the set of executions tht contining only finite non-empty numer of ctions nd such tht finite executions end with ction. For the ske of simplicity, if sequence σ = τ; ( end ) ω with τ A is such tht ˆP(σ) we sy tht ˆP(τ). 4.2 Progrm Astrction The progrm is strcted s leled trnsition system (LTS). This is conservtive strction, widely used in model checking nd sttic nlysis, in which progrm is strcted s grph, whose nodes represent progrm points, nd whose edges re leled with instructions (or strctions of instructions, or ctions). Formlly, LTS M, over lphet A is deterministic grph (Q, q 0, δ) such tht: A is finite or countly infinite set of ctions;

6 6 H. Chot, R. Khoury nd N. Twi Q is finite set of sttes; q 0 is the initil stte; δ : Q A Q is trnsition function. For ech q Q, there must e t lest one A for which δ(q, ) is defined. Here lso finite sequence σ is extended with the suffix ( end ) ω yielding the infinite sequence σ; ( end ) ω. In generl, sttic nlysis tools do not lwys generte deterministic LTSs. Yet, this restriction cn e imposed with no loss of generlity. Indeed, non-deterministic LTS M over lphet A cn e represented y n equivlent deterministic LTS M over lphet A N, which is equivlent to M if we ignore the numers i N ssocited with the ctions. Ech occurrence of n ction is ssocited with unique index in N so s to distinguish it from other occurrences of the sme ction. In wht follows, we cn thus consider only deterministic LTSs. Furthermore, we focus exclusively on infinite length executions. The exmple progrm tht we use to illustrte our pproch is modeled y the LTS in Figure 2, over the lphet A. The issue consisting of how to strct progrm into LTS is eyond the scope of this pper. As with the Rin Automt, we define pth π s finite or infinite sequence of sttes q 1, q 2,... such tht there exists corresponding sequence of ctions ( 1, 2...) clled the lel of π, for which the δ(q i, i ) = q i+1. The set of ll lels of infinite pths strting in q 0 is the lnguge generted or emitted y M nd is noted L M. 4.3 Algorithm The lgorithm s input consists of the progrm model M nd Rin utomton R which encodes the property. The output is trunction utomton T representing model of n in-lined monitored progrm cting exctly identiclly to the input progrm for ll the executions stisfying the property nd hlting d execution fter producing vlid prefix of this execution. A high level description of the lgorithm is s follows: 1. Build product utomton R P whose recognized lnguge is exctly : L R P = L R L M. 2. Build R T from R P y the ppliction of trnsformtion llowing it to ccept prtil executions of the progrm modeled y M tht stisfy the property ˆP. 3. Check if R T could e used s trunction utomton nd produce LTS T modeling the progrm instrumented y trunction mechnism otherwise produce error. The following sections give more detils on ech step. Automt Product The first phse of the trnsformtion is to construct R P, Rin utomton tht ccepts the intersection of the lnguge ccepted y the utomton R nd the lnguge emitted y M. This is exctly the product of these two utomt. Thus

7 Generting In-Line Monitors 7 R P ccepts the set of executions tht oth respect the property nd represent executions of the trget progrm. Given property utomton R = (R.Q, R.q 0, R.δ, R.C) nd Leled Trnsition system M = (M.Q, M.q 0, M.δ) the utomton R P is constructed s follows: R P.Q = R.Q M.Q R P.q 0 = (R.q 0, M.q 0 ) q R.Q, q M.Q (A { end }) (R.δ(q, ), M.δ(q, )) if R.δ(q, ) nd M.δ(q, ) R P.δ((q, q re defined ), ) = undefined otherwise R P.C = (L,U) R.C{(L M.Q, U M.Q)} The utomton uilt for our exmple using the property in Figure 1 nd the progrm model presented in Figure 2 is given in Figure 3. (3, 2) (4,2) (3, 3) (3, 2) (4,2) (3, 3) (1,1) (2, 4) (3,5) (4, 5) (2, 6) (3,7) (4, 6) C = {( {(3, 2),(3,3),(3,5),(3,7)}, {(4, 2),(4,5),(4,6)} )} Fig. 3. Exmple - Rin utomton R P (1, 1) h hlt hlt (2, 4) (3,5) (4, 5) h (2, 6) (3,7) (4, 6) h C ={({(3,2),(3, 3),(3, 5),(3,7)}, {(4, 2),(4, 5),(4, 6)}),(, {h})} hlt hlt hlt hlt Fig. 4. Trnsformed Product Automton Since R P ccepts the intersection of the lnguges ccepted y the utomton R nd M, it would seem n idel strction from which to uild the instrumented progrm. However, there is no known wy to trnsform such n utomton into progrm. Indeed, since the cceptnce condition of the Rin utomton is uilt round the notion of infinite trces reching some sttes infinitely often, dynmic monitoring system uilt from such n utomton with no help provided y prior sttic nlysis, my never e le to determine if given execution is vlid or not.

8 8 H. Chot, R. Khoury nd N. Twi Insted, we extrct deterministic utomton, T = (T.Q, T.q 0, T.δ), from the Rin utomton R P. This utomton is the leled trnsition system which is returned. It forms in turn the sis of the instrumented progrm we seek to construct. The instrumented progrm is expected to work s progrm monitored y trunction utomton mening tht its model T hs to stisfy the following conditions: (1) T emits ech execution of M stisfying the security property without ny modifiction, (2) for ech execution tht does not stisfy the property, T sfely hlts it fter producing vlid prtil execution, nd (3) T does not emit nything else prt those executions descried in (1) nd (2). The next step towrd this gol is to pply trnsformtion tht llows R P to ccept prtil executions of M which stisfy the property. Indeed, ll finite initil pths in R P represent prtil executions of M, only some of them stisfy the security property. We dd trnsition, leled hlt, to new stte h to every stte in R P where the execution could e orted fter producing prtil execution stisfying the property, i.e. stte (q 1, q 2 ) for which R.δ(q 1, end ) is defined. The stte h is mde dmissile y dding the trnsition (h, hlt, h) to the set of trnsitions nd the pir (, {h}) to the cceptnce set. We hve to e creful in choosing h nd hlt such tht h R.Q M.Q nd hlt A the lphet of ctions. We refer to this updted version of R P s R T, uilt from R P s follows : R T.Q = R P.Q {h} R T.q 0 = R P.q 0 R T.δ = R P.δ {(q, hlt, h) R P.δ(q, end ) is defined } {(h, hlt, h)}. R T.C = R P.C {(, {h})} After this trnsformtion our exmple product utomton ecomes the utomton depicted in Figure 4. The hlt stte h hs een duplicted three times in order to void cross edging. The lnguge recognized y R T is L R T = (L R L M ) {τ; ( hlt ) ω (τ A ) ( σ L M : τ σ) (τ; ( end ) ω L R )}. Extrcting Model of the Instrumented Progrm The next phse consists in extrcting, if possile, leled trnsition system T = (Q, q 0, δ), from the Rin utomton R T. This utomton is expected to ehve s the originl progrm monitored y trunction utomton. To understnd the need for this step, first note tht the cceptnce condition of Rin utomton could not e checked dynmiclly due to its infinite nture. Should we uild n instrumented progrm directly from R T, y ignoring its cceptnce condition, nd treting it like simple LTS, the resulting progrm would still generte ll trces of M tht verify the property ˆP ut it would lso generte the invlid sequences of M representing lels of infinite pths in R T trpped in non dmissile cycles. In other words, the enforcement of the property would e trnsprent ut not sound. In order to generte T, we prune R T of some of its sttes nd trnsitions, eliminting indmissile cycles while tking cre to preserve the ility to generte ll the vlid

9 Generting In-Line Monitors 9 sequences of L M. Furthermore, we need to scertin tht T orts the execution of every sequence of L M not stisfying ˆP nd tht T genertes only executions stisfying ˆP. We cn now restte the correctness requirements of our pproch. In the formultion of these requirements, the ctions end nd hlt re ignored, s they merely model the end of finite sequence. ( σ L M : ( τ L T : ((τ = σ) (τ σ)) ˆP(τ) ( ˆP(σ) = (τ = σ)))) (4.1) τ L T : (( σ L M : ((τ = σ) (τ σ))) ˆP(τ) (4.2) Note tht the requirements 4.1 nd 4.2 re not only sufficient to ensure the respect of soundness nd trnsprency requirements introduced t the eginning of Section 4 following [20, 2, 9], ut lso tht of more restrictive requirement. Indeed, requirement 4.1 lso sttes tht the mechnism is trunction mechnism. It ensures the complince to the security property y orting the execution efore security violtion occurs whenever this is needed. We cn thus prove tht for ny invlid sequence present in the originl model, the instrumented progrm outputs vlid prefix of tht sequence. Our enforcement mechnism is not llowed to generte sequences tht re not relted to sequences in L M either y equlity or prefix reltion. Furthermore these sequences must stisfy ˆP. This is stted in requirement 4.2. Requirements 4.1 nd 4.2 give the guidelines for constructing T from R T. The trnsformtions tht re performed on R T to ensure meeting these requirements re elorted round the following intuition. The utomton R T hs to e pruned so s to ensure tht it represents sfety property even though R is not. Note tht this is not possile in the generl cse without violting the requirements. The ide is tht dmissile cycles re visited infinitely often y executions stisfying ˆP nd must thus e included in T. Likewise, ny other stte or trnsition tht cn rech n dmissile cycle my e prt of such n execution nd must e included. On the other hnd, indmissile cycles cnnot e included in T s the property is violted y ny trce tht goes through such cycle infinitely often. In some cses their elimintion cnnot occur without the loss of trnsprency nd our pproch fils, returning error. The underlying ide of the susequent mnipultion is thus to check whether we cn trim R T y removing d cycles ut without lso removing the sttes nd trnsitions required to ensure trnsprency. The following steps show how we perform the trim procedure. The nest step is to determine the strongly connected components (scc) in the grph representing R T using Trjn s lgorithm [21]. We then exmine ech scc nd mrk it s contining either only dmissile cycles, only indmissile cycles, oth types of cycles, or no cycles (in the trivil cse).to perform this lst opertion, we hve developed heuristics sed on the notion tht grphs which model progrms re structured. A discussion of these heuristics is however eyond the scope of this pper. The next step is to construct the quotient grph of R T in which ech node represents scc nd n edge connecting two scc c 1 nd c 2 indictes tht there exists stte q 1 in scc c 1 nd stte q 2 in scc c 2 nd n ction such tht R T.δ(q 1, ) = q 2. We ssume, without loss of generlity, tht ll the scc sttes re ccessile from the initil node, the scc contining q 0.

10 10 H. Chot, R. Khoury nd N. Twi The nodes of the quotient grph R T re then visited in reverse topologicl ordering. We determine for ech one whether it should e kept intct, ltered or removed. In the wht follows the scc contining the hlting stte h is referred to s H. A scc with no cycle t ll is removed with its incident edges if it cnnot rech nother scc. In Figure 4 the scc consisting of the stte (3, 3) would thus e eliminted. A scc contining only dmissile cycles should e kept, since ll the executions reching it stisfy ˆP. Eliminting it would prevent the enforcement mechnism from eing trnsprent. In our exmple in Figure 4 the scc consisting of the single stte (4, 2) hs only dmissile cycles nd should e kept. A scc contining only non dmissile cycles cn e removed if it cnnot rech nother scc with only dmissile cycles. Otherwise, we re generlly forced to return error. However, in some cses, we cn either rek the indmissile cycles or prevent them from reching H y removing some trnsitions nd keeping the reminder of the scc. This occurs when the only successor, hving dmissile cycles, of this scc is H. In our exmple, the scc contining the sttes (3, 7) nd (4, 6) hs only non dmissile cycles nd H is its only successor. We cn eliminte this scc nd hlt with error t this point. Yet, if we oserve tht eliminting the trnsition ((4, 6),, (3, 7)) would rek the indmissile cycle, we cn eliminte tht trnsition nd keep the rest of the scc. A trnsition cn only e removed if its origin hs h s immedite successor. This is ecuse, should the instrumented progrm ttempt to perform the ction tht corresponds to this trnsition, its execution would e orted. However, prtil execution only stisfies the property if it ends in stte tht hs h s n immedite successor. A scc contining dmissile nd non dmissile cycles my cuse good or d ehvior. Actully, n execution reching this scc my e trpped in n indmissile cycle for ever or my leve it to rech n dmissile cycle thus stisfying the property ˆP. We hve no mens to dynmiclly check whether the execution is going to leve cycle or not. Thus, in this cse we must ort with error. In the exmple given in Figure 4 the scc consisting of the two sttes (3, 5) nd (4, 5) hve one dmissile cycle, (4, 5), (4, 5) nd one indmissile cycle (3, 5), (4, 5), (3, 5). This lst cycle is visited if the invlid sequence () ω is eing generted. Note tht the utomton ccepts n infinite numer of vlid trces of the form () ω, nd tht no trunction utomton cn oth ccept these trces nd reject the invlid trce descried ove. Hence we hve to ort the lgorithm with error in such cses. After removing ll the scc with indmissile cycles nd provided we hve not orted, we cn e sure tht n instrumented progrm uilt from T would not contin ny infinite length execution which does not respect the security property. We must still verify tht whenever the execution is hlted, the prtil sequence emitted stisfies ˆP. The lst step is to check whether the eliminted sttes nd trnsitions could not llow invlid prtil executions to e emitted. This verifiction is sed on the following oservtion: if removed trnsition hs n origin stte tht is not n immedite predecessor of h this would then llow to emit prtil execution tht does not stisfy ˆP. Hence, the verifiction merely consists in checking whether we hve removed trnsitions from sttes tht re not immedite predecessors of h; if such is the cse we hve to ort with error. More precisely, for stte q = (q 1, q 2 ) in T we hve to check

11 Generting In-Line Monitors 11 whether it is possile from q 2 in M to perform ctions tht re not possile from q; if this is the cse, q must hve h s immedite successor; otherwise, we hve no other option thn to terminte the lgorithm without returning suitle LTS nd with n error messge. We my lso remove the trnsitions of the form (h, hlt, h) nd (q, end, q), where q R T.Q. 5 Mechnism s Enforcement power In this section, we show tht non-uniform enforcement mechnisms, which occur when the set of possile executions Σ is suset if A ω, re more powerful thn uniform enforcers, i.e. those for which Σ = A ω, in the sense tht they re le to enforce lrger clss of security properties. This demonstrtion will revel tht monitors tht re tilored to specific progrms my e le to enforce wide set of properties nd rgues for the use of sttic nlysis in conjunction with monitoring. Let us egin with more forml definition of the concepts we discussed in the previous sections, following the nottions dopted in [3, 9]. We specify the enforcement mechnism ehvior of security utomton S y judgments of the form (q, σ) τ S (q, σ ) where q is the current stte of the utomton; σ is the ttempted execution; q is the stte the utomton rech fter one execution step; σ is the remining execution trce to e performed; nd τ is the execution trce consisting of one ction t most tht is emitted y the security utomton fter one step. The execution of the security utomton is generlized with the multi-step judgments defined through reflexivity nd trnsitivity rules s follows. Definition 1 (Multi-step semntics). Let S e security utomton. The multi-step reltion (q, σ) = τ S (q, σ ) is inductively defined s follows. For ll q, q, q Q, σ, σ, σ A nd τ, τ A we hve (q, σ) ε = S (q, σ) (5.1) if (q, σ) τ = S (q, σ ) nd (q, σ ) τ S (q, σ ) then (q, σ) τ;τ = S (q, σ ) (5.2) We re now le to give the definition of wht security enforcement mechnism is. Intuitively, we cn think of security enforcement mechnisms s sequence trnsformers, utomt tht tke progrm s ctions sequence s input, nd output new sequence of ctions tht respects the security property. This intuition is formlized s follows: Definition 2 (Trnsformtion). A security utomton S = (Q, q 0, δ) trnsforms n execution trce σ A into n execution τ A, noted (q 0, σ) S τ, if nd only if q Q, σ A, τ A : ((q 0, σ) τ = S (q, σ )) = τ τ (5.3) τ τ : q Q, σ A : (q 0, σ) τ = S (q, σ ) (5.4)

12 12 H. Chot, R. Khoury nd N. Twi We hve seen tht security enforcement mechnism must respect two properties nmely soundness nd trnsprency. The former requires tht no invlid execution e permitted, while the ltter sttes tht ll vlid executions must e trnsformed into semnticlly equivlent executions. But for enforcement to e meningful, the notion of equivlence must e constrined. Otherwise, one might rgue, for instnce, tht the empty sequence ǫ is equivlent to every vlid execution, nd enforce ny property y orting every execution t its onset. Insted, we rgue tht two executions τ, σ A re equivlent if there exists reflexive, symmetric nd trnsitive, equivlence reltion = s.t. τ = σ. We cn now stte formlly wht it mens for n enforcement mechnism to effectively enforce security property Definition 3 (effective Σ = Enforcement). Let Σ A e set of execution trces. A security utomton S = (Q, q 0, δ) enforces effectively = security property ˆP for Σ if nd only if for ll input trce σ Σ there exists n output trce τ A such tht (q 0, σ) S τ (5.5) ˆP(τ) (5.6) ˆP(σ) = σ = τ (5.7) Informlly, security utomton enforces effectively = property for Σ iff for ech execution trce σ Σ, it outputs trce τ such tht τ is vlid, with respect to the property, nd if the input trce σ is itself vlid then σ = τ. Definition 4 (S Σ = -enforcele). Let Σ A e set of execution trces nd S e clss of security utomt. The clss S Σ = -enforcele is the set of security properties such tht for ech property in this set, there exists security utomton S S tht effectively = enforces this property for the trces in Σ. Our pproch is uilt round the ide, first suggested y Ligtti et l. in [3, 9], tht the set of properties enforcele y monitor could sometimes e extended if the monitor hs some knowledge of the progrm s possile ehvior nd thus cn rule out some executions s impossile. We cn now stte this ide more formlly. Theorem 1. Let S e clss of security utomt nd let Σ, Σ A e two sets of execution trces Σ Σ then we hve S Σ = -enforcele SΣ = -enforcele (5.8) The proof is quite strightforwrd, nd sed upon the intuition tht security mechnism possessing certin knowledge out its trget my discrd it, nd then ehve s n enforcement mechnisms lcking this knowledge.the proof hs een omitted for spce considertion.

13 Generting In-Line Monitors 13 Corollry 1. Let S e clss of security utomton. For ll execution trce set Σ A we hve S A = -enforcele SΣ = -enforcele (5.9) Corollry 1 indictes tht ny security property tht is effectively = enforcele y security utomton in uniform context (Σ = A ) is lso enforcele in the nonuniform context (Σ A ). It follows tht our pproch is t lest s powerful s those previously suggested in the literture tht we uilt round tht lst frmework. It would e interesting to prove tht for ll security utomton clsses, S nd for ll equivlence reltions =, we hve S A = -enforcele SΣ = -enforcele. This is unfortuntely not the cse, s there exists t lest one clss of security utomton (ex. S = ), nd one equivlence reltion (ex. τ = σ τ, σ A ) such tht S A = -enforcele = SΣ = -enforcele for ll set of trces Σ A. However in our pproch, we focus oth on specific clss of security utomt nd on specific equivlence reltion. In our prticulr cse, the set of policies enforcele in nonuniform context is strictly greter thn the one tht is enforcele in the uniform context. The monitors used in this pper re trunction utomt, first descried in [1]. These re monitors which, when presented with potentilly invlid sequence, hve no option ut to ort the execution. Definition 5 (Trunction Automton). A trunction utomton is security utomton where δ : Q A Q {hlt} nd hlt Q. Furthermore, we use syntctic equivlence (=) s the equivlence reltion etween vlid trces. We cn now stte the centrl theorem of this pper, tht the enforcement power of the trunction utomton is strictly greter in the nonuniform context thn in the uniform context, when we consider =-enforcement. Theorem 2. For ll set of trces Σ A we hve T A = -enforcele TΣ = -enforcele (5.10) The proof is sed on the following oservtions. First, it hs een shown in [1, 3] tht property is T A = -enforcele iff it is sfety property. Second. Let ˆP e security property, ˆP is trivilly enforcele on Σ iff for every sequence σ Σ, ˆP(σ). The proof thus consists in showing tht for ny Σ A, nonsfety property cn e stted, nd trivilly enforced.more specificlly, this proof seeks to demonstrte for sequence υ A s. t. υ / Σ the non-sfety security property ˆP(σ) (σ υ) for ll σ A is T Σ = -enforcele. The proof hs een omitted for spce considertion. 6 Conclusion nd Future Work The min contriution of this pper is the elortion of method iming t in-lining security enforcement mechnism in n untrusted progrm. The security property to e

14 14 H. Chot, R. Khoury nd N. Twi enforced is expressed y Rin utomton nd the progrm is modeled y LTS. The in-lined monitoring mechnism is ctully trunction mechnism llowing vlid executions to run normlly while hlting d executions efore they violte the property. In our pproch, the monitor s enforcement power is extended y giving it ccess to stticlly gthered informtion out the progrm s possile ehvior. This llows us to enforce non-sfety properties for some progrms. Nevertheless, severl cses still exist where our pproch fils to find suitle instrumented code. These re cses where n execution my lternte etween stisfying the property or not nd could hlt in n invlid stte, or cses where n invlid execution contins no vlid prefixes where the execution could e orted without lso ruling out some vlid executions. Another contriution of this study is to provide proof tht trunction mechnism tht effectively enforces security property under the equlity s n equivlence reltion is strictly more powerful in non uniform context thn in uniform one. A more elorte prdigm deling with wht constitutes monitor could llow us to ensure the stisfction of the security property in t lest some cses where doing so in currently not fesile. For exmple, the monitor could suppress su-sequence of the progrm, nd keep it under oservtion until it is stisfied tht the progrm ctully stisfies the property nd output it ll t once. Alterntively, the monitor my e llowed to insert some ctions t the end of n invlid sequence in order to gurntee tht the sequence is orted in vlid stte. Such monitors re suggested in [3], their use would extend this pproch to more powerful frmework. Another question tht remins open is to determine how often the lgorithm will succeed in finding suitle instrumented code when tested on rel progrms. We re currently developing n implementtion to investigte this question further nd hope to gin insights s to which of the ove suggested extensions would provide the gretest increse in the set of enforcele properties. Finlly, distinctive spect of the method under considertion is tht unlike other code instrumenttion methods, ours induces no dded runtime overhed. However, the size of the instrumented progrm is incresed in the order O(m n), where m is the size of the originl progrm nd n is the size of the property. The instrumenttion lgorithm itself runs in time O(p c), where p is the size of the utomton s cceptnce condition nd c is the numer of cycles in the product utomton. In prctice, grphs tht strct progrms hve comprtively smll numer of cycles. References 1. F. B. Schneider, Enforcele security policies, Informtion nd System Security, vol. 3, no. 1, pp , K. W. Hmlen, G. Morrisett, nd F. B. Schneider, Computility clsses for enforcement mechnisms, ACM Trnsctions on Progrmming Lnguges nd Systems (TOPLAS), vol. 28, no. 1, pp , L. Buer, J. Ligtti, nd D. Wlker, More enforcele security policies, in proceedings of th Foundtions of Computer Security Workshop, Copenhgen, Denmrk, Jul D. Perrin nd J.-E. Pin, Infinite Words, ser. Pure nd Applied Mthemtics. Elsevier, 2004, vol. 141, ISBN

15 Generting In-Line Monitors P. J. Rmdge nd W. M. Wonhm, The control of discrete event systems, IEEE Proceedings: Specil issue on Discrete Event Systems, vol. 77, no. 1, pp , Jn M. Lngr nd M. Mejri, Optimizing enforcement of security policies, in proceedings of the Foundtions of Computer Security Workshop (FCS 05) ffilited with LICS 2005 (Logics in Computer Science), June-July A. V. Aho, R. Sethi, nd J. D. Ullmn, Compilers, Principles, Techniques, nd Tools. Addison-Wesley, D. Beyer, T. A. Henzinger, R. Jhl, nd R. Mjumdr, The softwre model checker BLAST: Applictions to softwre engineering, Interntionl Journl on Softwre Tools for Technology Trnsfer (STTT), vol. 9, no. 5-6, pp , J. Ligtti, L. Buer, nd D. Wlker, Edit utomt: Enforcement mechnisms for run-time security policies, Interntionl Journl of Informtion Security, , Enforcing non-sfety security policies with progrm monitors, in proceedings of the 10th Europen Symposium on Reserch in Computer Security (ESORICS), Miln, Sep P. Fong, Access control y trcking shllow execution history, in proceedings of the 2004 IEEE Symposium on Security nd Privcy, Oklnd,Cliforni, USA, My C. Tlhi, N. Twi, nd M. Dei, Execution monitoring enforcement under memorylimittions constrints, Informtion nd Computtion, vol. 206, no. 1, pp , A. Buer, M. Leucker, nd C. Schllhrt, Monitoring of rel-time properties, in FSTTCS 2006: Foundtions of Softwre Technology nd Theoreticl Computer Science, ser. Lecture Notes in Computer Science, 2006, pp U. Erlingsson nd F. B. Schneider, SASI enforcement of security policies: A retrospective, in proceedings of the WNSP: New Security Prdigms Workshop. ACM Press, T. Colcomet nd P. Frdet, Enforcing trce properties y progrm trnsformtion, in proceedings of the 27th ACM SIGPLAN-SIGACT Symposium on Principles of Progrmming Lnguges, Jn M. Kim, Informtion extrction for run-time forml nlysis, Ph.D. disserttion, University of Pennsylvni, M. Kim, M. Viswnthn, S. Knnn, I. Lee, nd O. Sokolsky, Jv-mc: A run-time ssurnce pproch for jv progrms, Forml Methods in Systems Design, vol. 24, no. 2, pp , I. Lee, S. Knnn, M. Kim, O. Sokolsky, nd M. Viswnthn, Runtime ssurnce sed on forml specifictions, in proceedings of the Interntionl Conference on Prllel nd Distriuted Processing Techniques nd Applictions, O. Sokolsky, S. Knnn, M. Kim, I. Lee, nd M. Viswnthn, Steering of rel-time systems sed on monitoring nd checking, in proceedings of the Fifth Interntionl Workshop on Oject-Oriented Rel-Time Dependle Systems, WORDS 99. Wshington, DC, USA: IEEE Computer Society, 1999, p U. Erlingsson, The inlined reference monitor pproch to security policy enforcement, Ph.D. disserttion, Cornell University, Ithc, NY, USA, R. E. Trjn, Depth-first serch nd liner grph lgorithms, SIAM Journl on Computing, vol. 1, no. 2, pp , 1972.