Probabilistically Tracking System Calls

Transcription

1 Probabilitically Tracking Sytem Call Saurabh Goyal, and Akah Lal Computer Science Department Univerity of Wiconin, Madion {aurabh, May, 2005 Abtract Monitoring ytem call made by an application i ueful for debugging, for diagnotic a well a for ecurity application. Exiting tool for monitoring ytem call either uffer from a large runtime overhead, or require root permiion to change the kernel of the operating ytem. We propoe an approach that tackle both of thee iue. We implemented a tool that run completely in uer pace and require no change to the operating ytem. It work by probabilitically ampling a few of the ytem call made a proce, thu lowering it overhead. We how that by uing proper ampling technique, our tool i able to figure out the ame information a a tool that track all ytem call. 1 Introduction With increaing oftware complexity, debugging i becoming an important tak. There i a need for tool that can help a uer undertand the execution behavior of a program running on hi ytem. In the abence of detailed undertanding, or the ource code, of a program, the only way to undertand it execution i by monitoring it interaction with the environment. Thi type of black-box debugging approach, where no aumption are made about the program ha the advantage of being applicable to all oftware. In thi paper, we decribe an approach for monitoring the interaction of a program with the operating ytem by recording the ytem call made by the program. We focu our attention on the Linux operating ytem a it i commonly ued, and i le helpful than other operating ytem. Thu, our approach will work on other operating ytem a well. There are a hot of utilitie on Linux that can monitor ytem call of a proce [3, 4]. However, they either uffer from a large runtime overhead, or from requiring root permiion to change the kernel of the operating ytem. We propoe an approach that tackle both of thee iue. We have implemented a tool that run completely in uer pace without requiring any change to the operating ytem. It probabilitically ample a few of the ytem call made a proce, thu lowering it overhead. We how that by uing proper ampling technique, our tool i able to figure out the ame information a a tool that track all ytem call. We alo how that our tool will epecially be helpful in ecurity application that require continuou ytem call monitoring. One uch application i hotbaed intruion detectin ytem (HIDS). Like blackbox debugging, HIDS guard againt ecurity vulnerabilitie without making any aumption about the tructure of a program. They monitor ytem call made by an application and look for deviation from normal ytem call behavior to detect a compromie in the ecurity of the application. The ret of thi paper i organized a follow. Sec- 1

2 tion 2 decribe a Linux tool that can monitor ytem call in uer pace. In Section 3 we modify thi tool to probabilitically monitor ytem call. In Section 4, we preent reult that how overhead and preciion of our tool. The ection alo explain hot-baed intruion detection ytem further. Section 5 dicue ome of the related work, and Section 6 conclude with ome final remark and future work. 2 STRACE A common Linux utility ued for tracking ytem call made by a proce i trace [3]. It entirely execute in uer-mode, and can be ued to attach onto any proce running in the ame uer pace. Once attached, it can track all ytem call made by the proce, and print them out on tandard output in human-readable form. trace alo build up a ummary of ytem call made that maintain information like the number of ytem call made, and the time pent per ytem call type, e.g., it can how the total time taken by all read ytem call, and the average time pent per read ytem call. trace i baed on a Linux ytem call, called ptrace, which i alo the heart of all debugger in Linux. ptrace i ued to attach a parent proce onto a running proce, uch that the parent can oberve and control it execution. The ret of thi ection decribe how trace ue the ptrace ytem call. Once a parent proce P i attached onto a child proce C, it can ak the kernel, via ptrace, to raie a ignal (SIGTRAP) whenever C enter and exit a ytem call. Thi ignal i then caught by P, at which point in time the execution of C i topped. P can now examine the regiter and the core image of C. After examining the content of C, P can reume the execution of C, again via theptrace ytem call. The baic tructure oftrace i hown in Fig. 1, where we aume that ignal are only raied becaue of ytem call. The firt line attache trace onto the proce with ID pid and become it parent proce. Inide the loop, at line 3, trace wait for a ignal from the child, which it get once the child make a ytem call. trace then examine the regiter and tack content of the child to figure out what ytem call wa made, (1) ptrace(pid, ATTACH); (2) while(true) { (3) wait4(pid); (4) proce call tart (5) ptrace(pid, CONT); (6) wait4(pid); (7) proce call end (8) ptrace(pid, CONT); (9) } Figure 1: The baic tructure of trace. The ytem call wait4(pid) upend the execution of calling proce until a ignal i raied by proce with ID pid. The call ptrace(pid, CONT) i ued to reume the execution of the (topped) proce with ID pid. and with what argument (line 4). The child proce i then reumed, after which trace repeat thee three tep in line 6 8 to proce the exit from the ytem call. At the exit, trace record the return value, and alo the time it took for the ytem call to complete. Thi whole proce repeat for the next ytem call. 3 Probabilitic Strace A i evident from the decription of trace preented in Section 2, each ytem call entry and exit require two context witche: from the monitored proce to trace, and back. Thi reult in a large overhead, which ometime low down the monitored proce coniderably. Some meaurement of thi overhead are hown in Section 4. The context witche are actually unavoidable if we want to track ytem call in uer pace. Therefore, in order to reduce the overhead, we mut decreae the number of ytem call monitored, or implement the monitoring 2

3 inide the kernel. A mentioned in the introduction, we want a tool that can be ued by any uer without changing hi ytem. Thu, we dicard the latter approach of changing the kernel. The ret of thi ection decribe ptrace (Probabilitic-trace), a modified verion of trace that lower the overhead by a uerdefined amount by probabilitically monitoring only ome of the ytem call. Suppoe that we want ptrace to monitor every 10 th ytem call of a proce. One approach would be to monitor a ingle ytem call like trace doe; detach from the proce to let it run freely; leep until the proce make 9 ytem call; and then attach back onto the proce to repeat thi whole thing again. Thi would decreae the overhead by 9 time, becaue we only require 2 context witche for 10 ytem call, intead of the 20 context witche that trace required (at the cot of monitoring fewer ytem call). However, implementing thi approach on top of the Linux operating ytem i impoible becaue there i no way of finding out the number of ytem call that a proce made while ptrace wa detached from it. To get around thi difficulty, we meaure the time that the proce took to make a ingle ytem call, and then ue it to etimate the time it would take the proce to make 9 more ytem call. ptrace can then leep for thi amount of time, and attach back onto the proce with the hope that it made exactly 9 ytem call during the time for which ptrace lept. We now formalize thi approach. Suppoe that the uer want to meaure percent of the ytem call (0 < < 100). Then for each ytem call made by a proce, ptrace hould leep for n = ( 100 ) ytem call, i.e., for = 25, ptrace will leep for 3 ytem call for each monitored call. The baic tructure of ptrace i hown in Fig. 2. It involve one mall, but important change. Intead of monitoring jut one ytem call and then leeping for n ytem call, we monitor b ytem call and then leep for b n ytem call. There are two advantage of doing thi. Firt, the mallet time meaurement for the uer time taken by a proce i 10 m (provided in the /proc file ytem), which i too large to meaure the time taken to make a ingle ytem call. A reaonably well choen value of b (1000 or more) enure that our time meaurement are accurate. The (1) ptrace(pid, ATTACH); (2) while(true) { (3) do(b time) { (4) wait n proce tart(pid); (5) wait n proce end(pid); (6) } (7) ptrace(pid, DETACH); (8) leep(b 100 (9) ptrace(pid, ATTACH); (10) } ytem call); Figure 2: The baic tructure of ptrace. The function wait n proce tart i an abbreviation of line 3 5 in Fig. 1, and the function wait n proce end i an abbreviation of line 6 8 in the ame figure. econd advantage of having the parameter b i that it allow u to obtain conecutive equence of ytem call. A we hall in Section 4.1, thi i eential for ecurity application to obtain a fair etimate of a proce behavior. We call b a the burt ize. The etimate of time taken by a proce to make a ytem call i calculated a follow: proce time(pid) + (real time proce time(ptrace)) 2 b Here, proce time i the um of uer and ytem time taken by a proce, and real time i wall-clock time. Thi formula i baed on averaging two etimate of how much proceing the proce took to make a ytem call. Each meaurement i taken for the time it took to execute the loop in line 3 6 in Fig. 2. A running average of thi time i ued to calculate the exact time ptrace will leep for in line 8 by multiplying it by (b 100 ). There i till one deficiency in the ptrace model preented in Fig. 2. Conider monitoring a program that make 100read call followed by 100write call, 3

4 where each call take approximately the ame amount of time. If we chooe b = 100, and our timing i accurate enough, we will oberve all read call but will entirely mi out on the write ytem call. Thi violate our goal of providing a fair etimate of a program behavior. We ue randomization to olve thi problem. Conider a equence of event (in our cae, event are ytem call). We want to oberve each event with an equal probability, ay p (0 < p < 1). Then the following derive the probability of kipping the firt n event, and meauring (n + 1) t event: Pr (Meauring 1 t event) = p Pr (Skipping 1 event, Meauring 2 nd event) = (1 p) p Pr (Skipping 2 event, Meauring 3 rd event) = (1 p) 2 p Pr (Skipping n event, Meauring (n + 1) t event) = (1 p) n p The above i a geometric ditribution with mean (1/p). Therefore, intead of leeping for a fixed number (b 100 ) of ytem call, we hould leep for k ytem call, where k i a random number obtained from a geometric ditribution with mean (b 100 ). Thi give u two probabilitic guarantee: Firt, on an average ptrace will leep for (b 100 ) number of ytem call; and econd, each ytem call i meaured with an equal probability. Thee guarantee are baed on the aumption that the ti mining meaurement are accurate. In practice, the variation in timing i alo a ource of randomne, but i not enough to enure fair ytem call etimate, a we hall ee in Section 4. A imilar trategy of ampling event i ued in program analyi a well, where the objective i to ample certain predefined runtime event, intead of ytem call [8]. The final code tructure of ptrace i hown in Fig Perfect Knowledge ptrace In order to meaure the accuracy of ptrace, we modified the Linux kernel 1 to provide a per-proce meaure of the number of ytem call the proce 1 Appendix A decribe the change. (1) ptrace(pid, ATTACH); (2) while(true) { (3) do(b time) { (4) wait n proce tart(pid); (5) wait n proce end(pid); (6) } (7) ptrace(pid, DETACH); (8) k = rand geo(b 100 ); (9) leep(k ytem call); (10) ptrace(pid, ATTACH); (11) } Figure 3: The complete tructure of ptrace. The function wait n proce tart and wait n proce end are the ame for Fig. 2. The function rand geo(n) generate a random number with geometric ditribution with mean n. made ince it wa tarted. We ued thi meaure to rewrite ptrace and create ktrace (Kernel-baedprobabilitic-trace), which alo run a a uerproce but ha a much better etimate of the time to leep for while detached from the monitored procee. Intead of leeping, ktrace could have pinlocked on the ytem call number to tart monitoring a oon a the right number of ytem call have taken place, but thi would incur an unneceary overhead. Thu, we ue ame time etimate that ptrace ue to calculate leep-time, but we ue the ytem-callnumber information to reolve inaccuracie in uch timing meaurement. After waking up ktrace look at the number of ytem call made while it wa leeping. If thi number, ay n 1, wa different from the required number n 2 = (b 100 ) of ytem call it wanted to leep for, then it adjut the leep time by multiplying it with (n 2 /n 1 ). Thi approach will dynamically try to 4

5 approach the correct leep time value. A a fall-back guard, ktrace alo maintain two global count: the ytem call it ha monitored (g 1 ), and the total number of ytem call the proce ha made (g 2 ), the latter of which i imply obtained from the modified kernel. If the lag in ytem call monitored, defined a ((g 2 /100) g 1 ), exceed twice the burt ize, ktrace decide not to leep at all and continue to monitor the next b ytem call. Thi enure that ktrace i never off it target by much. We preent a detailed evaluation of ktrace along with that of ptrace a it illutrate the improvement in performance of ptrace when the kernel i more helpful that what the Linux kernel i currently. 4 Evaluation For probabilitic tracing of ytem call, there are two major criteria that we have to evaluate the output on. The firt i overhead of the probabilitic tracing and the econd i how well the output tatitic match the true tatitic. In thi ection we how the reult for both the above criteria. We alo preent an experimental reult comparing the reliability of ptrace with ktrace. The lat experimental reult will how the effect of uing the geometric ditribution for calculating the leep time. We evaluated our tool baed on three program, potmark [2], which i a tandard fileytem benchmark, thttpd [5], which i a imple web erver and a toy fileytem program Toy-f that doe a read(), write() and leek() in a loop. Thee program were firt run tandalone and then attached to the original trace program. Then, we ran the program with ptrace and ktrace, four time each. The burt ize for potmark and thttpd wa fixed to 1000, and the burt ize for Toy-f wa fixed to The parameter for percentage of ytem call monitored wa et to 50%, 25%, 10%, and 1% in the four run repectively. Fig. 4 how the overhead of tracking ytem call for the potmark benchmark. The original program ran in 5.62 econd. The total number of ytem call made by the program i 517K. Tracking all thee call uing trace took econd. A we reduce the Execution time of program (in econd) Number of call traced (in thouand) Figure 4: Change in overhead with the number of ytem call traced for potmark. The original program ran in 5.62 econd Execution time (in econd Number of ytem call traced (in million) Figure 5: Change in overhead with the number of ytem call traced for Toy-f. The original program ran in econd number of ytem call traced, the execution time goe down linearly. For tracing 50K call the time taken i 6.55 econd. The time reduce further, but a we how in other graph, the accuracy of the output below thi point uffer. Fig. 5 how the ame graph for Toy-f. It alo how a linear decreae in execution time. The tandalone program take econd. The table below compare the reliability of ktrace with ptrace. The firt column i the requeted ampling percentage and the other two column how how many call were actually tracked. Thi i for the potmark program. The total number of ytem call made by the program are 517K. 5

6 Sample % ptrace ktrace K 255K K 127K 10 63K 50K 1 7K 5K The table how that ktrace track almot the ame percentage of ytem call that it i aked to, wherea ptrace i lightly off the aked percentage. Thi difference wa more for the toy program. Out of a total of 6 million call, the following i what wa traced. Sample % ptrace ktrace M 3M M 1.48M M 0.57M M 0.04M Again, ktrace follow the requeted percentage cloely. ptrace monitor le ytem call than requeted becaue the ytem call rate of Toy-f wa very high, which made it overetimate the leep-time. The next table how the accuracy of the output tatitic. When the potmark program i run with trace, mot of the time i pent in the ytem call open(), write(), and unlink(). The percentage of time pent in thee call i 27%, 22%, and 19% repectively. The table below how how cloely thee percentage match when all the call are not traced. Sample % ptrace, ktrace open() write() unlink() 50 27, 36 22, 22 19, , 26 23, 24 19, , 27 23, 22 19, , 20 28, 25 17, 30 The time tatitic eem to be reaonably cloe for ampling percentage a low a 10%. For 1% ampling, the accuracy i a bit off, in fact the order of percentage time pent ha changed. More time i hown for write() than for open(). In other experiment, the accuracy for the toy fileytem program wa good only for 50% ampling, while for the thttpd program, it wa good down to 1%. One thing to note here i that the actual number of ytem call tracked wa not the ame for ktrace and ptrace, thee number have been reported earlier. The next experiment how that uing the geometric ditribution to calculate the leep-time reult in improved accuracy at lower ampling rate. We how the comparion on the thttpd program. We made a workload of imple HTTP GET requet and traced the weberver ytem call during that workload. The weberver make 270K call during thi time, and mot time i pent in the ytem call poll(), mmap2(), and end(). The actual percentage time pent in thee call i 84%, 11% and 3% repectively. The firt table, here, how the percentage time pent when the leep time wa jut a contant time the burt ize (Fig. 2). Sample % ptrace, ktrace Call open() mmap2() end() 10 47K,33K 82, 84 10, 10 3, 3 1 6K,4K 88, 88 8, 7 1, 2 We ee that the accuracy at 1% ampling i off the original value. The ame figure when taken after modeling the leep time uing a geometric ditribution are hown in the following table. The value at 1% ampling here, are cloer to the original value. Sample % ptrace, ktrace Call open() mmap2() end() 10 43K,33K 84, 84 10, 10 3, 2 1 5K,4K 82, 87 11, 8 3, 2 While not exhautive, thee reult demontrate that probabilitic tracing of ytem call can reult in ignificant reduction in overhead without much lo in the uability of it output. They alo demontrate the improvement made by making the kernel give ytem call information and by uing a geometric ditribution for calculating the leep time. 4.1 Hot-Baed Intruion Detection In thi ection, we briefly explain the utility of ptrace in ecurity application. A oftware i getting more complicated, it i increaingly common to 6

7 find application that have ecurity flaw in them. For example, wu.ftpd (Wahington Univerity ftpd 2.4), a ftp daemon, if miconfigured at compile time, allow uer SITE EXEC acce to /bin. Uer can then run executable uch a bah with root privilege. Such flaw, until dicovered, poe a big rik to the ytem uing thee application, epecially when the application interface with untruted uer on the Internet. Example of uch application are Weberver, client and ftp client [1]. Hot-baed intruion detection ytem (HIDS) are ued by ytem adminitrator to guard againt unknown vulnerabilitie in oftware. One way in which they are ued i the following [9]: Firt, the potentially-vulnerable application i ued by truted uer. HIDS monitor the ytem call made by the application under uch uage, and build a model of the ytem call that the application hould make. Thi model i typically built from conecutive ytem call equence made by the application. In the econd tage, the application in put on the Internet, where untruted uer can ue the application. The ytem call are till monitored, but are checked againt the model built in the previou tage. If in a hort period of time, many ytem call equence do not match any in the model, the application i topped. The intuition i that if there i a break-in into the application, the behavior of the application will change ubtantially o that the ytem call equence made will differ from thoe obtained under valid execution of the application. Thu, a deviation from the normal ytem call that the application make, mean a potential attack i taking place. HIDS preent an intereting cenario for uing ptrace. HIDS require continuou monitoring of ytem call, and overhead mut be mall enough to avoid affecting the performance of the application in quetion. We can ue ptrace to parely ample the ytem call made by the application with low overhead. A oon a ome ytem call equence doe not atify the ytem-call-model, we can increae the ampling rate to monitor more ytem call till the point we are convinced if thi i an attack (more ytem call equence do not match the model) or not (other equence match the model). With the aumption that attack are rare, ptrace will run with very little overhead mot of the time. We have not ued ptrace inide a HIDS, becaue we were not able to imulate real attack cenario. However, we ran ptrace on potmark and recorded all ytem call. Then, we ued a tool called tide [1] to build a databae of all contiguou equence of ytem call of length 6. We found that out of 480K ytem call made by potmark, there were only 159 unique equence. When we ampled down the meaured ytem call to 30K, we till obtained 125 equence. Thi how that there i a high redundancy in the ytem call equence made by a proce, i.e., the ame ytem call equence i made over and over again. Thu, ptrace mie very few ytem call equence. 5 Related Work The idea of ampling program execution behavior i common in dynamic program analyi [8, 6]. The goal there i to monitor the execution of a program by recording the value of program variable at certain program point. Intead of recording the program tate after each intruction, a ampling trategy i ued to monitor only a few location. After recording uch ampled data, certain propertie of the program can be inferred. Like ptrace, the effort i to have a ampling trategy that provide a fair etimate of the program execution. Unlike program variable, tracking ytem call doe not require any emantic knowledge about the program. The ytem community alo make ue of ampling to figure out program behavior. The VMware ESX erver [10] ample the memory page ued by a client operating ytem to etimate the idle memory of the operating ytem. Thi etimate i ued in the page-replacement policy, where higher preference i given to the page of an operating ytem with greater amount of idle memory. The Lottery Scheduler [11] ue a randomized election policy. The guarantee of fairne and proportional haring it provide are probabilitic. Anticipatory cheduling [7] make cheduling deciion baed on an etimate of the time it would take for a proce to make the next requet. Thi i imilar to the leep-time etimate 7

8 thatptrace make for time till the next ytem call. Beide trace, another tool ued for monitoring ytem call i ycalltrack [4]. It i not baed on the ptrace ytem call, but it hijack the ytem call table inide the kernel to call it own function before paing control onto the actual ytem call handler. Thi tool can only be ued by a root uer, a changing the ytem call table require root permiion. However, monitoring ytem call only require an extra function call intead of a context witch. Thi mean that ycalltrack will run with a very mall overhead. One can enviion implementing the ampling trategy of ptrace inide ycalltrack to further reduce it overhead in performance critical application. 6 Concluion and future work In thi paper we have hown the utility of ptrace, a probabilitic ytem call monitoring tool, a a low overhead alternative to trace. We have preented reult that how that overhead can be controlled by the uer by modifying the ampling rate of ptrace. We alo how that a decreae in the ampling rate i not necearily accompanied by a lo in preciion. The ytem call etimate, in the form of percentage time taken per ytem call type, remain fairly accurate in mot cenario. Moreover, the ytem call equence captured at low ampling rate alo faithfully model the actual ytem call equence. Thi motivate the ue of ptrace in ecurity ytem that require uch ytem call information, and alo have a great need for low-overhead continuou ytem call monitoring. A future work, one can add a recent-time-window ummary of ytem call. Thi will include timing information of recent ytem call, and will incrementally forget timing information of previou ytem call. The benefit of having uch a ummary i that it reflect the current interaction of an application with the operating ytem. Thi i ueful for diagnotic purpoe. In cae the network or the file ytem low down, the recent ummary will how the increaed time for ytem call made to thoe device. A limitation of ptrace i that it calculate leeptime baed on the etimate obtained from timing previou ytem call behavior regardle of what part of a proce i execution. Some part of a proce might make ytem call at a fater rate than other part. Thu, a better approach for ptrace would be to track the leep-time relative to the programcounter of a proce, uch that fewer call are monitored from region with high ytem call rate, and more call are monitored from region with low ytem call rate. Thi type of ampling trategy i followed in [6]. The current implementation of ptrace lack the ability to follow fork in a proce. Monitoring multiple procee uing one intance of ptrace would require implementing an event-driven loop, intead of a imple loop that leep when detached from the monitored proce. The current implementation can till be ued by manually invoking ptrace to track child procee. Reference [1] Computer immune ytem. immec/ytemcall.htm. [2] Potmark: A new file ytem benchmark. library/3022.html. [3] Strace: Sytem call tracing. wichert/trace/. [4] ycalltrack: Tracking ytem call inide the kernel. [5] thttpd: tiny/turbo/throttling http erver. [6] Matthia Hauwirth and Trihul M. Chilimbi. Low-overhead memory leak detection uing adaptive tatitical profiling. In Proc. of Int. Conf. on Arch. Support for Prog. Lang. and Operating Sytem, page , [7] Sitaram Iyer and Peter Druchel. Anticipatory cheduling: A dik cheduling framework to overcome deceptive idlene in ynchronou I/O. In Sympoium on Operating Sytem Principle, page ,

9 [8] Ben Liblit, Alex Aiken, Alice X. Zheng, and Michael I. Jordan. Bug iolation via remote program ampling. In SIGPLAN Conf. on Prog. Lang. Deign and Impl., page , [9] Anil Somayaji Steven A. Hofmeyr, Stephanie Forret. Intruion detection uing equence of ytem call. Journal of Computer Security, 6(3): , [10] Carl A. Waldpurger. Memory reource management in VMware ESX erver. In Operating Sytem Deign and Implementation, [11] Carl A. Waldpurger and William E. Weihl. Lottery cheduling: Flexible proportional-hare reource management. In Operating Sytem Deign and Implementation, page 1 11, A Modifying the Linux Kernel In thi ection, we decribe ome of the change we made to the Linux kernel verion to make it record the number of ytem call made by a proce in it lifetime. The kernel maintain a data-tructure of type tak truct, declared in include/linux/ched.h, for each proce running on the machine. We modified thi tructure to include an extra field (ycall) that count the ytem call made by the proce. The macro INIT TASK, declared in the ame file, i ued to initialize the tak tructure for a new proce. Thi wa changed to et ycall to zero. The next tep wa to change arch/i386/kernel/entry.s aembly file (for i386 architecture). It contain the piece of code that a proce trap to whenever it make a ytem call. We added a call to a new function at thi point that increment the ycall of the current proce. Thee change together keep a perfect account of the ytem call made by a proce. The next change wa to report thi value to the uer. Thi wa accomplihed via the /proc file ytem. The file f/proc/bae.c and f/proc/array.c were changed to create an extra file in /proc/pid/ directory that reported the ytem call count. 9