A graph-theoretic framework for isolating botnets in a network

Transcription

1 SECURITY AND COMMUNICATION NETWORKS Securty Comm. Networks (212) Publshed onlne n Wley Onlne Lbrary (wleyonlnelbrary.com)..5 SPECIAL ISSUE PAPER A graph-theoretc framework for solatng botnets n a network Padmn Jakumar* and Avnash C. Kak Department of Electrcal and Computer Engneerng, Purdue Unversty, West Lafayette, IN 4796, U.S.A. ABSTRACT We present a new graph-based approach for the detecton and solaton of botnets n a computer network. Our approach depends prmarly on the temporal co-occurrences of malcous actvtes across the computers n a network and s ndependent of botnet archtectures and the means used for ther command and control. As practcally all aspects of how a botnet manfests tself n a network such as the onlne bot populaton, bot lfetmes, and the duraton and the choce of malcous actvtes ordered by the bot master can be expected to vary sgnfcantly wth tme, our approach ncludes mechansms that allow the graph representng the nfected computers to evolve wth tme. Wth regard to how such a graph vares wth tme, of partcular mportance are the edge weghts that are derved from the temporal co-occurrences of malcous actvtes at the endponts of the edges. A unque advantage of our graph-based representaton of the nfected computers s that t allows us to use graph-parttonng algorthms to separate out the dfferent botnets when a network s nfected wth multple botnets at the same tme. We have valdated our approach by applyng t to the solaton of smulated botnets, wth the smulatons based on a new unfed temporal botnet model that ncorporates the current best understandng about how botnets behave, about the lfetmes of bots, and about the growth and decay of botnets. We also valdate our algorthm on real network traces. Our results ndcate that our framework can solate botnets n a network under varyng condtons wth a hgh degree of accuracy. Copyrght 212 John Wley & Sons, Ltd. KEYWORDS botnet detecton; botnet modelng; network ntruson; graph parttonng *Correspondence Padmn Jakumar, Robot Vson Laboratory, Department of Electrcal and Computer Engneerng, Purdue Unversty, West Lafayette, IN 4796, U.S.A. E-mal: pakuma@purdue.edu 1. INTRODUCTION Botnets are now consdered to be a maor securty threat n computer networks. A botnet s a network of malwarenfected machnes controlled by a sngle entty called the Botmaster. The malware tself may consst of vruses, worms, Troans, and so on, bascally any pece of software wth networkng and communcaton capabltes that was nstalled surrepttously for the purpose of spammng, esponage, launchng of dstrbuted denal-of-servce (DDoS) attacks, and so on [1]. A botnet may nfect mllons of computers. The Storm botnet s estmated to have nfected 5 mllon. The botnet dsmantled most recently, Rustock, was beleved to have nfected over a mllon computers; ths botnet was sendng bllons of mostly fake-prescrptondrugs related spam messages every day [2]. Detectng botnets n a network s the frst step towards makng a network free of such malware. Although t may be possble to brng to bear on the botnet elmnaton problem the more tradtonal vrus scannng and removal tools even wthout knowng explctly about the presence of a botnet, dong the same after a botnet has been properly detected and dentfed would allow for superor strateges to be used for gettng rd of the malware and for the preventon of botnet re-nfecton. Addtonally, detectng ndvdual nfected hosts may be harder than detectng the botnet as a whole because, n the absence of collectve network nformaton, the ndvdual malcous actvtes must be detected wth a hgher degree of accuracy. A key property that s exhbted by botnets s that of temporal correlatons, n the form of co-occurrences, n ther actvtes [3 6]. Ths property, whch dstngushes botnets from other forms of malcous software such as solated vruses and worms, s a key to desgnng an effectve botnet detecton system, as we show n ths paper. Although temporal co-occurrences n malcous actvtes at the nfected computers especally when such co-occurrences are analyzed n a proper statstcal Copyrght 212 John Wley & Sons, Ltd.

2 A graph-theoretc framework for solatng botnets n a network P. Jakumar and A.C. Kak framework facltate botnet detecton, the problem s, nevertheless, made challengng by the followng reasons: as botnet archtecture has evolved over the years, t has become more dffcult to dentfy them by analyzng ther communcaton patterns. Early botnets used the easer to recognze Internet Relay Chat (IRC)-based messagng for command and control (C&C). However, more recent botnets are based on Hypertext Transfer Protocol (HTTP)-based control (that s masked by the more routne HTTP traffc). Fnally, when the bots use peer-to-peer (P2P) protocols as s the case wth the most advanced botnets today for messagng and delvery of malware, that makes t harder stll to dentfy the botnets through protocol-based traffc analyss [7]. Yet, another factor that makes detectng botnets dffcult s that, of all the computers that are nfected n a network, only a small and varable number may be onlne at any gven tme [6]. That dfferent botnets n a network may be engaged n smlar malcous actvtes also contrbutes to the dffculty of solatng the ndvdual botnets [1]. The growth and decay of botnet malware have been compared to the spread of an epdemc [8,9]. At the begnnng, the computers n a network may become nfected at a hgh rate. However, as patches become avalable, the dsnfecton rates would also ncrease. Subsequently, the nfecton and the dsnfecton rates may compete to create large varatons n the footprnt of a botnet. So, the challenge of desgnng an effectve botnet detecton tool bols down to, on the one hand, explotng n a statstcal framework the temporal actvty co-occurrences that can be expected to exst between the dfferent nfected computers and on the other, avodng the several dffcultes mentoned earler. Fulfllng that goal succnctly summarzes the contrbuton we have made n ths paper. Our contrbuton here demonstrates how a graph-based framework for representng the nfected nodes n a network a framework n whch the edge weghts are calculated on the bass of the botnet actvty co-occurrences at the endponts can be used to get around the problems that would otherwse arse when solutons are based on specfc botnet archtectures, specfc actvtes, or on recognzng specfc command-andcommuncaton patterns. Snce the botnet actvtes at the nfected computers can vary sgnfcantly wth tme, our graph-based representaton s allowed to change wth tme. In partcular, the edge weghts that are derved from the temporal co-occurrences of the botnet actvtes at the edge endponts change wth tme. On the one hand, ths leads to a dynamc representaton of a botnet n a network, and on the other hand, ths allows the representaton to be ndependent of the archtecture of the botnet and of the protocols used for ther command and control. Our botnet representaton as derved from the parwse temporal cooccurrences of potentally malcous actvtes s dynamc not only because t vares wth tme but also because t automatcally allows for newly nfected computers to be ncluded n the representaton and for the dsnfected computers to leave the representaton. An mportant ssue related to any new formalsm for botnet detecton s ts valdaton especally when a formalsm s desgned to capture the temporal behavor of a botnet (as ours s). A true statstcally meanngful expermental valdaton would nvolve a controlled necton of botnets nto a large network, placng bot montors at each of the computers n the network and then demonstratng botnet detecton and solaton wth the proposed framework. However, a more practcally feasble alternatve s to smulate botnets accordng to the current best understandng of botnet behavors and ther growth and decay n networks. That s what we have carred out n our work. We have constructed a new unfed botnet model that captures the current best understandng n the research communty about how botnets behave, bot lfetmes, and the growth and decay of botnets. Our valdaton conssts of demonstratng results on such smulated botnets. To add to ths valdaton, we also show results on some real network traces. To summarze, our contrbutons are as follows: We present a new graph-based framework for the solaton of botnets n a computer network. Our approach depends prmarly on the temporal co-occurrences of malcous actvtes across the computers n a network, whch makes t ndependent of the underlyng botnet archtecture. Our approach ncludes mechansms to take nto account the varatons seen n botnet behavor over tme, such as churn n the onlne bot populaton, bot lfetmes, malcous actvty duratons, and the growth and decay of botnets n the network. We valdate the proposed approach by applyng t to real network traces and to a smulated botnet model that captures the current best understandng of botnet behavor over tme. The rest of the paper s organzed as follows: Secton 2 dscusses prevous approaches to botnet detecton. Secton 3 dscusses botnet behavor and the correspondng challenges n botnet detecton n greater detal. Secton 4 presents our graph-based framework for botnet detecton. Secton 5 presents the detals of a new unfed botnet model, whch s used for valdatng our approach to botnet detecton. Secton 6 presents our botnet detecton results on real and smulated network traces. Secton 7 dscusses some lmtatons of the proposed approach and possble ways to get around them. Fnally, Secton 8 concludes the paper. 2. RELATED WORK There now exst several surveys that descrbe botnet archtectures and botnet behavors n consderable detal [6,8,1 13]. Whereas Stone-Gross et al. [1] and Holz et al. [11] have focused on the malcous actvtes that botnets typcally engage n, Raab et al. [6] have delved deeper nto certan specfc ssues related to bot behavor,

3 P. Jakumar and A.C. Kak A graph-theoretc framework for solatng botnets n a network these beng bot on off tmes, bot mgraton, and so on. The reader s attenton s also drawn to [9,14], and [15] where the authors have reported on the propagaton patterns of malware n networks. Wth regard to the lterature that deals specfcally wth botnet detecton, only a few technques have been reported that are archtecture ndependent [13]. A maorty of the proposed technques detect botnets that are centrally controlled wth ther communcatons based on the IRC protocol. As a case n pont, Goebel and Holz [16] have proposed an approach for botnet detecton that s based on dentfyng smlartes n the IRC ncknames used by the hosts. For another example related to IRC-based botnets, Mazzarello [17] has shown how to detect the nfected hosts by analyzng ther IRC communcatons for the vocabulary of known botnets. For a dfferent approach to the detecton of IRC-based botnets, BotSnffer [3] focused on detectng the centralzed C&C center of a botnet; subsequently, t tres to dentfy the bots as hosts that connect to the control center. As bots use encrypted communcatons and more mportantly, wth the evolvng changes n botnet archtecture, these and other IRC-based botnet detecton approaches may no longer be useful. As opposed to the work on IRC-based botnets, only a few approaches have been developed that try to detect the botnets that are based on the P2P archtecture. Holz et al. n [11] descrbed work that deals wth dsablng the P2P-based Storm worm. An approach to P2P botnet detecton that s based on contact tracng was presented by Huang et al. [18]. BotGrep [19] s another technque where the authors attempted to detect P2P botnets by lookng for specfc C&C patterns that can be observed n the overlay topology. There exsts another category of botnet detecton approaches that bases the detecton decsons on the features extracted from network traffc logs and temporal correlatons between these features across the dfferent hosts n a network. These approaches tend to be free of assumptons regardng the archtecture of the botnets and ther communcaton protocols. To cte prevous such contrbutons, the BotGraph framework [4] dentfes spammng botnets as those hosts that show strong correlatons across the hosts n ther transmsson of spam. Along the same lnes, the BotMagnfer s framework [5] dentfes a spammng botnet as a group of hosts that show temporally smlar patterns of spammng behavor. In [2], the authors detected bots as hosts that exhbt temporal correlaton n ther Doman Name System (DNS) queres. BotHunter [21] takes a slghtly dfferent approach where the focus s on detectng ndvdual nfected bots as opposed to botnets a bot s detected by recognzng the varous stages that characterze a typcal bot nfecton. A smlar approach, also amed at the detecton of ndvdual nfected hosts, s adopted n [22] where behavor-based clusterng s used to automatcally analyze and classfy Internet malware by usng system state changes n terms of fles wrtten, processes created, and so on. From the standpont of beng ndependent of the botnet archtectures and makng no assumptons about the communcaton protocols used, of partcular nterest to us s the BotMner contrbuton [1]. In ths approach, a botnet s detected by the followng: () extractng features from the potentally malcous actvty events at the hosts and ther traffc logs; () clusterng the communcaton traffc and the actvty events at the ndvdual hosts n the feature space thus formed; and () carryng out network-wde correlatons of the clusters for the detecton of smlar malcous traffc and actvtes that characterze botnets. Clusterng s also used n the work reported n [23] where the authors have demonstrated botnet detecton by clusterng the dfferent types of applcaton-layer traffc nto the traffc generated by unnfected hosts and that generated by bots. Note that both these approaches do not take nto account temporal co-occurrences of the potentally malcous actvty events and traffc logs n the manner we descrbe n the rest of ths paper. To summarze, most of the exstng technques today are lmted n what they can accomplsh by way of botnet detecton and solaton. Some of the exstng technques are lmted by the fact that they focus on detectng only specfc knds of malcous actvtes that the hosts n a botnet may engage n, whereas some others are lmted by ther ust tryng to fgure out whether a specfc host n a network was nfected by a botnet. The more recent work, such as that reported n [1], s more general, n the sense that t s nether lmted to any specfc botnet archtecture nor based on any assumptons regardng ther communcaton protocols and at the same tme, t seems to dentfy the botnet as a whole n a network. However, t stll does not fully explot the nformaton contaned n the temporal varatons n the malcous actvtes and the related traffc logs. We beleve that what we need are approaches that, n a manner ndependent of the botnet archtecture, not only nclude a wder repertore of malcous actvtes n ther analyss of the actvty and network traffc data but also take nto account temporal aspects of how the malcous actvtes, consdered smultaneously, manfest themselves across the network. The work we present n ths paper s a step n that drecton. 3. BOTNET ACTIVITIES AND THEIR PROPERTIES As mentoned prevously, a bot s a self-propagatng pece of code that nfects hosts through varous explots such as explots based on buffer overflow and other smlar vulnerabltes, explots based on socal engneerng attacks that drop Troans n the hosts, and so on [21]. In ths regard, they are smlar to vruses and worms. Bots, however, dffer from solated vruses and worms n the sense that they exhbt certan specfc communcaton patterns and are externally controlled by a sngle entty that s commonly referred to as the C&C server. The communcaton patterns

4 A graph-theoretc framework for solatng botnets n a network P. Jakumar and A.C. Kak assocated wth bots reflect ther ablty to engage n coordnated malcous actvtes. Detectng botnets s a key goal of network securty admnstrators for reasons that are easy to understand. If a botnet can be dentfed and successfully solated n a network, the same dsnfecton strategy can be quckly appled to all the hosts all at once. Ths consderably reduces the chance of renfecton from the stll nfected hosts when a network s cleaned up one host at a tme. Network securty admns are also generally of the opnon that f one could smultaneously detect all of the nfected hosts partcpatng n a botnet through an analyss of the temporal relatonshps between the potentally malcous actvtes exhbted by the hosts, that would be much more effcent than havng to test each host n solaton for the presence or the absence of a botnet. The goal of ths paper, as descrbed earler, s to detect all the bots n each of the botnets that may have nfected a network. The problem s challengng because of some unque features of bot behavor and growth patterns, whch wll be hghlghted n the followng dscusson. Fgure 1 shows the functonng of a centrally controlled botnet, whch s one of the most common botnet archtectures. There are fve bots n ths botnet. At the nstant that corresponds to the depcton n Fgure 1, three of the fve bots are Onlne, meanng that they are connected to the C&C center of the botnet. These three bots can be assumed to be actvely recevng commands from the botmaster and respondng to those commands. These three bots are dentfed n Fgure 1 by usng sold arrows wth the label Command. The other two bots n Fgure 1 are Offlne and therefore not currently avalable to do the botmaster s bddng. These two bots are dentfed n Fgure 1 by usng dashed lnes wth the label Not Connected. At the nstant to whch the fgure corresponds, the botmaster has commanded hs or her bots to attack a computer that s shown as Vctm n the fgure. All the bots that are currently Onlne are engaged n the attack. The bots that transton from the state Onlne to Offlne wll cease performng the botmaster-dctated command, whereas the bots transtonng from the state Offlne to Onlne wll become nvolved n executng the dctated commands Varable onlne footprnt It has been observed n networks that typcally only a small and varable porton of a botnet s Onlne at any gven tme. Ths s because of a combnaton of factors: Bots may go offlne because some folks want to turn off ther computers at nght; ths may cause a botnet to exhbt a durnal pattern n ts functonng. Addtonally and even more mportantly, the bots n a network may be kept dormant for long perods to avod detecton and possble removal [6,24]. Fgure 2 (from [6]) shows the estmated onlne footprnt of a botnet whose total sze was beleved to be between 8 and 1, hosts. As can be seen n the fgure, only a fracton of the botnet was onlne at any gven tme. In addton, the onlne porton exhbted consderable varaton n sze. The fact that only a small fracton of a botnet may be actve at any gven tme makes detectng the complete botnet extremely challengng Evolvng archtecture and communcaton patterns A maorty of botnets exhbt two dstnct archtectures centralzed and P2P. Centralzed botnets, such as the one n Fgure 1, usually employ the IRC protocol for C&C. The onlne bots of a centralzed botnet receve the Fgure 1. The functonng of a typcal centrally controlled botnet. Fgure 2. Varaton n onlne footprnt of a botnet (from [6]).

5 P. Jakumar and A.C. Kak A graph-theoretc framework for solatng botnets n a network botmaster s commands nstantaneously. Ths form of botnet communcaton has been referred to as the push mode for messagng the bots [1]. Alternatvely, botnets based on the P2P archtecture (such as the Nugache and the Storm worms) receve ther commands n what may be referred to as the pull mode. In the pull mode, a botmaster publshes the command-bearng fles n the P2P network of the botnet. Assocated wth the fles are specfc search keys known to the bots. Each onlne bot polls ts neghbors/peers to locate the command fles by usng the search keys. Such a mechansm s used by the Storm botnet, whch uses an unauthentcated publsh/subscrbe communcaton system [11]. In addton to the centralzed and P2P botnets, we also have HTTP botnets. The bots n HTTP botnets query a desgnated web server for new commands every couple of seconds [3]. It s obvously the case that at any gven tme, only the bots of the botnet that are n the Onlne state can receve the botmaster s commands (n any archtecture). Gven that there exst several dfferent archtectures for botnets (and the fact that these archtectures are constantly evolvng), t s mperatve to not te a botnet detecton strategy to any specfc archtecture. Rather, the focus should be on explotng the temporal co-occurrences n the actvtes exhbted by the bots of a botnet across the board (as an llustraton of the usefulness of temporal co-occurrences, t s noteworthy that temporal correlatons are used to dentfy the spam spewed out by P2P botnets n SNARE [25]). To gve the reader an apprecaton of what sort of bot behavors we are talkng about wth regard to measurng temporal co-occurrences, we show n Table I typcal botmaster commands along wth ther frequences. The data n ths table s based on the observatons of over 192 IRC botnets (n [6]) and some P2P botnets (n [11]) Multple botnets n a network nvolved n smlar actvtes It s no longer uncommon for a computer network to be nfected by multple botnets. These botnets are generally engaged n smlar actvtes, as observed n [6] and [3]. Table I. Relatve frequency of botmaster commands. Command type Frequency (%) Control 33 Scan 28 Spam 15 Mnng 7 Download 7 Attack 7 Other 3 The category Other stands for a grab bag of actvtes such as a nonserver host actng surrepttously as an HTTP server, or actng as an IRC relay server, and so on. Except for the row labeled Spam, ths table was taken from [6]. That row n [6] s labeled Clonng. We beleve that the choce of the label Spam reflects the current best understandng of the relatve frequences of botnet commands. The relatve frequences of occurrence of the dfferent actvtes can be assumed to be accordng to the data shown n Table I. Dfferent botnets beng engaged n smlar actvtes adds to the challenge of solatng the botnets accurately. Fgure 3, whch s an example based on the observatons of common malcous actvtes as reported n [6], presents the actvtes of three botnets n a network that are nvolved n smlar actvtes. In the absence of drectly takng nto account the tmes of occurrence, the botnets are lkely to be dentfed from network traffc logs on the bass of all of the entres up to the tme a decson needs to be made about the presence or absence of botnet malware and the nature of that malware. Fgure 4 shows the expected results at three dfferent nstants (marked as three tme slces n Fgure 3) regardng whch specfc botnets mght have nfected the network when only tme-accumulated actvty reports are taken nto account. Just actvty-based decsons usng network logs from the start of the montorng perod up to Tme Slce 1 yeld three dstnct botnets because each botnet up to that tme was nvolved n one dstnctly dfferent actvty. On the other hand, the logs collected up to Tme Slce 2 (agan from the start of the montorng perod) group the botnets 2 and 3 together because they were nvolved n a smlar mx of actvtes up to that moment. Fnally, by Tme Slce 3, all the botnets look smlar because the accumulated mx of actvtes for each s the same. Therefore, any approach, such as the one used by BotMner [1], that bases botnet dentfcaton decsons on ust the accumulated actvty reports n network traffc logs wll yeld dfferent results at dfferent ponts n tme. It should be noted that, n the presentaton of the results shown n Fgures 3 and 4, we could not have stopped ust after Tme Slce 1 for botnet dentfcaton decsons because, as noted earler, the bots of a botnet come onlne on a dscontnuous bass. One must therefore record the network traffc logs over a suffcently long tme to obtan a more realstc pcture of the botnet actvtes n a network. Fgures 3 and 4 are presented to hghlght the fact that one must also take nto account the temporal varatons n botnet actvtes, n addton to the actvtes themselves, n makng botnet dentfcaton decsons Changng growth patterns Botnets may grow or shrnk over tme. Because of many smlartes between bots and vruses, t s reasonable to assume that the models that descrbe vrus propagaton [9,14,15] would also apply to botnets. Intally, when a new vrus appears n a network, ts growth s exponental. One would expect the same to be true for bot propagaton, and then, as patches become avalable and as newer botnets appear, the footprnt of the orgnal botnet mght shrnk. Rapd varatons n botnet nfecton szes can create addtonal challenges for botnet detecton.

6 A graph-theoretc framework for solatng botnets n a network P. Jakumar and A.C. Kak Fgure 3. Overlappng botnet actvtes n a network. Fgure 4. Identfyng botnets ust on the bass of accumulated actvtes as shown n Fgure Changng botnet membershp over tme The hosts n a network may go from beng nfected by one botnet to beng cured and then to beng renfected by a dfferent botnet. Therefore, the botnet label of a host may change wth tme. It has been observed that at a gven tme, usually only one botnet s actve n a host. Most botnets are reslent to takeover attempts by other botnets, and n the case that such a takeover happens, the new botnet makes sure to remove exstng nfectons [11,26]. In [26], SRI researchers who analyzed the Confcker C worm observed that the worm prevented the other well-known botnet nfectons from attackng the host n whch the worm resded. The ssues we have descrbed n ths secton pont to the challenges that must be addressed by a relable botnet detecton framework. On the strength of our understandng of these ssues, n ths paper, we propose a graph-based framework that has the potental to address several of these ssues. We model the nfected hosts n a network as a graph G =(V, E) where V s the set of nodes that represent the hosts that have exhbted sgns of malcous actvty and E s the edges between the nodes, each edge representng the lkelhood that the two nodes connected by the edge are part of the same botnet. Edges, updated perodcally, encode both the actvtes and the temporal co-occurrences between them. Once the actvtes have been observed over a suffcently long tme (to allow for a large enough populaton of the bots to become actve at some pont) and we have updated the edges, we partton the graph by usng the Normalzed Cuts algorthm proposed n [27]. Ths parttonng assgns a bot label to each of the nodes n the graph, wth each label correspondng to a dfferent botnet. In ths manner, we can dentfy the dfferent botnets operatng n a network and fgure out whch host s actve n what botnet. The combnaton of our edge update procedure and graph parttonng allows for the dentfcaton of the bots of the same botnet even when they are actve at dfferent nstants. Our framework s also able to dstngush between the dfferent botnets that may have been nvolved n the same actvty at the same nstant. Even more partcularly, our approach does not requre storage of network logs. At any pont, all that needs to be stored s the updated graph. Hosts that become nfected are added to the graph as new nodes, and the ones that stop exhbtng sgns of malcous actvty (and are relably known to be free of nfecton) are removed from the graph. The followng secton dscusses the proposed approach n greater detal. 4. A GRAPH-BASED FRAMEWORK FOR BOTNET DETECTION The goal of assgnng an accurate botnet label to each nfected host n a network s acheved by representng the nfected hosts by a fully connected attrbuted graph- G =(V, E), where V s the set of nodes, wth each nfected host standng for a node, and E s the set of edges. A node s added to the set V when a new host exhbts malcous actvty. On the other hand, a node s removed from V when the correspondng host ceases to exhbt malcous actvty and s relably known to have been dsnfected. These nodes can be added back to V f and when the correspondng hosts exhbt malcous actvty agan. What makes the graph G attrbuted s that the edges have weghts assocated wth them. The weght of an edge connectng two nodes s the lkelhood that the two nodes are part of the same botnet. Edge weghts take values between and 1. An edge weght close to 1 ndcates that there s a hgh probablty that the two nodes are part of the same botnet, whereas an edge weght close to ndcates that t s almost certan that the two nodes belong to dfferent botnets. Edge weghts are updated at the end of each tme slot to reflect the observatons n that partcular tme. For nstance, f the two nodes performed the same malcous actvty durng ths perod, the correspondng edge weght s ncreased because our confdence that the two nodes are part of the same botnet wll have ncreased. Weghts for dfferent actvtes dffer and are theoretcally

7 P. Jakumar and A.C. Kak A graph-theoretc framework for solatng botnets n a network derved (n a manner we wll explan shortly) to represent the confdence we can place n our observaton of those actvtes. As hosts belongng to the same botnet tend to exhbt temporally smlar actvtes over an extended tme, the edges between such hosts n graph G acqure large weghts. On the other hand, the edges between the hosts that belong to two dfferent botnets exhbt low edge weghts. Assumng that a network of computers has been attacked by multple botnets, for botnet detecton and dentfcaton, we are nterested n parttonng G nto dsont subgraphs, wth each subgraph correspondng to a unque botnet. Ths obvously assumes that each nfected machne can be assumed to be under the control of a sngle botnet at any gven tme an assumpton whose ustfcaton was provded n Secton 3. We want our botnet detecton algorthm to assgn each nfected node a unque botnet label. We use the Normalzed Cuts algorthm developed n [27] to segment the graph. Graph parttonng can be performed repeatedly to assgn botnet labels as new bots/botnets make appearances n the network. Fgure 5 summarzes the dfferent stages of our approach Malcous actvty detecton at a host To detect and measure the actvty co-occurrences needed by our approach, one must have n place the tools needed for the detecton of malcous actvtes. Over the years, varous methods have been developed for that purpose. For example, the router n charge of a local area network can be equpped wth a montor that can log network traces assocated wth each clent n ts Dynamc Host Confguraton Protocol (DHCP) and statc-ip-address clent lsts. Such loggng of network traces s supported by many routers, such as those made by Csco and Junper. Subsequently, these logs may be analyzed to detect f malcous actvtes have occurred at the hosts. To cte some prevous efforts that perform effcent detecton of malcous actvtes, scannng actvtes can be detected by montorng the faled connecton rate from a host as n SCADE [21]. Spammng by a host can be detected by montorng the followng: () the frequency of Doman Name System queres for mal exchanger records emanatng from the host; and () the frequency at whch the host ntates the Smple Mal Transfer Protocol connectons to outsde mal servers [1]. Malcous download detecton can be carred out by usng Snort s sgnature detecton engne. Other malcous actvtes can be smlarly detected usng freely avalable network ntruson detecton and preventon systems. Alternatvely, such malcous actvtes can be detected locally at a host by usng tools such as Snort and BotHunter. Our contrbuton n ths paper s not to rebuld tools such as those mentoned earler. Rather, our am s to use the output of the tools already n exstence to detect malcous actvtes and to then use the measures of those actvtes to detect and dentfy the botnets by usng a graph-based statstcal framework for the analyss of data Updatng edge weghts Assumng that the actvtes lsted n Table I have been measured durng dscretzed tme ntervals, we wll now explan how the ntal graph G comes nto exstence and how ts edge weghts are subsequently updated as the actvty measures change wth tme. We wll assume that each dscretzed tme nterval s of unt duraton. Let, be two nodes n graph G, that s, these are hosts n the network that have exhbted sgns of malcous actvty. Let C be a (latent) random varable ndcatng f the hosts, are part of the same botnet or not: C = 1 ndcates that the two hosts are part of the same botnet, and C = ndcates that they are not. We are nterested n computng p(c =1)to estmate the probablty that the two hosts, are part of the same botnet on the bass of the observatons up to the current tme t. At any gven tme t, the edge weght between the two nodes, s denoted as pðc t ¼ 1Þ. Because the edge weght s a probablty, t takes on a value between and 1. As tme s dscretzed nto unt duraton ntervals, we have t = {, 1, 2,...}. The edge update at each tme t wll be based on the observatons of malcous actvty between t 1andt. Let A ðtþ and A ðtþ be the random varables that descrbe the actvty shown by the th and th host, respectvely, between tmes t 1 and t. Let a and a be the actual values taken on by the varables A ðtþ and A ðtþ. a and a then ndex nto the malcous actvty table shown n Table I. a can take values from, 1,..., K, where a = ndcates that the host showed no malcous actvty durng the tmeslot under consderaton, whereas a = m (1 m K) ndcates that the host performed the mth malcous actvty n Table I between t 1 and t (because there are seven malcous actvtes lsted n Table I, K = 7). Fgure 5. Stages n our approach.

8 A graph-theoretc framework for solatng botnets n a network P. Jakumar and A.C. Kak At each dscrete tme t, we update the pror value of pðc ðt 1Þ Þ between every par of nodes to a posteror value pðc ðtþ Þ n a probablstc fashon by usng Bayes rule. The posteror (updated) edge weght between two nodes depends on the prevous edge weght pðc ðt 1Þ Þ, the lkelhood of seeng ths group of actvtes (A ðtþ and A ðtþ ) gven the value of C, and the overall lkelhood of seeng ths par of actvtes ; A ðtþ Þ: pðc ðtþ A ðtþ ¼ ¼ a ; A ðtþ ¼ a ¼ a ; A ðtþ ¼ a C ðtþ p C ðt 1Þ p A ðtþ ¼ a ; A ðtþ ¼ a The lkelhoods ¼ a ; A ðtþ ¼ a C ¼ 1 and ¼ a ; A ðtþ ¼ a C ¼, whch are essental for calculatng the left-hand sde of Equaton (1), are estmated for every par of the possble botnet actvtes lsted n Table I. These ont probabltes are estmated theoretcally, as dscussed n the followng secton. It s mportant to realze that there s never a need to drectly estmate the denomnator on the rght-hand sde of Equaton (1), that s, because the normalzaton constrant on the probabltes assocated wth the two outcomes for C ndrectly yelds the denomnator n Equaton (1). That s, f we estmate these two probabltes wthout knowledge of the denomnator, then the fact that these probabltes must add up to 1 wll gve us the unknown value for the denomnator. Edges are updated between a par of hosts, at the end of a tmeslot only f at least one of them exhbted malcous actvty durng the tmeslot, that s, ether A ðtþ 6¼ or A ðtþ 6¼. At the start of the montorng perod, that s, when the graph G s frst constructed, the edge weghts are ntalzed to.5 between every par of nodes exhbtng malcous actvty. A weght of.5 ndcates that the odds are even that the hosts at the two endponts of the edge are nfected by the same botnet, on the one hand, and by two dfferent botnets, on the other hand. If a network s nfected by only one or two botnets, one could argue for ths ntal probablty value to be hgher. But then, one can also argue for ths ntal probablty to be lower f a network has been attacked by many small botnets. In the absence of any pror nformaton regardng the number and/or the szes of the (1) botnets n a network, we choose to make an unbased ntalzaton of.5 for all the edges between the nodes exhbtng actvtes of the sort shown n Table I. After the graph G s created, when a new host n the network starts showng sgns of malcous actvty (and s not part of G yet), t s added to G, and new edges are created from the ust nfected host to each node already n G wth an ntal unbased edge weght of Estmatng the ont actvty dstrbutons. As mentoned earler, A ðtþ and A ðtþ are the actvtes detected by the network montor at the nodes and, respectvely, between the nstants t 1 and t. We are nterested n estmatng the ont actvty dstrbutons ¼ a ; A ðtþ ¼ a C ¼ 1 and ¼ a ; A ðtþ ¼ a C ¼ for all possble parngs of a and a. Let B ðtþ and B ðtþ be the botmaster-ordered actvtes at the nodes and between tmes t 1 and t, wth b and b beng two specfc values for such actvtes. Just smlar to A ðtþ B ðtþ s a K + 1-valued random varable, wth K beng the number of actvtes n Table I. Note that b = ndcates that the botmaster chose no malcous actvty for the tmeslot (.e., let hs botnet dle) and that b = l (1 l K) ndcates that the botmaster pcked the lth malcous actvty n Table I to be performed between t 1 and t. The probabltes pðb ðtþ ¼ bþ represent the relatve frequences of the dfferent malcous actvtes ordered by the botmaster. On the bass of the emprcal data, we can assume that, on average, a botmaster lets hs botnet dle 25% of the tme. For the rest of the tme, the actvtes ordered by a botmaster are accordng to the relatve frequences n Table I. The specfc values taken on by A and A at tme t depend on the actvtes B and B selected by the botmasters of the nodes and for that tme slot. We must also allow for the possblty that both nodes mght be controlled by the same botmaster. Recall the use of C to represent whether or not the two bots and are part of the same botnet. If C = 1, that ndcates that and are part of the same botnet and have the same botmaster, and when C =, the bots and are under the control of two dfferent botmasters. We are obvously nterested n the probablty that two dfferent nodes exhbt two gven actvtes when both nodes are n the same botnet and when they are not. The followng formula expresses these ont probabltes: We wll now show smpler versons of the aforementoned formula for the two cases of C = 1 and C =., ¼ a ; A ðtþ ¼ XK b ¼ X K b ¼ ¼ a C Þ ¼ a ; A ðtþ ¼ a B ðtþ ¼ b ; B ðtþ ¼ b ; C Þ (2) pðb ðtþ ¼ b ; B ðtþ ¼ b C Þ

9 P. Jakumar and A.C. Kak A graph-theoretc framework for solatng botnets n a network When C = 1, both the bots and are controlled by the same botmaster. We ndcate that fact by sayng B = B = B, that s, B s now the random varable for the common actvty at the two nodes. In ths case, Equaton (2) becomes ¼ a ; A ðtþ ¼ XK b¼ pðb ðtþ ¼ bþ ¼ a C ¼ 1Þ ¼ a ; A ðtþ ¼ a B ðtþ ¼ bþ where b s any specfc value for the random varable B, the common actvty ordered by the botmaster for the botnet. Note that we have removed the dependence on C n Equaton (3) because the fact that the two bots have the same botmaster mplctly assumes that C = 1. To further smplfy the equaton shown earler, we note that A ðtþ s ndependent of A ðtþ because the actvty actually carred out by bot depends only on the actvty selected by the bot s botmaster and s ndependent of what s gong on at the bot. We can, therefore, rewrte the aforementoned equaton as ¼ a ; A ðtþ ¼ a C ¼ 1 ¼ XK b¼ ¼ a B ðtþ ¼ bþ ¼ a B ðtþ ¼ bþpðb ðtþ ¼ bþ In the other case, that s, when C =, the bots and have two dfferent botmasters B and B. Agan, the dependence on C = can be removed n the formula for the ont probablty ¼ a ; A ðtþ ¼ a C ¼ as long as we mantan the dstncton between the actvtes B and B at the two nodes. As wth the C = 1 case, A ðtþ s agan ndependent of A ðtþ because the actvty actually carred out by bot depends only on the actvty selected by the bot s botmaster and s ndependent of what s gong on at the bot. We can, therefore, rewrte Equaton (2) for the case C = as follows: ¼ a ; A ðtþ ¼ XK b ¼ XK b ¼ ¼ a C ¼ Þ ¼ a B ðtþ ¼ a B ðtþ ¼ b Þ pðb ðtþ ¼ b Þ pðb ðtþ ¼ b Þ ¼ b Þ Both Equatons (4) and (5), whch provde the ont dstrbutons over the observed malcous actvtes at a par of hosts under the condton that they belong to the same botnet and under the condton that they belong to two dfferent botnets, depend on us beng able to estmate p(a (t) = a B (t) = b) the probablty that the observed malcous actvty at a bot s a whle the botmaster chose b. Ths probablty can be ascertaned from the followng consderatons: (3) (4) (5) The probablty p(a (t) = a B (t) = b) depends obvously on whether the nfected host for whch the probablty s beng estmated s n the Offlne state as a bot or the Onlne state. When an nfected host s n the Onlne state, we assume t can receve the commands of the botmaster and do the bddng of the botmaster as drected. We must also account for the possblty that a network montor may ms-dentfy the malcous actvty at a bot. As was mentoned earler n Secton 3 and for reasons presented n that secton, of all the hosts nfected by a botnet, typcally only a small number wll be n the state Onlne at any gven tme. It has been observed emprcally that the tme a bot remans n the state Onlne s exponentally dstrbuted wth mean a, whereas the tme t s n the state Offlne s also exponentally dstrbuted but wth mean b. On the bass of expermental observatons on botnets, a and b are beleved to be roughly 25 and 5 mn, respectvely [6], so we can say that the probablty that an nfected host s n the bot state Onlne at any gven nstant s p on ¼ a a þ b Let us now talk about the probablty of a network montor ms-dentfyng the malcous actvty at a node. We wll denote ths probablty by p error.if^a ðtþ s the true actvty performed by bot between t 1andt and A ðtþ s the actvty as detected by the network montor, then obvously, p error ¼ ðtþ 6¼ ^A Þ. We assume that ths probablty of erroneous detecton by the network montor s unformly dstrbuted over all possble ways to ms-dentfy a gven malcous actvty. We now brng together these consderatons nto the followng result for the probablty p(a (t) = a B (t) = b): (6) ¼ ab ðtþ ¼ b ¼ pon ½ð1 p error Þ 1ða ¼ bþ (7) þ p error K 1ða 6¼ bþš þ ð1 p onþ 1ða ¼ Þ where 1(s) s an ndcator functon that equals 1 when ts argument s s true and otherwse. To understand the rght-hand sde earler, note that, n the absence of observaton errors (whch happens wth probablty 1 p error ), the node where a s observed must exhbt the same actvty as the actvty b ordered by the botmaster provded, of course, the node s n the state Onlne, whch happens wth probablty p on. To ths, we must add the probablty of the montor erroneously tellng us that the actvty a was observed whle the true actvty at the node was really b the actvty ordered by the botmaster. The last part of the rght-hand sde earler s the estmate of p(a (t) = a B (t) = b) for the specal case when a =, meanng when the nfected host to

10 A graph-theoretc framework for solatng botnets n a network P. Jakumar and A.C. Kak whch A (t) apples s n the state Offlne, whch obvously happens wth the probablty (1 p on ). Usng the formulas shown earler, the estmated ont dstrbutons ; A ðtþ ðtþ C ¼ 1 and pða ; A ðtþ C ¼ are gven n Tables II and III, respectvely. For the estmates shown n the tables, we assumed p error =1%, a = 25, and b =5. The equatons provded n ths secton make relatvely few assumptons and are based on the observed behavor of the botnets. We wll use them to estmate the temporal actvty co-occurrences at pars of nfected nodes under the condton that both nodes n a par are nfected by the same botnet and under the condton that the nfecton at the two nodes s by two dfferent botnets. These actvty dstrbutons wll be used to update the edge weghts n the graph representaton of the nfected hosts n a network, as dscussed earler, at the end of each unt of observaton tme. At the end of the montorng perod, we wll end up wth a graph representaton n whch the edge weghts gve us the lkelhood that the correspondng nodes belong to the same botnet. We, then, segment out the botnets n the graph by usng graph parttonng, the detals of whch are dscussed n the succeedng text Graph parttonng The nput to the graph-parttonng step s a graph whose nodes are the hosts that have exhbted malcous actvtes of the sort lsted n Table I and whose edges represent the lkelhood that the two nodes lnked by an edge are part of the same botnet. The lkelhood of any two nodes belongng to the same botnet s based on the cumulatve temporal co-occurrences n the malcous actvtes exhbted at those nodes. Only those hosts whose actvty counts exceed a certan threshold are ncluded n the graph to be parttoned. Ths s to ensure that the nodes to be used for botnet dentfcaton, and whch are to be assgned a botnet label, have shown a more or less stable pattern of malcous behavor. A count of 15 for the threshold has worked well n our experments. By that, we mean that a node s ncluded n the graph to be parttoned f the montor has reported at least 15 malcous actvty counts at the node, whch could be 15 of the same actvty over 15 unts of observaton tme or a total of dfferent malcous actvtes over whatever perod t takes for the count to add up to 15. We are now nterested n segmentng G nto dfferent botnets, assumng that more than one botnet has nfected the network as s not uncommonly the case. We wll use the Normalzed Cuts algorthm [27] for the same. Ths algorthm nvolves the followng steps: Estmate a V dmensonal partton vector to b-partton the graph. Decde f the parttons created should be further subdvded, and repartton the segmented parts f necessary B-parttonng the graph. Gven the graph G =(V, E), we are nterested n parttonng t nto two dsont subgraphs, wth one nvolvng the subset P of the nodes and the other the subset Q of nodes such that P Q = V and P Q =. The cost of bparttonng s the weght of the edges that go from the nodes n one partton to the nodes n the other partton. Table II. ; A ðtþ C ¼ 1. Actvty e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e Table III. ; A ðtþ C ¼. Actvty e-5

11 P. Jakumar and A.C. Kak A graph-theoretc framework for solatng botnets n a network For optmal b-parttonng, we would want to determne P and Q so that ths cost s mnmzed. The followng formula s a normalzed verson of ths cost: where and NcutðP; QÞ ¼ cutðp; QÞ cutðp; QÞ þ assocðp; VÞ assocðq; VÞ cutðp; QÞ ¼ assocðp; VÞ ¼ X 2P; 2Q X 2P; 2V (8) w (9) w (1) where w s the edge weght between the nodes and. In our case, w ¼ pðc ðtþ Þ. Mnmzng the normalzed cost Ncut(P, Q) smultaneously mnmzes the assocaton between the parttons and maxmzes the assocaton wthn a partton. Ths creates tght, well separated parttons. As explaned n [27], we seek a V dmensonal partton vector y that mnmzes the Ncut cost functon such that y = + 2 f node s n P and 2b f the node s n Q, where b s the rato of the sum of the weghted vertex degrees n the two subgraphs yelded by the partton. For our dscusson here, the mportant pont s that y s a vector wth bnary valued elements. Unfortunately, determnng such a vector y drectly s known to be NP-Hard. However, an approxmate dscrete soluton can be obtaned by allowng the elements of y to take on real values. Subsequently, the calculated value for the elements of y can be thresholded to yeld a good approxmaton to the optmal soluton. Let W be an N N symmetrc matrx where W(, )=w. Let d()= P w where d() measures the total connecton weght from node to all other nodes. Let D be an N N dagonal matrx wth d on ts dagonal. If y s relaxed n the manner descrbed earler, we can obtan a good approxmaton to the optmal soluton for the Ncut problem by solvng the followng generalzed egendecomposton problem: ðd WÞy ¼ ldy (11) We can transform Equaton (11) to a standard egen system of equatons by wrtng t as D 1 2 ðd WÞD 1 2 z ¼ lz (12) where y ¼ D 1 2 z. The second smallest egenvector, represented by z 1, s then the soluton we want for the aforementoned system of equatons. Ths soluton can be turned nto a soluton for y by y ¼ D 1 2z 1. The contnuous values obtaned for y as explaned earler can subsequently be dscretzed by usng a threshold that acheves the lowest Ncut value. After we splt the graph G nto two parts by usng the vector y, we determne f the parttons created should be further subdvded. If t s deemed that further parttonng s to take place, then each partton s subdvded by followng the steps outlned earler, else the recurson for that partton termnates Recursve parttonng. A recursve applcaton of the algorthm we have descrbed so far s used to further partton the graph as long as the followng two crtera are met: () the partton vector s stable; and () the Ncut value of a partton vector s below a certan threshold, as explaned n [27]. Wth regard to the stablty of the partton vector, we do not want partton vectors whose element values are smoothly varyng (such partton vectors are ambguous because they exhbt multple possbltes for the splttng pont). The possblty that a gven y conssts of smoothly varyng values can be ascertaned by makng a hstogram of the values of the vector elements and by computng the rato between the mnmum and maxmum values n the bns. The rato wll be low for stable partton vectors. In our expermental work, we reect parttons for whch ths rato, defned as the Cut Stablty Threshold, s greater than.7. The second crteron s based on the Ncut value of the partton vector. We would want partton vectors to have an Ncut value as low as possble. Hence, we must reect partton vectors wth Ncut values above a threshold the Ncut threshold. Ths threshold s based on how dscrmnatng one wants to be wth regard to separatng out the botnets that may behave very smlarly. 5. A UNIFIED BOTNET MODEL FOR VALIDATING DETECTION FORMALISMS An mportant ssue related to any new formalsm for botnet detecton s ts valdaton. Valdaton s partcularly challengng for formalsms that seek to take nto account the temporal aspects of botnet actvtes, actvty duratons, bot lfetmes, botnet growth and decay, and so on. Because any statstcally meanngful valdaton must of necessty nvolve a large network, a controlled necton of botnets nto such networks, placement of bot montors at each of the computers n the network, and so on, t s easy to see why truly expermental valdatons are not feasble n real lfe. One must therefore resort to the alternatve of smulatng botnets that take nto account our latest collectve understandng of botnet behavors and lfetmes. The goal of ths secton s to present a unfed model of botnets that pulls together the latest understandng n the research communty about how botnets behave, bot lfetmes, and the growth and decay of botnets. The unfed model conssts of two man components a behavoral component that captures the actvtes that a botnet carres out, how long the bots of a botnet stay onlne, and so on (these are based on the observatons n [6,11]), and a

12 A graph-theoretc framework for solatng botnets n a network P. Jakumar and A.C. Kak propagaton component that captures how a botnet grows and decays over tme. The propagaton of botnets s based on vrus nfecton models proposed n [9,14], and [15]. Because botnets are essentally vruses/worms that are centrally controlled, these vrus propagaton models are readly extended to modelng the spread of botnets. Our model uses dscrete tme. Let N be the number of hosts n a network. At any gven tme t, each host n the network s n one of the three categores: Infected (I), Susceptble (S), or Cured (C) (based on the categorzaton n [9]). Hosts n the Infected category are part of some botnet, and each host has a unque botnet label. Hosts that are n the Susceptble category are presently not nfected but can be nfected n the future. Hosts n the Cured category are machnes that were recently dsnfected and are mmune to new nfecton attempts for a certan (fxed) tme. After ther mmunty to nfectons expres, these hosts wll move from the Cured category to the Susceptble category. The number of hosts n each category vares wth tme as some Susceptble hosts may become nfected, whereas some exstng Infected hosts may become cured. At any tme t, the hosts n all the categores sum to N (the total number of hosts n the network), that s, as follows: reman mmune to nfecton attempts. Infecton attempts only succeed on hosts n the Susceptble category. These hosts are then assgned the same botnet label as the botnet label of the host that nfected them and move nto the Infected category at the next samplng nstant. Ther behavor s then governed by the bot model descrbed earler, where they transton between Onlne and Offlne states and obey the commands of the botmaster whle Onlne. Fnally, the hosts n the Infected category can become cured at any tme wth a probablty l. If a host becomes cured, t moves from the Infected category to the Cured category. Fgure 6 shows the transtons of hosts between the dfferent states n a network. Table IV summarzes the symbols n the model. IðtÞþSðtÞþCðtÞ ¼N (13) We are nterested n modelng the hosts n the Infected category as these are the computers n the network that are members of some botnet. As dscussed earler, at any gven tme, each such bot s n one of the two states Onlne or Offlne. The tme duratons for whch a bot stays Onlne or Offlne are exponentally dstrbuted wth means a and b, respectvely. Ths model of bot behavor s based on observatons n [6]. At tme t, the hosts that exhbt malcous behavor are those that are part of the Infected category and are n the bot state Onlne. These are the hosts that are actve bots and follow the commands dctated by the botmaster. As dscussed earler, botmasters occasonally let ther botnets dle. Durng ths tme, even those bots of the botnet that are Onlne wll not perform any malcous actvty. For the remanng tme, botmasters command ther botnets to perform malcous actvtes shown n Table I (wth assocated frequences). The duraton (n mn) of each malcous actvty chosen s exponentally dstrbuted wth mean g. The malcous actvtes (or the nactvty) chosen by the botmasters of the dfferent botnets at a specfctmeare ndependent of one another, as are the duratons for whch the actvtes are executed. One of the actvtes that can be chosen by a botmaster s the scannng actvty. Ths s the actvty where bots of a botnet scan other computers to nfect them. When the botmaster of a botnet chooses the scannng actvty, each bot of that botnet that s Onlne sends an nfecton to the other hosts n the network wth probablty r. Hosts that are sent an nfecton by the scannng hosts can belong to any category (Infected, Cured, or Susceptble). Infecton attempts on hosts that are n the Infected or the Cured category have no effect, that s, the hosts n the Infected category mantan ther botnet label, whereas the hosts n Cured category (a) (b) Fgure 6. (a) The transtons of a host n a network between the dfferent states related to nfecton n the botnet model, (b) transton of an nfected host between the bot states Onlne and Offlne. Symbol N B t I(t) S(t) C(t) a b g r l Table IV. Symbols n botnet model. Stands for Number of hosts n a network Number of botnets n the network The dscrete tmeslot under consderaton Number of Infected hosts at tme t Number of Susceptble hosts at tme t Number of Cured hosts at tme t Mean tme a bot stays Onlne Mean tme a bot stays Offlne Mean tme a malcous actvty s performed The probablty of nfectng a new host The probablty of an nfected host becomng dsnfected

13 P. Jakumar and A.C. Kak A graph-theoretc framework for solatng botnets n a network 6. EXPERIMENTS AND RESULTS Our goal n ths secton s to provde expermental and smulated valdaton for the proposed framework for botnet detecton. We wll show results on two real network traffc traces and one smulated trace that s based on the new unfed botnet model presented n Secton Experment 1 The frst network trace that we show our results on conssts of a montored network of 15 hosts. There are two IRC botnets n the network, each wth three hosts. Hosts labeled 1 through 3 are part of botnet 1 and those labeled 4 through 6 are part of botnet 2. All the computers are nvolved n regular Internet communcaton wth actvtes ncludng P2P downloads, HTTP web communcaton, Fle Transfer Protocol, Telnet, and so on. The malcous hosts are nvolved n C&C communcatons wth the IRC server, DDoS attacks, and port scans. There s sgnfcant overlap n tme when both botnets are nvolved n the same actvtes. Addtonally, there are no unque actvtes that can be used to segment out the ndvdual botnets. We montored the network contnuously for 24 h, durng whch tme we detected malcous actvtes. Port scans and attacks beng performed by the malcous hosts n the network were detected usng Snort and patterns of faled connecton rates. Durng ths tme, we contnuously update the edges between the nodes n the graph (each node corresponds to a malcous host n the network) by usng Equaton (1). The resultng edge weghts between the nodes (correspondng to hosts 1 through 6 n the network) are shown n Table V. As can be seen n Table V, the edge weghts between the nodes belongng to the same botnet are hgh, whereas the edge weghts between the nodes belongng to dfferent botnets are low. For nstance, the edge weght between the nodes 1 and 3 (whch belong to the same botnet) s.94, whereas the edge weght between the nodes 1 and 4 (whch belong to dfferent botnets) s Ths updated edge weght matrx s then parttoned to dentfy the botnets n the network. Table VI shows the recursve steps n the graph-parttonng algorthm. Both the Ncut and the Cut Stablty thresholds were set to.7. The graph s recursvely parttoned as long as the Ncut value of the new partton s lesser than the set Ncut threshold and the cut s stable (t acheves a stablty value lower than the Cut Stablty threshold). The values of the two crtera that do not meet the thresholds are underlned. For ths real network trace, our approach correctly dentfes both the bots n each botnet and the number of botnets n the network Experment 2 We valdate the performance of our approach on the botnet datasets generated on the bass of the botnet model dscussed n Secton 5. We test our botnet detecton approach n a wde varety of stuatons dfferng numbers of botnets n the network, varyng actvty levels, and varyng growth patterns of botnets. All the experments were run on a network of 2 hosts. The varatons n our datasets are presented n the followng text Varaton n the number of hosts n dfferent categores. Experments were conducted n whch the number of hosts n dfferent categores (Infected, Susceptble, and Cured) exhbt dfferent patterns over the course of a smulaton, such as those shown n 7a and 7b. These patterns depend on the number of ntal nfected hosts (I()) and the number of botnets n the network. The value of I() was vared from 3 to 18. The number of botnets n the networks also plays a maor role n ths experment. In Fgure 7a, there are 2 botnets n the network that all compete for the susceptble hosts; as a result of whch, the number of nfected hosts remans hgh. On the other hand, Fgure 7b wth ust three botnets shows a greater avalablty of susceptble hosts Varaton n the number of botnets. The ntal number of botnets n the network (B n Table IV) s vared smulatons are repeated wth 3, 5, 9, and 2 ntal botnets. Havng dfferng numbers of ntal botnets creates dfferent types of condtons for the graph-parttonng algorthm for botnet detecton. When there are a small number of botnets, say ust three, t s less lkely that any botnet wll be completely elmnated from the network because botnets wll be able to acqure new bots at a rate that s comparable wth the rate at whch bots of the botnet become cured. However, havng a large number of botnets, for example 2, would lead to the elmnaton of some botnets. Fgure 8 shows ths varaton as observed n our datasets wth 2, 9, and 3 ntal botnets. We are partcularly Table V. Edge weghts between malcous nodes n Experment 1. Node # Table VI. Steps n graph parttonng. Nodes to partton {1, 2, 3, 4, 5, 6} {1, 2, 3} {4, 5, 6} Partton 1 {1, 2, 3} Partton 2 {4,5,6} Ncut value Stablty value

14 A graph-theoretc framework for solatng botnets n a network P. Jakumar and A.C. Kak Number of hosts Number of hosts Infected hosts Susceptble hosts Cured hosts Tme Infected hosts Susceptble hosts Cured hosts (a) Tme (b) Fgure 7. Varaton of hosts n dfferent categores (a) for I() = 8 and B = 2, and (b) for I() = 18 and B =3. nterested n the varaton of the number of botnets n the network, as t can be drectly used to measure the accuracy of our graph-parttonng algorthm. Addtonally, as was shown n Fgure 7, havng dfferng numbers of botnets n smulatons also causes the number of hosts n the dfferent categores to exhbt varyng patterns Varaton n botnet szes. Botnet szes vary rapdly over tme as new hosts are nfected and exstng nfected hosts become dsnfected. Fgure 9 shows the varaton n the szes of four botnets n a smulaton wth 2 ntal botnets. Ths rapd varaton s a challenge for botnet detecton, as new nfected hosts have to be added to the graph, actvty co-occurrences have to be establshed, and accurate botnet labels have to be determned, whle older nfected hosts that have snce been cured are removed Varaton n onlne footprnt. Fnally, we also vared the amount of tme a bot stays n the Onlne state to measure the mpact on detecton performance. We vared p on from.1 to.9, that s, from a bot beng actve 1% 9% of the tme. When bots are Onlne Number of botnets n the network Number of botnets n the network Number of botnets n the network Tme (a) Tme (b) Tme (c) Fgure 8. Varaton n the number of botnets n the network over tme for (a) 2 ntal botnets, (b) 9 ntal botnets, and (c) 3 ntal botnets. only brefly t becomes more dffcult to establsh strong temporal co-occurrences between the actvtes at the dfferent hosts. Addtonally, t s tougher to segment out

15 P. Jakumar and A.C. Kak A graph-theoretc framework for solatng botnets n a network Number of bots n each botnet 1 Bots n Botnet 1 9 Bots n Botnet 2 Bots n Botnet 3 8 Bots n Botnet Tme Fgure 9. Varaton n the number of bots n four botnets n a smulaton wth 2 ntal botnets, I() = 18, p on =.5. Botnet 1 (from the legend) s shown gettng elmnated as all ts bots have been cured, and t has been unable to obtan new bots n the meantme. the complete botnet as dfferent subsets may be onlne at dfferent ponts n tme Detecton accuracy. Tables VII and VIII summarze the parameters used for our smulatons. As shown n Table VII, smulatons are run wth varyng numbers of ntal botnets, dfferng numbers of ntal nfected hosts, and varyng actvty levels of bots n the botnet (controlled by p on ). Bots n each of the botnets ndependently transton between Onlne and Offlne states, and the percentage of tme they stay Onlne s controlled by the actvty level. As n Fgure 9, botnet szes vary rapdly. Table VIII summarzes the parameters common to all the experments. Fgure 1 shows the detecton accuracy of our approach on the datasets generated from these smulatons. We measure the accuracy of assgnng the correct botnet label to each malcous host n the network by comparng the assgned botnet label wth the ground truth. In case the number of botnets estmated by the graph-parttonng algorthm dffers from the true number of botnets at the tme, we classfy the bggest chunk of a botnet as correctly classfed, whereas the rest of the botnet s deemed ms-classfed. The detecton accuracy of our approach n Fgure 1 s compared for dfferent actvty levels of the bots wth dfferent numbers of botnets n the network. Each pont Table VII. Parameters for Experment 2. Number of botnets I() p on 3 3, 72, 18.1,.3,.5,.7, , 76, 18.1,.3,.5,.7,.9 9 4, 7, 18.1,.3,.5,.7,.9 2 4, 6, 18.1,.3,.5,.7,.9 Table VIII. Common parameters for Experment 2. Common parameters Values N 2 g 1 r.1 l.1 p error.1 s the average of 15 smulaton runs, and results obtaned for dfferng number of ntal nfected hosts are averaged. As seen n Fgure 1, the average detecton rate of our approach s close to 9% for actvty levels above 1% (.e., bots are actve more than 1% of the tme). The approach assgns the correct botnet label to 95% or more of the bots n the network for the values of p on between.5 and.9. We beleve that ths s mpressve, consderng that the number of botnets s contnually varyng, the number of bots wthn each botnet are varyng rapdly, dfferent subsets are onlne at dfferent ponts n tme, and all the botnets are nvolved n actvtes derved from the same underlyng botnet actvty dstrbuton. The standard devaton of our approach s less than 3% for over 15 runs of smulaton for each pont, ndcatng that the framework consstently produces the same results. The performance of our graph-based framework ncreases steadly wth the ncrease n the actvty levels of the ndvdual bots. Ths s to be expected because hgher actvty levels gve us more nformaton to work wth. However, there s a dp n the performance for the case when there are 2 botnets and the bots are onlne 9% of the tme. Such hgh levels of actvty lead to a substantally ncreased detecton of co-occurrences between the dfferent botnets, to the extent that t s dffcult to segment them out. Detecton accuracy botnets 5 botnets 55 9 botnets 2 botnets Percentage of tme bots are Onlne Fgure 1. Detecton accuracy for smulatons wth dfferent numbers of ntal botnets and varyng actvty levels of bots n the botnets.

16 A graph-theoretc framework for solatng botnets n a network P. Jakumar and A.C. Kak Detecton Accuracy Detecton Accuracy Detecton Accuracy Detecton Accuracy % actvty 7% actvty 5% actvty 3% actvty 1% actvty Tme (a) Detecton Accuracy: 3 botnets % actvty 7% actvty 5% actvty 3% actvty 1% actvty Tme (c) Detecton Accuracy: 5 botnets Tme (e) Detecton Accuracy: 9 botnets % actvty 7% actvty 5% actvty 3% actvty 1% actvty 9% actvty 7% actvty 5% actvty 3% actvty 1% actvty Tme (g) Detecton Accuracy: 2 botnets Number of botnets n trace Number of botnets n trace Number of botnets n trace Number of botnets Botnets detected Actual number of botnets Tme (b) 7% actvty: 3 botnets Botnets detected Actual number of botnets Tme (d) 7% actvty: 5 botnets Botnets detected Actual number of botnets n trace Tme (f) 7 % actvty: 9 botnets Botnets detected Actual number of botnets Tme (h) 7% actvty: 2 botnets Fgure 11. (a, c, e, and g) Detecton results of contnuous montorng on a network wth dfferent numbers of ntal botnets. (b, d, f, and h) Comparson of the true versus the estmated number of botnets over tme.

17 P. Jakumar and A.C. Kak A graph-theoretc framework for solatng botnets n a network For the case of 1% actvty, the average performance s around 72%. Ths reduced performance occurs because t s dffcult to segment out complete botnets at low actvty levels. At such actvty levels, not only are the dfferent bots onlne at dfferent ponts n tme but also each bot s onlne only very brefly, whch leads to over segmentaton of the botnet. Addtonally, there s 1% erroneous detecton, whch makes accurate co-occurrences more dffcult to establsh. Gven these constrants, these results are acceptable Contnuous detecton. Fgure 11 shows the results of usng our graph-based framework for the case when a network s montored contnuously for the detecton of botnets, and we have varyng numbers of botnets and varyng actvty levels for each botnet. The ntal number of nfected hosts was 18 n each case. Note that the results shown are not averaged. That s, each result shown s obtaned from a sngle experment. We detect and segment botnets every 1 unts of tme. As can be seen from the contnuous detecton results, the average performance s around 95% for actvty levels between 5% and 9%, around 9% for actvty levels of 3%, and around 7% for actvty levels of 1%. We also show a comparson between the actual and the estmated number of botnets n the network for actvty levels of 7% for all the experments. The dfference between them s never more than 1 at almost all nstances, except at the frst segmentaton nstance for the network of 2 botnets. The proposed approach s effcent computatonally. On an Intel 64-bt machne wth 8 cores and 6 GB of random access memory, a sngle sequental network-wde edge update (.e., the edge between every par of hosts s updated once, and ths process s repeated sequentally for all such edges) for a network of 2 hosts took only a couple of seconds. Ths s well under the duraton of a tmeslot. Recursve graph parttonng on the same szed network for 2 botnets took between 5 and 1 s. The algorthm was mplemented wth a mxture of Matlab, Perl, and C/C Experment 3 In our fnal experment, we show the results of our approach on the nfecton profles generated by a well-known bot detecton tool called BotHunter, a tool developed and mantaned by SRI Internatonal [21] ( com/). BotHunter detects the presence of a bot nfecton on a gven host by passvely montorng the two-way communcaton between the host and the Internet. It then uses a set of predefned rules to declare a bot nfecton on the bass of dfferent combnatons of malcous actvtes seen. When enough evdence for reportng a bot nfecton has been gathered, an nfecton profle for the host s generated. Each nfecton profle contans nformaton on several actvtes consdered potentally malcous: Inbound Explot, Egg Download, C&C Traffc, Outbound Scan, Outbound Attack, and so on. Dataset Table IX. Datasets used for Experment 3. No. of malcous hosts For our experment, we used three dfferent sets of nfecton profles, each generated over a perod of 24 h. On the bass of the actvty co-occurrences n tme, we manually grouped the bots nvolved nto consttuent botnets. Table IX descrbes the number of botnets and bots wthn each botnet n the three datasets. The datasets have varyng numbers of botnets and wdely varyng numbers of bots n each botnet. They also have consderable overlap n the actvty space. Fgure 12 provdes a feel for the datasets. Fgure 12a shows the actvtes that botnets n the Dataset 3 are nvolved n. The strong overlap n actvty space ndcates that there are no dstnct actvtes that can be used to segregate one botnet from another. Fgure 12b shows the overlap n one actvty, Egg Download, n the same dataset. The overlap n actvty observed n Fgure 12b s seen n the other actvtes n Dataset 3 too. Datasets 1 and 2 show smlar patterns. Table X shows the fnal results obtaned usng our approach on the bot nfecton profles generated by BotHunter. The Ncut threshold was set to.98, whch s hgher than the thresholds used for Experments 1 and 2. Ths s because BotHunter generates more succnct nfecton profles, and we expect a hgher degree of actvty cooccurrences at the bots that belong to the same botnet. The results ndcate that our approach can segment out the botnets wth a hgh degree of accuracy. All the errors n ms-classfcaton are from malcous hosts that have exhbted very low levels of actvty, whch makes t dffcult to establsh temporal co-occurrences wth the actvtes at the other hosts. 7. LIMITATIONS No. of botnets Bots n each botnet , 2, 23, 6, , 19, 33, 3, , 8, 5, 7, 2, 3, 2 As wth all network securty solutons, the proposed framework has some lmtatons, whch are dscussed here. Our framework s based on the assumpton that a botnet nfecton has resulted n the hosts exhbtng temporally cooccurrng malcous actvtes. Ths s based on observatons of botnet actvtes and ther communcaton patterns n real networks [3,6]. If the bots are desgned to engage n ther pre-programmed behavors n a random manner, our graph-based method as presented here wll not work. But when the computers n a botnet do not act n any concerted and coordnated behavor, can the nfected hosts stll be consdered to belong to a botnet? It s perhaps safe to assume that the bots behavng randomly and chaotcally to avod detecton wll also be of sgnfcantly dmnshed utlty to a botmaster. Addtonally, f there s no temporal co-occurrence n the malcous actvtes between the

18 A graph-theoretc framework for solatng botnets n a network P. Jakumar and A.C. Kak only one malcous actvty at any gven tme. Wth the ncrease n computng power/cores n the machnes of today, ths assumpton may no longer hold. However, t would be relatvely straghtforward to extend our approach to the case that allows for hosts to exhbt multple malcous actvtes n any gven tme slot. To explan, assume that for a gven edge, the node at one end exhbts a sngle actvty n a gven tme slot but the node at the other end exhbts m actvtes smultaneously n the same tme slot. When we get to updatng the weght of ths edge, nstead of usng ust a sngle update of the sort descrbed n our paper, we wll now perform m such updates, one for each of the multple actvtes seen. Ths can be further extended to the case when both ends exhbt multple actvtes. (a) (b) Fgure 12. (a) Actvtes exhbted by botnets n Dataset 3, and (b) overlap n Egg Download n the same dataset. Dataset Table X. Results on BotHunter traces. Botnets n trace Botnets dentfed No. of malcous hosts No. msclassfed Accuracy (%) dfferent hosts, t wll be mpossble for the nfected hosts to mount, say, a DDoS attack on a target. Another lmtaton of the algorthm as currently formulated s that we allow an nfected host to engage n 8. CONCLUSIONS In ths paper, we presented a graph-based framework for solatng botnets n a network. Our framework uses temporal co-occurrences n the actvty space to detect botnets. Ths makes the framework ndependent of the software archtecture of the malware nfectng the hosts. Another advantage of our approach s that the newly nfected hosts are automatcally ncorporated n the graph representaton of the nfected hosts that s mantaned by our framework. By the same token, dsnfected hosts are automatcally dropped from the representaton. The proposed framework was valdated by applyng t to a smulated botnet, wth the smulaton based on a new model of botnets that unfes the current best understandng of the growth and decay of botnets, our best knowledge of the malcous actvtes they engage n, the varous tme constants assocated wth ther actvtes, and so on. We also showed results on real traces generated by usng real bot detecton tools n nfected networks. On the strength of our valdaton, we beleve that t s reasonable to conclude that one can use a graph-based framework, such as the one descrbed here, to buld an effectve network-montorng tool for the purpose of detectng and solatng botnets n a network. REFERENCES 1. Gu G, Perdsc R, Zhang J, Lee W. Botmner: clusterng analyss of network traffc for protocol- and structure-ndependent botnet detecton. Proceedngs of USENIX Securty Symposum 28, 28; Gu G, Zhang J, Lee W. BotSnffer: detectng botnet command and control channels n network traffc. Proceedngs of the 15th Annual Network and Dstrbuted System Securty Symposum (NDSS 8), 28.

19 P. Jakumar and A.C. Kak A graph-theoretc framework for solatng botnets n a network 4. Zhao Y, Xe Y, Yu F, et al. BotGraph: large scale spammng botnet detecton. Proceedngs of the 6th USENIX Symposum on Networked Systems Desgn and Implementaton, USENIX Assocaton, 29; Strnghn G, Holz T, Stone-Gross B, Kruegel C, Vgna G. BotMagnfer: locatng spambots on the Internet. USENIX Securty Symposum, Raab MA, Zarfoss J, Monrose F, Terzs A. A multfaceted approach to understandng the botnet phenomenon. IMC 6: Proceedngs of the 6th ACM SIGCOMM on Internet Measurement, 26; Rutenbeek EV, Sanders WH. Modelng peer-to-peer botnets. Proceedngs of the 28 Ffth Internatonal Conference on Quanttatve Evaluaton of Systems, 28; Dagon D, Gu G, Lee CP, Lee W. A taxonomy of botnet structures. In Proc. of the 23 Annual Computer Securty Applcatons Conference (ACSAC 7), Km J, Radhakrshnan S, Dhall S. Measurement and analyss of worm propagaton on nternet network topology. Computer Communcatons and Networks, 24. ICCCN 24. Proceedngs. 13th Internatonal Conference on, 24; Stone-Gross B, Cova M, Cavallaro L, et al. Your botnet s my botnet: analyss of a botnet takeover. ACM Conference on Computer and Communcatons Securty (CCS), Holz T, Stener M, Dahl F, Bersack E, Frelng F. Measurements and mtgaton of peer-to-peer-based botnets: a case study on storm worm. LEET 8: Proceedngs of the 1st Usenx Workshop on Large-Scale Explots and Emergent Threats, 28; Frelng FC, Holz T, Wchersk G. Botnet trackng: explorng a root-cause methodology to prevent dstrbuted denal-of-servce attacks. ESORICS, 25; Baley M, Cooke E, Jahanan F, Xu Y, Karr M. A survey of botnet technology and defenses. Conference for Homeland Securty, Cybersecurty Applcatons and Technology 29; : Wang Y, Chakrabart D, Wang C, Faloutsos C. Epdemc spreadng n real networks: an egenvalue vewpont. In SRDS, 23; Stanford S, Paxson V, Weaver N. How to wn the Internet n your spare tme. Proc. 11th USENIX Securty Symposum, San Francsco, CA, Goebel J, Holz T. Rsh: dentfy bot contamnated host by IRC nckname evaluaton. Proceedngs of the 1st Workshop on Hot Topcs n Understandng Botnets (HotBots), Mazzarello C. IRC traffc analyss for botnet detecton. Informaton Assurance and Securty, Internatonal Symposum on, vol. ] 28; Huang Z, Zeng X, Lu Y. Detectng and blockng P2P botnets through contact tracng chans. Internatonal Journal of Internet Protocol Technology Aprl 21; 5: Nagaraa S, Mttal P, Hong Cy, Caesar M, Borsov N. BotGrep: fndng P2P bots wth structured graph analyss. USENIX Securty Symposum, Cho H, Lee H, Lee H, Km H. Botnet detecton by montorng group actvtes n DNS traffc. CIT 7: Proceedngs of the 7th IEEE Internatonal Conference on Computer and Informaton Technology, 27; Gu G, Porras P, Yegneswaran V, Fong M, Lee W. Bothunter: detectng malware nfecton through IDSdrven dalog correlaton. SS 7: Proceedngs of 16th USENIX Securty Symposum on USENIX Securty Symposum, 27; Baley M, Oberhede J, Andersen J, Mao ZM, Jahanan F, Nazaro J. Automated classfcaton and analyss of nternet malware. Proceedngs of the 1th Internatonal Conference on Recent Advances n Intruson Detecton, RAID 7, 27; Lu W, Tavallaee M, Ghorban AA. Automatc dscovery of botnet communtes on large-scale communcaton networks. ASIACCS 9: Proceedngs of the 4th Internatonal Symposum on Informaton, Computer, and Communcatons Securty, 29; Dagon D, Zou CC, Lee W. Modelng botnet propagaton usng tme zones. NDSS, Hao S, Syed NA, Feamster N, Gray AG, Krasser S. Detectng spammers wth snare: spato-temporal network-level automatc reputaton engne. Proceedngs of the 18th Conference on USENIX Securty Symposum, SSYM 9, 29; Sh J, Malk J. Normalzed cuts and mage segmentaton. IEEE Transactons on Pattern Analyss and Machne Intellgence 2; 22(8):