A Review on IRC Botnet Detection and Defence

A Review on IRC Botnet Detection and Defence Bernhard Waldecker St. Poelten University of Applied Sciences, Austria Bachelor programme: IT-Security 1 Introduction Nowadays botnets pose an enormous security threat to our networked society. Spam E-Mails, Distributed Denial-of-Service (DDOS) attacks or identity thefts are examples for problems caused by botnets. In their yearly published paper The IT-Security Situation in Germany in 2007 the German Federation for Information Security mentions that the sizes of botnets are scaling down, but the number of botnets steadily increases (cf. German Federal Office for Information Security 2007, p 28). One possible reason for this development arises from the fact that smaller botnets are more difficult to detect and consequently harder to combat. IT security professionals are aware of the problems emerging from botnets leading to intensive research effort to develop new approaches to detect and fight botnets. When it comes to detection approaches, the communication architecture plays an important role. To the current state of research the architecture of botnets can be classified by their communication structure between the Command & Control (C&C) server and the infected system (cf. Holz 2009, pp 4f.): centralized architecture: Internet Relay Chat (IRC) centralized architecture: Hypertext Transfer Protocol (HTTP) decentralized architecture: Peer-to-Peer (P2P). The major attention of this paper lies on the centralized architecture Internet Relay Chat. The Internet Relay Chat Protocol is standardised through the Internet Engineering Task Force (IETF), for example in the Request for Comments (RFC) like (1) RFC 2810 (Kalt 2000) and (2) RFC 1459 (Oikarinen & Reed 1993). Generally, the IRC protocol offers the possibility to communicate with other people across special IRC channels. The aim of this paper is to introduce the state of the art IRC botnet detection methods, the possibilities to defeat and how the botmasters can infiltrate the diverse detection methods. Lifecycle of an IRC botnet Generally, the lifecycle of a botnet (see Figure 1) consists of seven steps. The first step is that the botmaster has to find new possible targets. Then the botmaster tries to infect the system and after a successful exploitation, the system downloads the dedicated botnet binaries from one of the botnet servers (cf. Lu & Ghorbani 2008, p 1). After that, each bot on the botnet will attempt to find the IRC server address by DNS query [...] Next is the communication step between bots and IRC server. In IRC based communication mechanism, a bot first sends a PASS message to the IRC server to start a session and then the server authenticates the bot by checking its password. In many cases, the botmaster also needs to authenticate itself to the IRC server. Upon the completion of these authentications, the command and control channels among botmaster, bots, and IRC server will be established. To start a DDoS attack, the botmaster only needs to send a simple command like ''.ddos.start victim\_ip'' while all bots receive this command and start to attack the victim server (Lu & Ghorbani 2008, p 1) 1/9

Figure 1 Lifecycle of a botnet (Lu & Ghorbani 2008, p 1) 2 A State of the Art Review on Botnet Detection and Defence This section is dedicated to state-of-the-art techniques to detect and fight botnets. The remainder of this section is structured as follows: The first two subsections introduce different approaches to detect botnets. There are different approaches to detect IRC botnets, the approaches can be classified as follows: host-based procedures: detection of possible anomalies or modification of the file system network-based procedures: inspection of the network traffic respective to anomalies or specific singularities of the IRC protocol which can be a hint for a botnet combined procedures: the combination of host and network based methods. Subsequently, the subsection Botnet Defence outlines approaches that currently try to fight botnets. The last subsection Infiltration of the detection methods outlines the current methods with which methods botmasters could undercut the different detection methods are declared. 2.1 Botnet detection: Host-based procedures Host-based procedures try to detect possible anomalies or modifications of the host system. One method for such a host-based method is Antivirus (AV) software. One detection approach of AV software is the use of signature to detect malicious software and resembles a reactive approach. This means that there has to be an allocation for each malicious software in the signature database. The signatures are also the weakness of this procedure, because if the botmaster modifies the source code the signature that is allocating to this specific bot does not match anymore (cf. Wurzinger et al. 2009, p 1). Another disadvantage of the signature approach is that if there is no signature for a bot, the AV is not able to detect the malicious code. The conclusion that can be drawn out of these weaknesses is that if the size of the botnet is marginal, the probability that the Antivirus companies create a signature that could detect the bot binary is small (cf. German Federal Office for Information Security 2009, p 20). As a result of the diverse weaknesses another approach in host-based detection is the static of dynamic analysis of unknown software was developed. This means that the unknown software is 2/9

analysed according to its behaviour. However, for the additional step the software has to be installed on every system, thereby the analysis creates an overhead, consequently the system becomes duller (cf. Wurzinger et al. 2009, p 1). Another method for host-based procedures is the detection of new unintentional, modified or deleted files, new installed software or modifications in the Microsoft Windows Registry. This method is also used in honeypots to collect and analyse malware binaries (cf. Zhuge et al. 2007, p 4). By analysies of the collected malware binaries, it is possible to extract IRC relevant data such as IRC username, the IRC channel, DNS or IP addresses. 2.2 Botnet detection: Network-based procedures The detection of botnets using network-based procedures is mainly predicated on checking the network traffic for anomalies or specific singularities of the IRC protocol. The network traffic can be checked during the transfer or can be recorded and analysed at a later date. However, the diverse methods have not to prove every protocol, because there are only two relevant protocols, the Transmission Control Protocol (TCP) and the Internet Relay Chat protocol (IRC) (cf. Strayer et al. 2006, p 3). Vertical Correlation Vertical correlation means that the network-based detection focuses on individual or single bot infections. The referring software checks the network traffic with precast patterns for communications between the infected system and the C&C server or other relevant activities (cf. Wurzinger et al. 2009, p 1). This procedure has the disadvantage as the signature-based procedure, because without patterns or signatures the botnet traffic cannot be detected. Horizontal Correlation In contrast to the vertical correlation, the horizontal correlation tries to detect two or more infected systems in the network. The detection mechanism searches for analogies in the network traffic, for instance the same C&C server (cf. Wurzinger et al. 2009, p 1). The key problem with this explanation is that individual or different bots cannot be detected inside a network, because there is no affinity between at least two bots. Therefore, two different bots are able to remain unnoticed in the network. Anomaly detection procedure An anomaly in the network traffic is a variance or a special abnormity in comparison with the common network traffic. One procedure is to check the network traffic for high capacity utilisation, because this utilisation can have three reasons (cf. Binkley & Singh 2006, p 44): network scans lacking of servers Peer-to-Peer (P2P) applications. Binkley and Singh (2006) describe in their paper An Algorithm for Anomaly-based Botnet Detection that one infected host which performs a network scan is not an anomaly. However, if there are many hosts performing a network scan and they are in the same IRC channel this phenomenon is an abnormality compared to the common network traffic (cf. Binkley & Singh 2006, p 44). Another anomaly detection method is the comparison of sent and received e-mails. If more mails have been sent than received, it indicates for a potential spambot infection (cf. Abu Hamed Mohammad Misbah Uddin 2009, p 5) A specific IRC anomaly detection procedure is the measurement of the IRC response time. A human is not able to respond as fast as malicious software (cf. Lu, Tavallaee & Ghorbani 2009, p 74). Consequently, it is possible to compare all the response times. The botmaster tries to contact the infected systems in order to synchronise data or to check if the system is already up. The following two figures show this behaviour: 3/9

Figure 2 Average byte frequency over 256 ASCIIs for normal IRC flow (Lu, Tavallaee & Ghorbani 2009, p 74) Figure 3 Average byte frequency over 256 ASCIIs for botnet IRC flow (Lu, Tavallaee & Ghorbani 2009, p 74) IRC specific detection procedures All IRC Bots have in common that they have to receive commands from the C&C server or have to send messages, for example for synchronisation, to the C&C server. The standardisation of the IRC (Kalt 2000, Oikarinen & Reed 1993) protocol regulates also the recommended commands that should be used, for instance NICK, JOIN, USER, QUIT or MODE. Above all the selected nickname has to be unique in the IRC channels. The botmaster solves the problem for the unique nicknames with a trick. The nickname consists of two parts, a static and a dynamic part. The static part is commonly the name of the virus, trojaner or the country code and the dynamic one is a random number or letter combination (cf. Goebel & Holz 2007, p 5). Therefore, different methods are developed which try to detect automatically the noticeable IRC traffic. Examples for such applications are Rishi (Goebel & Holz 2007) or the diverse Intrusion Detection/Prevention Systems like Snort (Hanna 2004). The Intrusion Detection/Prevention Systems scan the network traffic with the help of predefined samples and so it is possible to extract the relevant data. Other methods like Rishi try to scan for IRC data like the nickname, extract the other relevant data like source and destination IP and save it in a so called connection object. Af- 4/9

terwards the data will be analysed and evaluated. In the following figure the concept of Rishi is presented: Figure 4 Basic concept of RISHI (Goebel & Holz 2007, p 5) 2.3 Botnet defence After the explanation of the diverse detection procedures, the question is how botnets could be combatted? Therefore, the first step is to look at the size of the network, because the defence in a local area network (LAN) is different to a wide area network (WAN). In a LAN, the responsible administrators can recover the affected systems with backups, can reinstall the operating system, or can harden the systems. In the WAN, the defence is different and more complex. Across the constraints of a LAN there has to be a good organized cooperation between the countries, the internet service provider (ISP) and the companies. One organizational possibility involves the adaptation and accordingly the modification of national and international laws, for instance within the EU (cf. Barroso 2007, p 6). Auxiliary the legitimate constraints of another countermeasure against botnets can be a better cooperation between the different national law enforcement agencies and private companies, for example working for a better dialogue and helping each other to detect, prevent and react to botnet incidents. Government Computer Emergency Response Teams (CERTs) are a valuable first point of contact, perhaps with ENISA acting as an additional focal point for long-term co-ordination and the sharing of best practice. (Barroso 2007, p 9). Internet service provider can check their network for noticeable traffic and can explore the IP addresses of the C&C servers or the infected systems. Consequently, the ISP can inform the affected customers to clean their system and the C&C server can be destructed or can be added to a special Blacklist. Another organizational method is informing the people about the danger of botnets, viruses and so on. Thereby the user awareness will be become better and people learn about the importance of, for example, security patches, Antivirus software or Firewall software. For this purpose, the problem of possible infections can be decreased, due to reduction of with potential vulnerabilities. 5/9

2.4 Infiltration of the detection methods Due to the detection procedures, the botmasters developed different methods to infiltrate the methods and consequently stay unnoticed. The botmasters utilise the weakness of the detection approaches, which are not able to embrace all possible detection methods (cf. Stinson & Mitchell 2008, p 6). Stinson and Mitchell (2008) classified the infiltration methods based on the complexity of the implementation and of the modification: low: without source code modifications medium: source code modifications with the aid of a bot-development kid high: marginal source code modifications very high: complex source code modification, for instance the modification of the C&C protocol Tactic 1: Encrypt Traffic; level of difficulty: medium Concerning one tactic of the botmasters is encrypting the traffic resulting in a difficult decision, if it is allowed or botnet traffic. The implementation is not hard, because the encryption can be activated with a bot-development kit and there is no limitation of the attacking effectiveness. Tactic 2: Threshold Attack; level of difficulty: hard A widely unnoticed method is to modify the time interval of the network packages. Consequently, the attacking effectiveness becomes wearer, however, the network detection approaches have problems to detect the correlation the slower network packages. Tactic 3: Perturb Flows; level of difficulty: very hard This attack is related to the second tactic, because the time interval of the network packages is modified. Auxiliary the next step is to play it safe that the C&C is not the whole time online and so this step limits the attacking rate, because the infected systems are not able to receive instructions from the C&C the whole time. Against IRC specific detection procedures the botmasters developed the attitude to change the standardised IRC commands. Therefore, the detection procedures are not able to identify noticeable network traffic without additional encodings. Another tactic against these detection procedures is to use regular nicknames instead of a static and a dynamic part. The following figures compare the automatic detection procedures with the infiltration methods: Figure 5 Description of some botnet characteristics upon which automated detection methods rely (Stinson & Mitchell 2008, p 4) 6/9

Figure 6 Automated botnet detection methods (in chronological order) and some characteristics on (Stinson & Mitchell 2008, p 4) Figure 7 The surveyed methods and an optimal evasive tactic which could be used to defeat each as well (Stinson & Mitchell 2008, p 6) 3 Conclusion This paper reviews about the state of the art of IRC botnet detection and defence approaches. The different detection approaches can be classified into host-based, network-based and combined methods. Host-based detection approaches try to detect anomalies or modifications of the host system. One method is to use signatures for the detection of viruses, trojaner or other malware. Therefore, the signatures have to be up to date, otherwise new malicious software cannot be detected. Another method in host-based detection is to check the file system for anomalies or modifications of the host system. The network-based approaches try to find anomalies or special IRC commands in the network traffic. Network scans or the number of sent e-mails can be a hint for a possible bot infection. In addition, the response time of the IRC response can be measured, because the response time of malicious software is faster than the time of a human. There are also some IRC specific detection methods, which check the network traffic for the IRC commands like the nickname or which work with predefined samples. The organisational methods to combat botnets contain the modification or the adaption of national and international laws, a better cooperation between national law enforcement agencies and private companies and the fortification of the user awareness. Because of the developed detection approaches, the botmasters developed different tactics to stay further unnoticed. These tactics can be, for instance, encryption, the decrease of the time interval of the network packages or the modification of the IRC commands. 7/9

List of Figures Figure 1 Lifecycle of a botnet (Lu & Ghorbani 2008, p 1)... 2 Figure 2 Average byte frequency over 256 ASCIIs for normal IRC flow (Lu, Tavallaee & Ghorbani 2009, p 74)... 4 Figure 3 Average byte frequency over 256 ASCIIs for botnet IRC flow (Lu, Tavallaee & Ghorbani 2009, p 74)... 4 Figure 4 Basic concept of RISHI (Goebel & Holz 2007, p 5)... 5 Figure 5 Description of some botnet characteristics upon which automated detection methods rely (Stinson & Mitchell 2008, p 4)... 6 Figure 6 Automated botnet detection methods (in chronological order) and some characteristics on (Stinson & Mitchell 2008, p 4)... 7 Figure 7 The surveyed methods and an optimal evasive tactic which could be used to defeat each as well (Stinson & Mitchell 2008, p 6)... 7 References Abu Hamed Mohammad Misbah Uddin 2009, Detecting Botnets Based on Their Behaviors Perceived from Netflow Data. Barroso, D 2007, Botnets - The Silent Threat. Available from: http://www.enisa.europa.eu/act/it/oar/botnets/botnets-2013-the-silent-threat/. Binkley, JR & Singh, S 2006, 'An Algorithm for Anomaly-based Botnet Detection'. Proceedings of USENIX Steps to Reducing Unwanted Traffic on the Internet Workshop (SRUTI), pp. 43 48. German Federal Office for Information Security 2007, The IT-Security Situation in Germany 2007. Available from: https://www.bsi.bund.de/cae/servlet/contentblob/479286/publicationfile/30726/lagebericht2007_pd f.pdf. German Federal Office for Information Security 2009, The IT-Security Situation in Germany 2009. Available from: https://www.bsi.bund.de/cae/servlet/contentblob/476182/publicationfile/30725/lagebericht2009_p df.pdf. Goebel, J & Holz, T 2007, 'Rishi: Identify Bot Contaminated Hosts by IRC Nickname Evaluation'. HotBots 07: Proceedings of the first conference on First Workshop on Hot Topics in Understanding Botnets, USENIX Association, Berkeley, CA, USA. Hanna, CW 2004, Using Snort to Detect Rogue IRC Bot Programs. Holz, T 2009, Tracking and mitigation of malicious remote control networks. Mannheim, Univ., Diss., 2009. Kalt, C 2000, Request for Comments (RFC) 2810: Internet Relay Chat - Architecture. Lu, W & Ghorbani, AA 2008, 'Botnets Detection Based on IRC-Community'. Global Telecommunications Conference, 2008, IEEE Computer Society, pp. 1 5. Lu, W, Tavallaee, M & Ghorbani, AA 2009, 'Automatic Discovery of Botnet Communities on Large-Scale Communication Networks'. ASIACCS 09: Proceedings of the 4th International Symposium on Information Computer, and Communications Security, ACM, New York, NY, USA, pp. 1 10. Oikarinen, J & Reed, D 1993, Request for Comments (RFC) 1459: Internet Relay Chat. 8/9

Stinson, E & Mitchell, JC 2008, 'Towards Systematic Evaluation of the Evadability of Bot/Botnet Detection Methods'. WOOT 08: Proceedings of the 2nd conference on USENIX Workshop on offensive technologies, USENIX Association, Berkeley, CA, USA, pp. 1 9. Strayer, WT, Walsh, R, Livadas, C & Lapsley, D 2006, 'Detecting Botnets with Tight Command and Control'. Proceedings of the 31st IEEE Conference on Local Computer Networks (LCN), pp. 195 202. Wurzinger, P, Bilge, L, Holz, T, Jan Goebel and Christopher Kruegel & Kirda, E 2009, Automatically Generating Models for Botnet Detection TR-iSecLab-0609-001. Zhuge, J, Holz, T, Han, X, Guo, J & Zou, W 2007, Characterizing the IRC-based Botnet Phenomenon, Universität Mannheim / Institut für Informatik, Mannheim. 9/9