Implementation of Botcatch for Identifying Bot Infected Hosts

Implementation of Botcatch for Identifying Bot Infected Hosts GRADUATE PROJECT REPORT Submitted to the Faculty of The School of Engineering & Computing Sciences Texas A&M University-Corpus Christi Corpus Christi, Texas In Partial Fulfillment of the Requirements for the Degree of Master of Science in Computer Science By Divya ParamJyoti Andolu Spring 2015 Committee Members Dr. Mario A Garcia Committee Chairperson Dr. David Thomas Committee Member

ABSTRACT Many machines are being infected by botnets. A network of compromised hosts is called a botnet. Botmaster establishes control on the bots using a command and control server. The botmaster sends commands to the bots using the server, and the bots perform certain malicious activities in response to the command. They send the results of the activities performed back to the botmaster, closing the loop of the botnet. Attackers have complete control over the machines using botnets and can send spam emails and leak the personal information of individuals. In this project, a bot detection system to identify bot-infected hosts is implemented. The proposed system makes use of two facts: one is that all bots receive the same command from the botmaster and two is that all bots respond at the same time when the botmaster gives a command. The proposed system will be implemented using Botcatch application that will capture the bot traffic for the purpose of identifying if the system is compromised by a bot or not. 2

TABLE OF CONTENTS Abstract...2 Table of Contents...3 List of Figures...4 1. Background and Rationale...6 2. Narrative...11 2.1 Proposed System...11 2.2 IRC botnet...15 2.3 Client Server Architecture...16 2.4 mirc...17 3. Implementation and Results...18 4. Testing and Evaluation...31 5. Conclusion and Future work...39 Bibliography and References...40 Appendix...42 3

LIST OF FIGURES Figure 1. Centralized botnet...7 Figure 2. Distributed botnet...8 Figure 3. Hybrid botnet...9 Figure 4. Flowchart of Botcatch application...13 Figure 5. IRC botnet...15 Figure 6. A computer network diagram of clients communicating with server...16 Figure 7. mirc application...17 Figure 8. mirc server connection setup...20 Figure 9. Giving a nickname in the mirc application...21 Figure 10. Botmaster joins the channel...22 Figure 11. Bot 1 joins the channel...23 Figure 12. Bot 2 joins the channel...24 Figure 13. Capturing the hosts details...25 Figure 14. Capturing the time in the hosts system...26 Figure 15. Results of the status keyword...27 Figure 16. User interface of the Botcatch...28 Figure 17. Capturing the messages sent by the bot... 29 Figure 18.Normal traffic...30 Figure 19. Normal traffic...30 Figure 20. Bot 1 joins the channel...32 Figure 21. Bot 2 joins the channel...33 4

Figure 22. Bots displaying the host details...34 Figure 23. Response by the bot...35 Figure 24. Host details sent by bot captured by Botcatch...36 Figure 25. The Botcatch application captures all the messages sent by the bot...37 Figure 26. Bot traffic...38 Figure 24. Normal traffic...38 5

1. BACKGROUND AND RATIONALE A network of bot-infected hosts is a botnet. A bot is a malware that runs on the user s machine without the user s knowledge and captures sensitive information [11]. A botnet is operated by a botmaster also known as a bot herder controls the compromised machines from a remote location. The machines act like zombies once they are infected and communicate over the internet. The botmaster uses a command and control channel to maintain control over the bots. The botmaster commands the bots to perform malicious activities through the command and control channel; the bots respond to these commands by performing the activities and sending the results back to the botmaster. Based on the command and control channel used the botnets are classified into three types. They are: Centralized Distributed and Hybrid Centralized botnet In a centralized botnet [6] only one server acts as a command and control server. Based on the protocol used the centralized botnet are divided into two types. They are: HTTP botnet IRC botnet In an HTTP botnet, a web server is established by the botmaster and the bots poll to this 6

server to obtain commands from the botmaster. In an IRC-based botnet, the botmaster posts his command through a channel on the IRC server, and the bots join this channel in order to obtain commands from the botmaster. Figure 1: Centralized botnet [6] Distributed botnet Distributed botnets [6] use various peer to peer protocols to communicate with the bots. In this type of botnet, in order to find and the join the botnet, bootstrap procedure is used. Bootstrapping can be done in two ways: one is by using an initial peer list, and the other is by using rendezvous server hard-coded in each bot for obtaining the IP address to join a botnet. Distributed botnets do not depend on one command and control server like centralized botnets; instead, each bot connects to its peer bot and acts as both server and client. The main advantage of using this botnet is that even if some bots are detected the botmaster can control the botnet. 7

Figure 2: Distributed botnet [6] Hybrid botnet Hybrid botnets [6] are a combination of centralized and distributed botnets, and as a result it is difficult to discover and destroy the botnet. In Figure 3, there are two groups of bots, one is the servant bot, and the other is the worker bot. The servant bots have static non-private IP addresses, and they can act as both servers and clients. Worker bots have a private IP address, and they cannot accept incoming Internet connections as they are behind firewalls. 8

Figure 3: Hybrid botnet [6] Botnets are used to perform malicious activities such as launching distributed denial of service attacks, sending spam emails, generating click fraud attack, adware, and spyware attack. Many bots are found on the internet and most of the code that makes a botnet can also be found online. As most of the code is found on the web, botmasters can combine their code and launch a denial of service attack that can bring down networks and websites, and this can be a major problem. Different techniques have been employed to detect botnets. One such technique is BotMiner. BotMiner [2] is a botnet detection technique that divides the communication traffic and malicious traffic into two groups and then uses cross-cluster correlation to identify bot-infected hosts. This approach does not depend on command and control protocol. The advantage of using this approach is that it is accurate. The drawback of using BotMiner is that it cannot be used to detect botnets at an early stage. BotHunter [1] focuses on two aspects of the network. One is the coordination dialog and the other is identifying the infection. BotHunter recognizes these two aspects when the 9

malware is being infected. Then the BotHunter tracks down the two-way communication between the internal and external hosts, and then the data exchanges that match a state based infection model are extracted. BotOnus [6] is a botnet detection technique that uses a fixed width clustering algorithm. The advantage of using this method is that the detection rate is high. The disadvantage of BotOnus is that it has a high false alarm rate. BotCop [5] is a detection technique that is used for the classification of botnet communication traffic. This technique is helpful in detecting bots online. To classify the network traffic, the C4.5 decision tree algorithm is used. A hierarchical clustering algorithm is employed to figure out behaviors in an application community. BotGAD [4] is another technique for online botnet detection. By monitoring group activities in the DNS traffic, a metric was presented. The drawback of this approach is that this technique can be evaded by the botnet when it performs DNS queries. Another drawback of this approach is that it can only detect botnets that perform group activities in DNS traffic. Another technique was proposed by Castle and Buckley [3] to detect botnets that send spam messages. The email message headers were processed to produce synthetic headers. These headers were used to get a set of suspected botnet clusters through which the botinfected hosts were identified. The disadvantage of using this technique is that it can detect botnets that send only spam messages. It cannot be used to detect botnets at an early stage. 10

2. NARRATIVE The objective of the research is to implement a system that detect a bot that uses an IRC based command and control channel for communicating with the botmaster. Botcatch application will be used to identify if the host is bot-infected or not based on the analysis. The project is important because, with the increase in botnets, users are unaware that their systems have been compromised. Therefore, there is a need to implement a system that identifies a bot-infected host. 2.1 Proposed System In the project, Botcactch application is proposed for detecting a bot. For creating a bot that uses the IRC based command and control channel, the PircBot framework will be used. The time and host details of the system will be captured by the bot using this framework. In an IRC based command and control channel, the botmaster creates a channel by using the IRC server. The bots also join this channel using the IRC client for obtaining commands from the botmaster. It is a push-based method where the botmaster pushes commands to the bots and the bots respond to these commands. In the system, the IRC client is going to be mirc chat application and irc.foonode.net will be used as an IRC server. The bot will connect to the irc.foonode.net server and communicate with the botmaster using the mirc client application. The Botcatch application will be implemented which will capture the messages sent by the bot. 11

Botcatch is based on two facts [2], [14]. They are Bots in the same botnet receive the same commands from the botmaster. The bots perform the same malicious activities in response to the commands sent by the botmaster. These facts result in coordinated group activities. Be it two bots or three bots or even three hundred bots, as long as they are in the same botnet they behave in a similar manner. Figure 4 shows the flowchart of the Botcatch application. The network packets will be captured using the jpcap library and after selecting the network adaptor the botcatch application will be able to capture the packets sent and received. The messages sent by the bot messages will be identified, and the system will be identified as bot-infected after tracing the packet. 12

Figure 4: Flow chart of Botcatch application Start Capture the network traffic by initializing jpcap.dll Provide network adaptor of the local machine Capture packets sent and received Trace each packet received Identify the bot messages and the channel used Print the information in the Botcatch window End 13

Step by step procedure for project development In the first step, the mirc client will be installed. A bot will be created using the PircBot framework that will connect to the server and join a particular channel in the server using the IP address and the channel name. The botmaster will connect to the mirc application and will connect to the server and join a particular channel. When the botmaster enters the keyword host the host details of the system will be captured and sent to the botmaster by the bot. When the botmaster enters the keyword time the time on the machine where the bot resides will be captured and sent to the botmaster by the bot. Host details are critical information as this information can be used by the botmaster to access a system remotely or 1. Compromise confidential data 2. Delete important files 3. Obtain details of a bank account By using the Botcatch application, the bot messages will be captured, and the system will be identified as bot infected. 14

2.2 IRC botnet In an IRC-based botnet, the systems or machines get affected remotely by the botmaster. The affected machines install the code and connect to the web and search for the IRC server via querying DNS. A session is initiated by the bot when it passes a message to IRC server. It gets verified for its authenticity by its password. Figure 5: IRC botnet [13] The botmaster also authenticates itself to the IRC server. The command and control channels are established after verification. For launching, an attack, commands are sent to the bots by botmaster and the bots follow these commands and attack the victim s server. 15

2.3 Client Server Architecture Client-server architecture [8] focuses on a computer network that has many clients who request service from a centralized server. The client is the one that requests a service and the server is the one who offers the service Figure 6: A computer network diagram of clients communicating with server [10] The client computer makes use of an interface by which the user requests services of the server and results will be displayed once the server responds to the request. The servers provide a transparent interface so that the hardware and software details of the server are unknown to the client. 16

2.4 mirc mirc[9] is a scripting language that can be used to share, communicate or work with other IRC networks all over the world. It can be private discussions or group conferences. The mirc has features like file transfers, spoken messages, proxy support and many more. Various applications can be created that range from performing functions on networks to playing games using mirc. Figure 7: mirc application [9] 17

3. IMPLEMENTATION AND RESULTS It was not possible to create an IRC-based botnet using the university network because to create a network of compromised hosts that are controlled by the botmaster an IRC command and control server was required. All the bots were required to connect to the particular server to obtain commands from the botmaster. When using the university network the bots were unable to connect to the IRC server as packets with a request to the server port were dropped at the router. So the proposed system was implemented using a physical machine and a virtual machine. When two virtual machines were created, it affected the performance of the system. As there was not enough RAM available to create more virtual machines, only one virtual machine was created. In the real scenario, the botmaster or the attacker controls the bots remotely using an IRC command and control server (IRC server). In the project, the bot was planted on the virtual machine, and the botmaster was planted on the physical machine. The botmaster communicated with the bot using an irc.freenode.net which is an IRC server. For the purpose of detecting the bot, Botcatch application was used. The proposed system is an application that detects an IRC-based bot. The botcatch application is written in Java. It uses the jpcap library for capturing the bot traffic. The user interface of the Botcatch application is a window that contains three options. They are Start capture Save view Reset counter 18

Start capture The start capture option is selected for capturing the bot traffic. Whenever the bot communicates with the botmaster, all the messages sent by the bot will be captured. Save view All the information captured by the Botcatch can be saved to file using the save view option. Reset Counter The counter can be reset using this option. 19

First the botmaster connects to the mirc chat application (IRC client). In Figure 8, the botmaster connects to the server by entering the IRC server and the port number. Figure 8: mirc server connection setup 20

In Figure 9, for connecting to the server, a nickname has to be given. The botmaster gave the nickname master1. An alternative nickname can also be used. The optional fields are name and field, but the nickname is a mandatory field. Figure 9: Giving a nickname in the mirc application 21

After connecting to the server for joining the channel named #oper2 in the mirc client application the botmaster used the command /join #oper2. Figure 10 shows that the botmaster was able to join the channel using the command. Figure 10: Botmaster joins the channel 22

After the execution of the program, the bot was able to connect to the server and join the channel #oper2 by using the mirc application. In Figure 11, the bot named Bot 1 was able to connect to the #oper2 channel in which the master1 (botmaster) was already present. Figure 11: Bot1 joins the channel 23

When the program was executed the Bot2 on the virtual machine was able to connect to the server on the physical machine. Figure 12 shows that the bot was able to join the channel in which the master1 and Bot1 were present. Figure 12: Bot 2 joins the channel 24

When the botmaster issued the keyword host, the details of the host machine were captured and were displayed in the #oper2 channel by the bots. In Figure 13, the IP address, computer name, operating system and operating system version was captured by bots. The details of the host on the virtual machine were captured by Bot2. Figure 13: Capturing the host details 25

When the botmaster issued the keyword time the bots could display the time. In Figure 14, Bot 1 captured the time on the physical machine and Bot 2 captured the time on the virtual machine. Figure 14: Capturing the time in the hosts system 26

When the botmaster entered the keyword status both the bots displayed a message. In Figure 15, the bots responded to the keyword with the message successfully infected the host'. Figure 15: Results of the status keyword 27

The user interface of the Botcatch application consists of three options. The save view, start capture and the reset counter options. Figure 16 shows the user interface of the botcatch application. Figure 16: User interface of Botcatch application 28

After running the bot program, the messages sent by the bot were identified by the botcatch application. It contained information regarding the channel used by the bot, timestamp of the message and also the message content. Figure 17 shows the messages captured by the botcatch application. Figure 17: Capturing the messages sent by the bot 29

The main difference between bot traffic and normal traffic is that normal traffic contains normal messages like a conversation between two people whereas the bot traffic captures the system related information. Figure 18 and 19 show the normal traffic captured by the Botcatch application. It is a conversation between two people named tom123mmy and jimm123y.' Figure 18: Normal traffic Figure 19: Normal traffic 30

4. TESTING AND EVALUATION Software testing [12] makes sure that quality software is delivered to the client. There are many testing types available for testing a system. They are installation testing, compatibility testing, and usability testing. Installation testing For installation of the system on the client's hardware this type of testing is used. Compatibility testing It is used make sure that the software is fully compatible with the operating system if software failure occurs it could be due to incompatibility issues Usability testing It is used to make sure that the application is user-friendly, and the interface is easy to understand. Results The project made use of Botcatch application to identify bot-infected hosts. The Botcatch application was developed in Java, and it used the jpcap library for capturing the network traffic. PircBot framework was used to create bots that captured the host details and the time on the compromised machine. Whenever the bot communicated with the botmaster, all the bot traffic was captured by the Botcatch application. Botcatch application was able to detect the message content, channel used and also the time at which the messages were sent. 31

Test case 1 The Bot 1 was tested to determine if it could connect to the server and join the #oper2 channel in the mirc application. In the bot program the IP address and the channel #oper2 were included. By using these details, the bot was able to connect to the IRC server and join the channel. This can be seen in Figure 20. Figure 20: Bot 1 joins the channel 32

Test case 2 The bot running on the virtual machine was tested to check if it could join the channel in the mirc application. In the bot program the IP address and the channel #oper2 were included. In Figure 21 the bot was able to connect to the server and join the #oper2 channel using that information. Figure 21: Bot2 joins the channel 33

Test case 3 The bots were tested to check if they could capture the host details. In Figure 22, when the keyword host was entered by the master, the bots could capture and display the host details of the machine in the channel. Figure 22: Bots displaying the host details 34

Test case 4 The bots were tested to determine if they could give output to keywords other than host, time, status and hello. When the keyword hello was entered the message hello master was displayed by the bot. In Figure 23, when the keywords details, user, date were entered the bot did not respond to these commands. It was inferred that the bot responded only to the keywords host, time, status and hello. Figure 23: Response by the bot 35

Test case 5 The Botcatch application was tested to check if could capture the host details sent by the bots. In the figure, 24 shown below the botcatch application was able to capture the messages sent by the bots. Figure 24: Host details sent by the bot detected using Botcatch 36

Test case 6 The Botcatch application was tested to determine if it could capture all the messages sent by the bot. In the figure 25 the botcatch application was able to detect the messages sent by the bot along with the message content. The host details and the time which were captured by the bot were detected by the botcatch application. Figure 25: The Botcatch application captures all messages sent by the bot 37

Test case 7 A scenario was created where the physical machine was infected by a bot to generate bot traffic and the virtual machine was not infected so it could generate normal traffic. The botcatch application was tested if it could capture the messages. This is seen in Figure 26 and Figure 27. Figure 26: Bot traffic Figure 27: Normal traffic 38

5. CONCLUSION AND FUTURE WORK A bot is a type of malware that runs on a host unknown to its owner [14]. A network of bots is called a botnet. Bot detection has become a major issue in security, as most users are unaware that their systems have been compromised. Earlier many techniques were proposed for botnet detection, but they had drawbacks like lack of detection at any early stage, depending on a specific command and control channel and not being able to work offline. In the project, Botcatch technique was implemented for detecting bot-infected hosts. Botcatch[2] is based on the fact that bots within the same botnet behave in a similar manner and perform similar activities and, as a result, they exhibit coordinated behavior. In the future, the functionality of the Botcatch application can be extended to larger botnets where the normal traffic and bot traffic can be captured, and only the bot traffic can be filtered from the normal traffic. After filtering the bot traffic, two lists can be created one is the white list containing the normal traffic, and the other is the black list containing only the bot traffic. Instead of the user identifying the bot messages botcatch will automatically filter the bot traffic and place it in the black list by this functionality. Additional features can be added to the botcatch application that will be capable of detecting HTTP and also peer-topeer botnets. All these features will be helpful to the user as identification of bot traffic will be done automatically by the botcatch application. Instead of the user identifying the bot messages Botcatch can automatically place the bot traffic in the black list by this functionality. Additional functionality to the bots can be added where the bots can launch attacks such as distributed denial of service attack on the command of the botmaster. 39

BIBLIOGRAPHY AND REFERENCES [1] G. Gu, P. Porras, V. Yegneswaran, M. Fong, and W. Lee, BotHunter: Detecting malware infection through IDS-driven dialog correlation, in Proceedings of the 16th USENIX Security Symposium, Boston, MA, USA, August 2007. [2] G. Gu, R. Perdisci, J. Zhang, and W. Lee, BotMiner: Clustering analysis of network traffic for protocol- and structure- independent botnet detection, in Proceedings of the 17th USENIX Security Symposium, San Jose, CA, USA, July 2008. [3] Castle and E. Buckley, The automatic discovery, identification and measurement of botnets, in Proceedings of the 2nd International Conference on Emerging Security Information, Systems and Technologies, Cap Esterel, France, August 2008. [4] H. Choi, H. Lee, and H. Kim, BotGAD: Detecting botnets by capturing group activities in network traffic, in Proceedings of the 4th International ICST Conference on Communication System Software and Middleware, Dublin, Ireland, June 2009. [5] W. Lu, M. Tavallaee, G. Rammidi, and A. A. Ghorbani, BotCop: An online botnet traffic classifier, in Proceedings of the 7th Annual Conference on Communication Networks and Services Research, Moncton, Canada, May 2009. 40

[6] M. Yahyazadeh and M. Abadi, BotOnus: An online unsupervised method for botnet detection, The ISC International Journal of Information Security, vol. 4, no. 1, pp. 51 62, 2012. [7] W. Lu and A. Ghorbani. "Botnets Detection Based on IRC Community". IEEE Communications Society, 2008. [8] http://www.britannica.com/ebchecked/topic/1366374/client-server-architecture [9] http://www.mirc.com/ [10] http://en.wikipedia.org/wiki/client%e2%80%93server_model [11] http://www.spamlaws.com/how-botnets-work.html [12] http://en.wikipedia.org/wiki/software_testing [13] http://securityaffairs.co/wordpress/13747/cyber-crime/http-botnets-the-dark-side-of-an- standard-protocol.html [14] M.Yahyazadeh and M. Abadi, BotCatch: Botnet Detection Based on Coordinated Group Activities of Compromised Host, The 7 th International Symposium on Telecommunications (IST 2014). 41

Appendix 1 CatcherUI.java 42

Class Diagram 43