Framework for Distributed and Self-healing Hybrid Intrusion Detection and Prevention System

Transcription

1 Framework for Distributed and Self-healing Hybrid Detection and Prevention System Fauzia Idrees 1,2, Muttukrishnan Rajarajan 1, A.Y. Memon 2 1 School of Engineering and Mathematical Sciences, City University London, UK 2 National University of Sciences Technologies Karachi, Pakistan {Fauzia.Idrees.1, attaullah@pnec.edu.pk Abstract is a versatile security paradigm which can avert most of the computer and network related attacks, if efficiently employed. This paper presents a novel solution for and prevention of known and unknown network and cloud vulnerabilities. The proposed framework is an amalgamation of some of the existing state-ofthe-art intrusion and prevention technologies. The design of this novel system is adaptable with little customization by complicated networks, cloud, Voice over IP and Next Generation Networks in order to abate the versatile threat environments. Keywords and prevention; Cloud ; Classification; Clustering; Misuse and based. I. INTRODUCTION technology is gaining an indubitable appreciation after undergoing revolutionary changes. It has become the basic foundation of network security structure. Its primary role is to evaluate the information collected on a host or network points against the security policies, generate early warnings and responses thereafter to mitigate the intrusions. With the rapid developments and increasing reliance on computer and network technologies, the attack landscape has also expanded with new threat models to gain unauthorized access to lucrative resources and monetary benefits [1]. With the popularity of cloud, Voice over Internet Protocol (VoIP) and other bandwidth hungry applications, network speed and traffic have also tremendously increased alongside the security concerns to accommodate the performance challenges. The traditional Detection Systems (IDS) technology relying on the single approach is insufficient to fulfill the ever demanding requirements. New techniques are being exploited to make attack s efficient and effective [2] [3]. Still there is a dire need to have a compact intrusion and prevention system with all in one flavor to keep up with the network throughput and speed as well as to deal with the varying environments effectively. In this paper, we present the design of a multi-threaded distributed and prevention system with self-healing hybrid engines and Detection and Prevention Operation Centre (IDPOC). This framework uses NIDS and HIDS at distributed locations augmented with a central operation centre to monitor and coordinate their operations besides the individual and prevention capabilities. Two self-healing engines (misuse and anomaly based) are integrated in the system with multithreaded functionality to cope up with the throughput and speed bottlenecks. Our model consists of Snort and supervised and unsupervised classification and clustering stages based on Bayesian classification, Decision tree and Naïve Bayes algorithms. Different deployment scenarios including the cloud (IaaS, SaaS and PaaS) and VoIP are evaluated for the proof of concept. The rest of this paper is organized as follows: In section II related work is investigated. Sections III and IV present the proposed architecture, its components and suitable deployment scenarios respectively. Finally, section V concludes the work with the possible future work. II. LITERATURE REVIEW The earliest intrusion systems were performing two functions of data capture and analysis. Analysis part was based on signatures of known attacks [3] [4]. Later on anomaly based intrusion was introduced along with the prevention functionality to actively respond against the attacks rather passively monitoring of systems. Additionally, those architectures were centralized thereby relying on a single point of failure [5]. With the technology advancements, distributed IDS functionality was explored by various researchers [6] [9] [12] [13]. Different designed architectures have pros and cons while operating in their basic mode. However, researcher started integrating various methodologies to get best performance out of it. A comparative study on previously integrated solutions is presented in Table I. III. SYSTEM DESIGN The aim of proposed design is to develop a comprehensive and prevention model for accurate and fast intrusion mitigation at a par with the ever demanding network diversity and speed issues. A. Main Operations The idea behind this system design is to perform the and prevention operations on network and host machines efficiently and in a cooperative manner /13/$ IEEE 277 ICTC 2013

2 Ref Year Detection Detection Technique Time TABLE I. COMPARATIVE STUDY OF PREVIOUS APPROACHES Response Architecture Coverage Prevention Pros Cons This work 2013 Misuse Online Offline (Optional) Active Host Network Distributed Computers Networks Cloud VoIP NGN Hybrid, Distributed and optional nondistributed operations, Multithreaded processing for optimum speed and throughput, Versatile coverage scenarios, Real time and prevention functions. High computation cost which could be controlled with efficient implementation. [6] 2012 Misuse [7] 2012 Misuse [8] 2012 [9] 2012 [10] 2011 [11] 2011 [12] 2011 Misuse [13] 2010 Misuse [14] 2010 [15] 2010 Misuse [16] 2010 Misuse [17] 2009 [18] 2008 Misuse Online Active Host Networks VM Cloud Better accuracy of classifier Works only for Cloud. Offline Passive Network Networks Better accuracy of classifier Works only for network. Online Active Network Network Good of well-known anomaly based attacks Online Passive Distributed Cloud Fast computation due to multithreaded operations. P2P structure used to avoid single point of failure. Online Passive Host Networks Good accuracy of well-known Offline Passive VM Cloud Online Active Host Network t known Active Host Networks User s flexibility to choose between low and high security levels to control the speed and throughput. Works only for network. Misuse Works only for Cloud. Works only for Network. High Computation cost. Works only for Cloud. Management task increased due to monitoring of different security levels. Network Can detect many known attacks Detects only known Works only for IPv6 networks. Cannot detect unknown Networks Can detect known and unknown attacks with good accuracy Works only for Networks and and efficiently. may be hosts. Online Active Network Networks Performance not evaluated Detects only unknown Works only for cloud. Online Active Host Cloud Online Active VM Cloud Can detect many known attacks Cannot detect unknown Works only for cloud. Secures VM from DDoS attacks Detects only known prevention. Works only for cloud. Offline Passive Networks Networks Good accuracy for of unknown attacks Online Active Networks Networks Can detect known and unknown Slow anomaly. Works only for Networks. Slow and large data processing for anomaly. Works only for Networks. 278

3 Following functions are incorporated as a one box solution which makes this work unique:- Distributed IDPS. and misuse based s. Detection and prevention operations. Centralized and cooperative logistics. Customizable design for various environments. B. Proposed Architecture The architecture of proposed model is shown in Figure 1. It consists of three main components namely hybrid Network Detection Prevention System (NIDPS), hybrid Host Detection Prevention Systems (HIDPS) and a centralized Detection Prevention Operations Centre (IDPOC). Design of NIDPS and HIDPS are similar except the Multi-threading functionality introduced in NIDPS to efficiently process the large throughput and high speed network traffic. The engines of NIDPS and HIDPS are customized according to the specific threat domains of network and hosts. positioned on the critical machines/servers. Operations of NIDPS and HIDPS are supplemented by IDPOC, which coordinates with NIDPS and HIDPS for data logging, reporting, response and updating tasks. Each IDPS is integrated with SNORT based misuse engine and classification and clustering techniques based anomaly engine. Operations of individual IDPS are closely monitored, organized and upgraded with the help of a supervisor unit, which also communicates with the IDPOC for overall joint operations. Before elaborating the overall functionality of each of these components, the working methodologies of three common sub units of IDPS are discussed in the subsequent paragraphs. 1) Misuse Detection Engine: For misuse, we are using open source SNORT tool to distinguish between the suspicious and normal traffic. It analyses the network traffic for patterns matching with a library of known signatures called SNORT rules, which can be modified by a text editor. These rules are generalized and updated with the help of anomaly classifiers and clustering algorithms to extricate novel Using updated and generalized alert rules, intrusion would be fast and efficient. 2) Detection Engine: model consists of two stages of training and. In the training phase, the normal usage model is learnt by observing the normal traffic in a controlled environment. In the phase, the target data is compared with the learnt normal model to detect any deviations. It generates an alert if the observed events are out of threshold. In order to acquire the accurate classification and reduce the false positives, a layered scheme is developed. The architecture of proposed integrated classifier is shown in Figure 2. In first step, Decision Tree Algorithm is used to classify data as per anomalous attacks and other data. In the second layer, the unclassified data is further analyzed for the anomalous data with the help of Naïve Bayes technique. In the last step, Bayesian clustering is used to get advanced unknown Fig. 1. Architecture of proposed IDPS system. In this framework, NIDPS is to be deployed on perimeter network point in the DMZ zone, whereas HIDPSs are Fig. 2. Working model for engine. 3) Supervisor: Supervisors are local controller of each IDS which performs the reporting tasks and rule updating. It 279

4 also acts as a coordinator with the IDPOC for centralized reporting and updating operations. Whenever there is an alert generated from any of the engines, it is sent to the supervisor which will report to IDPOC for further requisite response in accordance to the nature of threat and policy guidelines. Consequently, the rules and signatures databases are also updated with the new threat patterns. C. NIDPS Architecture The NIDPS architecture is shown in Figure 3. It uses the multi-threading approach to sustain the speed and performance bottlenecks. The network traffic is captured and parsed into multiple threads for concurrent execution and sent to the queue handler. By adopting the concurrent executing threads the performance could be optimized in terms of latency and packet loss. The data packets from the queue handler are processed by the two engines against the available signatures and pre-defined normal behavior rule set. The normal data packets are forwarded for further processing and the detected alerts are sent to the supervisor unit for preparation of alert reports for subsequent forwarding to IDPOC. constantly upgrade their knowledge database in coordination with supervisor. HIDPS Internet Incoming Traffic based Alert Supervisor IDPOC NIDPS Internet Incoming Traffic Multi Threads Queue Handler based Run Fig. 3. Architecture of NIDPS. Alert Supervisor IDPOC D. HIDPS Architecture The HIDPS architecture is shown in Figure 4. It has same components as the NIDPS excluding the multi-threading and queue handler components. Its engines consist of customized signatures and rules specific to operating system and most common applications of host machines. These engines are designed with the self-healing feature to Run Fig. 4. Architecture of HIDPS. E. IDPOS Model IDPOS is a central management console used to collect the alerts data from supervisors and log them. It also compiles and sends reports to the administrator and users. IV. DEPLOYMENT SCENARIOS Recognizing the importance of IDS to fit the security needs of underlying network is the basic essence of information security. However, the efficiency of IDS depends on the deployment and configuration efficacy as per the needs of a particular network [9]. Accurate and proper IDS deployments will ensure the timely counter measures against the intrusions. Some of the possible deployment scenarios are discussed in subsequent paragraphs. A. Cloud Computing The advantages of cloud make it most wanted facility but due to its multi-tenant nature, security has been the biggest concern. Well-known cloud threats include abuse and nefarious use of cloud services, denial of service, shared technology related issues, insecure Interfaces and APIs, malicious insiders, data loss or leakage including insufficient authentication, authorization or audit controls, operational failures, and data center reliability, account and service hijacking, phishing, fraud, unknown risk profile and eavesdropping etc. These threats can, however, be surmounted with adequate security measures. The offered solution is depicted in Figure 5. It consists of NIDPS installed at the perimeter to monitor, detect, and alert on incoming traffic. Additionally, HIDPS are installed on individual hosts/hypervisors to monitor the hypervisor and traffic between the VMs on that hypervisor. In this deployment, appropriate filters are to be implemented to avoid 280

5 the overload of IDSs. IDPOS is processing the alerts from NIDPS and HIDPSs and generate the reports for the cloud provider and users (optional). This framework is workable for SaaS, PaaS and IaaS models of cloud. An additional layer of HIDPSs can be applied on individual VMs to monitor, detect and alert its activity. The proposed architecture can be custom installed to fit in the public, private or internal clouds scenarios. VoIP traffic is processed in VoIP rule engine. The engine compares with the legitimate behavior and looks for the deviations. It generates and sends the alerts to IDPOS for further analysis and countermeasure in case of finding intrusive activities. With multi- threading feature of IDS multiple calls can be monitored simultaneously with minimum processing delay to avoid the inherent issues of jitter and latency. VoIP clients VoIP clients IDPOS NIDPS PBX Gateway VoIP Serever Fig. 5. Deployment model for cloud. B. VoIP VoIP being a heterogeneous and real time application it is a bit challenging for the of malicious activities. Its use of multiple protocols (SIP, RTP, MGCP etc) for each call session and distributed operations (Servers, clients, gateways) make it different from other internet applications. VoIP specific attacks include Denial of Service, billing frauds, eavesdropping, session hijacking, registration hijacking, session tearing down, registration flooding, masquerading, buffer overflow and media stream-based A customized VoIP rule engine module based on stateful, cross-protocol, and VoIP specific signaling and media protocols (SIP and RTP) is proposed to detect the VoIP specific threats. The protocol based rules are generated from standard specifications of SIP and RTP defined in their respective RFCs and used to derive legitimate behavior of VoIP traffic. The normal behavior model is also trained on the session s legitimate transitions to cover the cross protocol states as well as the interactions. Once the legitimate behavior is built and the related attribute features are identified, this rule engine not only lowers the number of false alarms but is also capable of detecting unknown Our proposed architecture is shown in Figures 6 and 7 respectively for two typical VoIP configurations. The NIDPS are to be installed at intermediate nodes like servers, proxies or gateways and HIDPS at the vital clients (optional depending on the vulnerability of the client). The captured Analog Phones Fig. 6. Deployment model for Voice over IP. Fig. 7. Deployment model for Internet Telephony. 281

6 C. Organizational/ Ordinary Networks The IDS deployment in organizational networks is challenging not only due to the versatility of attacks and number of endpoint machines but also the diverse locations of these endpoints as well as the high throughput and the multirole nature. Organizations must deal with the issue of setting the IDS to capture relevant data only and block or ignore the intrusions [17]. Most common network related attacks include social engineering attacks, network sniffing, packet spoofing, session-hijacking Packet, cyber-threats bullying, automated probes and scans, distributed denial of attacks, industrial espionage, executable code attacks, DNS attacks, stealth scanning, remote access attacks, spams, trojan distribution, worms, botnet command and control attacks etc. One possible deployment scenario for typical organization network is shown in Figure 8. An organization with geographically diverse locations and multiple departments might need to deploy NIDS at each distinct location. An additional layer of HIDS might be added to sensitive points. Alerts from all the deployed systems are centrally administered by IDPOS. Figure 8. Deployment model for organizational networks. V. CONCLUSION systems form a necessary layer of a defense in-depth strategy and play a critical role in a comprehensive information protection program [9] [15]. In this paper, we proposed an integrated and hybrid IDPS solution comprising of NIDPS and HIDPS each with misuse and anomaly engines based on Snort, Bayesian classifier, Decision tree and Naïve Bayes techniques. The system is designed to detect known and unknown attacks in cloud, VoIP and standard networks as well as Next Generation Networks (NGN) with customized databases for each scenario. REFERENCES [1] A. Patcha, J. Park, An overview of anomaly techniques: existing solutions and latest technological trends, Int. J. Computer Networks, vol. 12, no. 51, pp , [2] R. Goel, A. Sardana, R. C. Joshi, Parallel misuse and anomaly model, Int. J. Network Security, vol. 14, no. 4, pp , July [3] C. Modi, D. Patel, B. Borisaniya, H. Patel, A. Patel and M. Rajarajan, A survey of intrusion techniques in cloud, Int. J. Network and Comp. Apps., 2012, pp [4] Y. K. Penya, P.G. Bringas, Integrating Network misuse and anomaly prevention, 6th IEEE Int. Conf. Ind. Informatics South Korea, 2008, pp [5] Xuedou Yu, A new model of intelligent hybrid network intrusion system, Int. Conf. Bioinformatics and Biomedical Tech. China, 2010, pp [6] N.M. Chirag, R.P. Dhiren, A. Patel, R. Muttukrishnan, Baysian classifier and Snort based Network Detection system in cloud, Third Int. Conf. Computing, Communication and Networking Tech., 2012, pp [7] H. Om, A. Kundu, A hybrid system for reducing the false alarm rate of anomaly intrusion system, 1st Int. Conf. Advances in Info. Tech., 2012, pp [8] N. Wattanapongsakorn, S. Srakaew, E. Wonghirunsombat, C. Sribavonmongkol, T. Junhom, P. Jongsubsook, C. Charnsripinyo, A Practical Network-based Detection and Prevention System, 11 th Int. Conf. Trust, Security and Privacy in Computing and Comm., 2012, pp [9] H.A. Kholidy, F. Baiardi, CIDS: A Framework for Detection in Cloud Systems, Int. J. Cloud Computing Services and Architecture, vol. 2, no. 6, 2012, pp [10] E.W.T. Ferreira, G.A. Carrijo, R. de Oliveira, N.V. de Souza, Detection System with wavelet and neural artificial network approach for network computers, IEEE Latin America Transactions, 2011, vol. 9,. 5, pp [11] Jun-Ho Lee, Min-Woo Park, Jung-Ho Eom, Tai-Myoung Chung, Multi-level Detection System and log management in Cloud Computing, 13th Int. Conf. on Advanced Communication Technology, 2011, pp [12] Ke Yun, Zhu Jian Mei, Research of Hybrid Detection and Prevention System for IPv6 Network, Int. Conf. on Digital Object Identifier, 2011, pp [13] D. Zhao, Q. Xu, Z. Feng, Research and Design for Detection System with Hybrid Detector and Apriori Algorithm, 2nd Int. Conf. on e-business and Information system security, 2010, pp [14] C. Mazzariello, R. Bifulco, R. Canonoco, Integrating a network IDS into an open source cloud, Sixth int. conf. information assurance and security (IAS), 2010, pp [15] A. Bakshi, B. Yogesh, Securing cloud from DDOS attacks using intrusion system in virtual machine, Second Int. Conf. Comm. software and networks, 2010, pp [16] H. Lu, J. Xu, Three-level Hybrid system, Int. Conf. Info. Engg. Comp. Science, 2009, pp [17] Y. K. Penva, P. G. Bringas, Integrating Network misuse and anomaly prevention, Sixth Int. Conf. Industrial Informatics, 2008, pp