Procedia Computer Science

Transcription

1 Procedia Computer Science 00 (2011) Procedia Computer Science WCIT-2011 Host Based Anomaly Detection Using a Combination of Artificial Immune Systems and Hypervisor Technology Reza Azmi a*, Boshra Pishgoo a, Hamed Nemati a Operating System Security Lab (OSSL), Alzahra University, Tehran, Iran, Abstract Intrusion Detection based upon Artificial Immune Systems is an attractive approach in research community. Host based Intrusion Detection System techniques usually model normal and abnormal behaviours of user level processes. Behaviour modeling has two requirements: a suitable pattern for describing behaviour of each process and an efficient method for constructing behavioural model of processes based on these patterns. In this paper, we used system calls together with their arguments as suitable patterns and applied an Artificial Immune System as an efficient method for anomaly detection. In security applications, behaviour patterns must be collected safely, so we proposed a novel hypervisor-based architecture for information gathering. Then applied negative selection methods of Artificial Immune Systems for anomaly detection. Our experimental results show that this intrusion detection system not only has high accuracy and low false alarm but also has a robust and secure architecture by using the advantages of both hypervisor technology and Artificial Immune Systems. Keywords: Intrusion Detection System; Anomaly Detection; Artificial Immune System; Negative Selection Algorithm; Hypervisor, System call Mining. 1. Introduction By looking back at intrusion history, we perceive that detecting and countering subversive actions is one of the most necessary steps for system immunization. Traditional intrusion prevention techniques, such as firewalls, access control or encryption, have failed to fully protect networks and systems from increasingly sophisticated attacks and malwares. As a result, Intrusion Detection Systems (IDS) have become an indispensable component of security infrastructure to detect these threats before they inflict widespread damage [1]. In general, IDSs fall into two categories according to the location they put in it, namely i) Network-based Intrusion Detection System (NIDS) and ii) Host-based Intrusion Detection System (HIDS). NIDSs [2] place in another computer or a network device and evaluate network traffic for intrusion detection. In contrast, HIDSs [3] reside inside operating systems and evaluate their behaviours. These IDSs have better system view and so higher detection power than NIDSs; but they will be prone to attacks by rootkits that are in the system. Thus, HIDSs can be suitable option for system protection by isolation and get better resistance to attacks. HIDSs can be implemented in different layers of software system. User level solutions [4] have low facility to detect malfunctioning processes because the only sight of system comes from application transaction log files or * Reza Azmi. Tel.: ; fax: address: azmi@alzahra.ac.ir.

2 overriding system libraries that nonetheless needs recompiling application. So they can be bypassed simply by processes having higher privileged. In contrast kernel level IDSs [5] have a complete view of the whole kernel subsystems and get protection from user level processes; but they are still vulnerable against kernel level intrusions. Putting IDS in user or kernel layer of software system will be putting a new function to a very complex with big chunk of code that is mostly not a trusted platform. In this paper, we put our HIDS into a small hypervisor that communicate with other layers by a known interface namely x86 ISA. This hypervisor will be more trustworthy container for an IDS Because of its smallness and definitive interface and makes IDS more robust by completely isolating it from the other software layers [6]. When building an IDS one needs to consider many issues, such as data collection, feature extraction, intrusion recognition and reporting [1]. For data collection, IDSs can use different methods such as profiling and feature vise analysis [2] to have a view of running processes. System calls auditing is a way of feature vise analysis of a process to get suitable pattern of its behaviour. In this area, various works have been done in different layers of software system like user, kernel or hypervisor layers [3,7-11]. In this paper, we propose a novel hypervisor-based architecture for auditing system calls and their arguments to safely collect data and create secure log files. Then we can select and extract some suitable features among these files, form them as normal or abnormal patterns and create a safe dataset based on them. After data collection and feature extraction phases, we must construct a detection model from secure dataset for intrusion recognition. In this area, IDSs fall into 2 categories according to the detection approaches they employ, namely i) anomaly detection and ii) Misuse detection. Misuse detection identifies intrusions by matching observed data with pre-defined descriptions of intrusive behaviour. Therefore, well-known intrusions can be detected efficiently with a very low false alarm rate. But this approach will fail easily when facing unknown intrusions. Anomaly detection is orthogonal to misuse detection. It hypothesizes that abnormal behaviour is rare and different from normal behaviour. Hence, it builds models for normal behaviour and detects anomaly in observed data by noticing deviations from these models. Anomaly detection has the capability of detecting new types of intrusions, and only requires normal data when building profiles. However, its major difficulty lies in discovering boundaries between normal and abnormal behaviour, due to the deficiency of abnormal samples in the training phase [1]. Anomaly detection is a very active research area and Different works have been frequently used to improve it, up now. Most of these studies are in the field of web anomaly detection and use web information for model construction and intrusion detection. The reminder of researches use low-level system information such as system calls and construct their models using simple classifier based on them [4,8,12-14]. We use Artificial Immune Systems (AIS) approach for system call mining and anomaly detection. This approach is inspired from Human Immune Systems (HIS) and almost is a new research field of computational intelligence in anomaly detection. AISs are suitable options for intrusion detection because of their distributed, self-organized and lightweight nature. Therefore, in this paper, we apply Negative Selection (NS) algorithms as a main branch of AIS to mine system calls and detect anomalies, for the first time. The remainder of this paper is organized as follows. Our proposed architecture and its implementation are described in Section 2 by details. Section 3 evaluates our model at different views like execution time and memory consumption. It investigates the experimental results of our model based on NS algorithms and compares them by Bayesian as a simple classifier. Finally, section 4 concludes all topics. 2. Proposed Architecture and Implementation In this section we describe the architecture of our proposed model and its implementation by details. As mentioned in previous sections, our goal is to detect anomalies using secure system calls mining and AIS approaches. To increase security, a good idea is to use an additional layer namely hypervisor layer in our model and do some vital actions like data collection and intrusion detection in it. Fig. 1 depicts the overall architecture of our proposed model where dashed lines indicate transactions between two different layers, while solid lines indicate internal transactions between components of a layer. This model contains three layers (user, kernel and hypervisor) and four main components namely secure auditor, feature extractor, online HIDS and security exception manager in hypervisor layer. When a user level process calls a system call, some actions are done in order that is shown in Fig. 1. First, control is transferred from kernel layer to the hypervisor through injected code in kernel (stages 1 and 2). In hypervisor, secure auditor does two actions simultaneously (stage 3). This component i) records system call and its arguments in a log file to create a secure dataset at next stages and ii) sends collected information to feature extractor as input of it. At the next stage, feature extractor extracts some suitable features which are able to

3 distinguish between normal and abnormal behaviors, among all received information for each system call. Then it forms these selected features as a pattern and sends it to HIDS as input of it (stage 4). Online HIDS which was trained using AIS methods in an offline training phase analyses received pattern and reports its type (normal or abnormal) to security exception manager (stage 5). If this manager receives an abnormal pattern, it does some suitable actions according to its security policies; and otherwise it returns the control from hypervisor to kernel layer (stages 6 and 7). In this stage, the routine of the system call can be run. The security policies of security exception manager can be different according to administrator s idea and we do not discuss them in this paper. We describe the implementation of the other vital components in the following subsections by details Secure Auditor The main task of this component is to audit system calls and their arguments in hypervisor layer. Implementation of system calls in Intel architecture can be done by one of the two mechanisms: 0x80 interrupt or SysEnter/SysExit. To support all kinds of system call implementation mechanisms, and prevention of OS source code modification, we decided to exert our auditing mechanism using code injection technique in run time. Since hypervisor have privileged access to all memory regions of guest OS, we used hypervisor to inject our intended byte code in kernel memory space of guest OS to change execution follow of system calls and call secure auditor to audit them. For this purpose, First we must allocate the needed space for injecting code through kmalloc() function and send the address of this space to hypervisor by VMMCall(). In this step, start and end addresses of a system call routine will be found in order to discover the address of the first call instruction. Then we replace the injected code from start address till the call instruction address, but we let the following code untouched. The injected code, first stores original values of registers in the end of allocated memory; then it calls getpid(), getuid() and getgid() system calls to gather some information about Pid, Uid and Gid and store them in a part of the allocated memory. At the next step, hypervisor is called by using VMMCall(). In hypervisor, secure auditor audits the type of system call and all of its arguments in a log file and finally, the control is returned to kernel for running the system call. The result of these models is a safe log file that its information is audited by secure auditor in hypervisor. This log file is used for creating a secure dataset. Fig. 1. Overall view of proposed architecture 2.2. Feature Extractor The main task of this component is to extract some suitable features among all information which were audited for each system call. Therefore selecting a subset of effective features which are able to distinguish between normal and abnormal behaviors is one of the most important issues for designing this unit. In this paper, we select 6 features as members of effective subset: the number of system call, access mode and flag arguments, Pid, Uid and Gid. Feature extractor forms these features as a pattern and sends it to HIDS as input of it.

4 2.3. Online HIDS The main task of this component is to detect intrusions online. Online HIDS is placed in hypervisor layer to get better resistance against kernel malwares. This unit of our model analyses received patterns and recognizes their types (normal or abnormal) using detectors that are generated based on AIS approaches and stored within a storage in an offline training phase. We need to describe AIS methods for explaining about training phase. AISs approach is inspired from HIS and almost is a new research field of computational intelligence in anomaly detection. HIS has successfully protected our bodies against attacks from various harmful pathogens, such as bacteria, viruses, and parasites. It distinguishes pathogens from self-tissue, and further eliminates these pathogens. This provides a rich source of inspiration for computer security systems, especially intrusion detection systems. A main branch of AIS are Negative Selection methods. NS algorithms simulate the process of selecting nonautoreactive lymphocytes. Consequently, given a set of normal pattern, they will generate a set of detectors which match none of these normal data samples. These detectors are then applied to classify new patterns as self (normal) or non-self (abnormal) [1]. There are various NS algorithms but in this paper we concentrate on two popular types of it for training our HIDS namely i) constant-sized detectors algorithm and ii) variable-sized detectors algorithm. Both of these algorithms are applied on all points of a cube with unit size, where d is the dimension of patterns (in this paper ). d U 0, 1 Training set contains only a normal set, S, that each member of it (a normal sample s i S, i s c, r i s, d 6 i 1,...,l where l is the size of S) is a sphere with centre c i and radius r s with constant size that is placed in cube, U. each point within each normal sphere s i is considered as a self point. In this step, NS algorithms can generate a set of detectors as Non-Self set, D, based on self set, S. each member of D ( a detector d c, r, j 1,..., m where m is the size of D) is a sphere with centre c j that is selected d j D, j j dj randomly as a point within the cube, U, and radius r dj that is defined using (1) and (2) in constant-sized detectors and variable-sized detectors algorithm, respectively. In (2), dist(.) is a function that calculates Euclidean distance between its arguments. For all j where r d is a constant value (1) r r dj r d dj min dist 1 i l c i, c j rs As described above, we need a self set (a set of normal samples) to generate our detectors in training phase. Self set can be provided through safe log file that is created using Secure Auditor in a trusted mode (we are sure there are not any malwares in system at this mode). In training phase, this safe log file is sent to Feature Extractor out of hypervisor layer and patterns that are formed by this component store in a secure dataset of normal patterns. These patterns are used for generating detectors by one of two algorithms that are explained already. Generated detectors are stored in storage and this storage is transmitted to hypervisor layer of system to detect anomalies online. 3. Evaluation Our model has been developed based on Bitvisor hypervisor [15] having security approach as its main focus. This hypervisor uses parapassthrough architecture which helps it to intercept every intended interaction between hardware and software. In this section we evaluate detection accuracy of our HIDS in offline mode but this is clear that when generated detectors have high accuracy in offline mode, they will have good performance in online mode too. For this evaluation, we need a set of normal and abnormal patterns together with their labels (normal or abnormal label). These patterns can be collected in safe mode and stored in a secure dataset using our proposed architecture. We used some malwares and rootkits to generate abnormal patterns. We put this malwares in system and started system call auditing. Meanwhile of auditing process we let different users to do their regular works to obtain system s normal behaviour too. For assigning correct label to the patterns of our dataset, we need to clarify rootkits from regular system processes. In this regard we used chkrootkit program. Chkrootkit searches for the known rootkits signatures. This tool found PID s of the suspicious processes and enabled us to assign correct label to each pattern of dataset. This generated dataset contained normal patterns and abnormal patterns at first but before evaluation, we deleted repetitive patterns from it for increasing the speed of experiment. For evaluation we used k-cross fold method with k=3 as evaluation method and accuracy (Acc) and false alarm (FA) as evaluation criteria. These two criteria can be defined according to (3) and (4), respectively. (2)

5 TP TN Acc TP TN FP FN FP FA TN FP Reza Azmi, Boshra Pishgoo, Hamed Nemati/ Procedia Computer Science 00 (2014) Where, TP is the number of abnormal patterns that are recognized as abnormal. TN is the number of normal patterns that are recognized as normal. FP is the number of abnormal patterns that are recognized as normal and FN is the number of normal patterns that are recognized as abnormal. Table 1 shows values of two above criteria for three algorithms: i) Bayesian classifier, ii) constant-sized detectors algorithm and iii) variable-sized detectors algorithm. Every algorithm has been ran 5 times and the average of values has been calculated. According to this table, NS algorithms that only have been used from normal data in training phase have higher accuracy and lower false alarm in comparison with Bayesian classifier that has been used normal and abnormal data for training. Table 1. Experimental results for Bayesian classifier and NS algorithms Detection Methods Bayesian classifier AIS (Negative Selection Methods) Constant_Sized Detectors Variable_Sized Detectors Experimental Train Phase Normal + AbNormal Only Normal Conditions Test Phase Normal + AbNormal Normal + AbNormal Evaluation Criteria Acc (%) FA (%) Acc (%) FA (%) Acc (%) FA (%) Experimental Results Run # Run # Run # Run # Run # Average 69.35± ± ± ± ± ± Conclusion In regard to widespread of malware threats and their efforts for canceling out the intrusion detection systems, recently, the most significant part of researches focused on using hypervisor for intrusion detection to isolate IDSs from monitored OS. For this purpose, we have presented novel hypervisor based architecture for auditing system calls and their arguments to safely collect data and create secure dataset. Then we have applied AIS methods on this dataset for system call mining and anomaly detection. Our model is based on a Bitvisor hypervisor and has a high efficiency in detecting intrusions befallen in guest operating system. This architecture is aided by hardware based virtualization technology and will be protected against upper layer attacks using processor s restrictions. Our experimental results show that our model has high accuracy and low false alarm but we will concentrate on the other algorithms of artificial immune systems like danger theory or immune network for achieving better results in future. Reference 1. S.X. Wu and W. Banzhaf, The use of computational intelligence in intrusion detection systems: A review, Applied Soft Computing, vol. 10, pp. 1 35, (2010) 2. Snort. snort,open source intrusion prevention and detection system Koichi Onoue, Y. O. and Yonezawa,. Control of system calls from outside of virtual machines. In Proceedings of the 23rd Annual ACM Symposium on Applied Computing, (2008), pp By Daniel P. Bovet, M. C. Understanding the Linux Kernel, 3rd Edition. O Reilly, (2005). 5. Huangang, X. Building a secure system with lids. lids-0.2.html. 6. Vinod Ganapathy, M. J. R. A. B. M. M. S. S. J. The design and implementation of microdrivers. ASPLOS, (2008), pp Author,Y. L. V. R. V. Using text categorization techniques for intrusion detection. USENIX Security Symposium, (2002), pp A. K. Ghosh, A. S. and Shatz, A. M. Learning program behavior profiles for intrusion detection. Proceedings of 1st USENIX Workshop on Intrusion Detection and Network Monitoring, Santa Clara, CA, (1999). 9. W. Lee, S. J. S. and Chan, P. K. Learning patterns from unix process execution traces for intrusion detection. Proceedings of AAAI97Workshop on AI Methods in Fraud and Risk Management, (1997), pp C. Ko, G. F. and Levitt, K.. Automated detection of vulnerabilities in privileged programs by execution monitoring. Proceedings of 10th Annual Computer Security Applications Conference, Orlando, FL, (1994), pp Xuxian Jiang, X. W. out-of-the-box monitoring of vm-based high-interaction honeypots. Proceedings of the 10th International Symposium on Recent Advances in Intrusion Detection (RAID 2007), Queensland, Australia, (2007), pp Almassian Negar, A. R. B. S. Aidslk:an anomaly based intrusion detection system in linux kernel. Information Systems Technology and Management, (2009), pp (3) (4)

6 13. M. Asaka, T. O. T. I. S. O. and Goto, S. A new intrusion detection method based on discriminant analysis. IEEE TRANS. INF. & SYST., E84-D(5), (2001), pp M. Mehdi, S.& Bensebti, M. A bayesian networks in intrusion detection systems. Journal of Computer Science 3 (5), (2007), pp Takahiro Shinagawa, Bitvisor: a thin hypervisor for enforcing i/o device security. VEE, (2009), pp