Data Mining Techniques in Network Security

Transcription

1 in Network Security Mgr. Rudolf B. Blažek, Ph.D. Department of Computer Systems Faculty of Information Technologies Czech Technical University in Prague Rudolf Blažek Network Security MI-SIB, ZS 2011/12, Lecture 12 The European Social Fund Prague & EU: We Invest in Your Future

2 Data mining v síťové bezpečnosti Mgr. Rudolf B. Blažek, Ph.D. Katedra počítačových systémů Fakulta informačních technologií České vysoké učení technické v Praze Rudolf Blažek Síťová bezpečnost MI-SIB, ZS 2011/12, Přednáška 12 Evropský sociální fond Praha & EU: Investujeme do vaší budoucnosf

3 Reference Anoop Singhal (Editor) Data Warehousing and Data Mining Techniques for Cyber Security (Advances in Information Security) Springer, 2006 ISBN:

4 Introduction Data Mining in Network Security Do not use signatures, but learn about usage patterns Also viewed as Machine Learning techniques Two basic detection approaches Misuse detection: detection of normal vs. intrusive data flows Anomaly detection: classification algorithms, rare class predictive models, association rules, cost sensitive modeling 4

5 Introduction Data Mining in Network Security Misuse Detection Models of misuse are created automatically Often better rules than manually created signatures Very good for detection of known (or modified) attacks But deciding normal vs. intrusive is human resource intensive Anomaly detection Models of normal behavior are created automatically Deviations from normal behavior are detected automatically Can potentially detect unexpected and unknown attacks 5

6 Introduction Data Mining for Anomaly Detection Advantages Detection of unknown attacks Detection of unusual behavior interesting to IT managers Limitations Possibly very high false alarm rate (FAR) Supervised Detection Models for normal behavior are built from training data Unsupervised Detection No training data, attempts to learn about anomalies automatically 6

7 Introduction Unsupervised Anomaly Detection No training data Attempts to learn about anomalies automatically Algorithms used Statistical methods, clustering, outlier detection, state machines We will focus on unsupervised anomaly detection We will study MINDS, the Minnesota IDS, as an example It will illustrate the usage of data mining techniques 7

8 MINDS MINDS Minnesota Intrusion Detection System 8

9 MINDS Minnesota Intrusion Detection System Abbreviated as MINDS Uses data mining techniques for unsupervised anomaly detection (assigns an anomaly score to each network connection) association pattern analysis (for suspected anomalous network connections) Performance Detects new intrusions that escape signature-based IDSs 9

10 MINDS Minnesota Intrusion Detection System 4 CHAPTER THREE The MINDS System Network Association Pattern Analysis Summary of anomalies Data Capture Device Anomaly Detection Anomaly Scores Analyst Labels Storage Filtering Feature Extraction Known Attack Detection Detected known attacks Figure 3.1: MINDS System Source: Data Warehousing and Data Mining Techniques for Cyber Security Feature name Table 3.1: Time-window based features Feature description 10

11 MINDS Minnesota Intrusion Detection System Data capture Netflow v. 5 (Cisco protocol) Using flow-tools ( Header information only, summarized into (one-way) flows Much less data storage than tcpdump (or e.g. Wireshark) Flow analysis period: 10 min. (1-2 mil. flows) MINDS processing speed Processing of 10 min. data: less than 3 min But unwanted traffic is filtered out before processing 11

12 MINDS MINDS Operation Steps 1. Feature Extraction Basic features: source and destination IP addresses and ports, protocol, flags, number of bytes, number of packets Derived features for a T-seconds time-window: Capture connections with similar recent characteristics Useful to detect sources of high volume connections per unit time (e.g. fast scanning) Derived features for a connection-window: Similar characteristics, but for the last connections from distinct sources (good for e.g. slow scanning) 12

13 MINDS Time-window based features Feature name count-dest count-src count-serv-src count-serv-dest Feature description Number of flows to unique destination IP addresses inside the network in the last seconds from the same source Number of flows from unique source IP addresses inside the net- work in the last seconds to the same destination Number of flows from the source IP to the same destination port in the last seconds Number of flows to the destination IP address using same source port in the last seconds 13

14 MINDS Connection-window based features Feature name count-dest-conn count-src-conn count-serv-srcconn count-serv-destconn Feature description Number of flows to unique destination IP addresses inside the network in the last flows from the same source Number of flows from unique source IP addresses inside the network in the last flows to the same destination Number of flows from the source IP to the same destination port in the last flows Number of flows to the destination IP address using same source port in the last flows 14

15 MINDS MINDS Operation Steps 2. Signature based detection for known attacks Detected attacks are not further analyzed. Not our focus. 3. Anomaly detection Outlier detection assigns an anomaly score to network flows Humans can analyze only the most anomalous connections 4. Association pattern analysis Summarization of highly anomalous network connections Humans decide whether the summaries can be used to create signatures for detection in step 2 15

16 MINDS MINDS Operation Steps 2. Signature based detection for known attacks Detected attacks are not further analyzed. Not our focus. 3. Anomaly detection Outlier detection assigns an anomaly score to network flows Humans can analyze only the most anomalous connections 4. Association pattern analysis Summarization of highly anomalous network connections Humans decide whether the summaries can be used to create signatures for detection in step 2 16

17 MINDS MINDS Anomaly Detection Local Outlier Factor (LOF) Assigned to each data point It is local: outliers with respect to their neighborhood LOF Calculation 1. Compute density of each point s neighborhood 2. Calculate the ratio of the point s density and the density of all its neighbors 3. Find the average of the density ratios over all its neighbors 17

18 Outlier Detection Outlier Detection Minnesota Intrusion Detection System 18

19 considered as outlier. Therefore, the simple nearest Data Mining Techniques Outlier Detection neighbor approaches based on computing the distances fail in these scenarios. However, the example p 1 may be detected as outlier using only the distances to the nearest neighbor. On the other side, LOF is able to capture both outliers (p 1 and p 2 ) due to the fact that it considers the density around the points. 2-D Outlier Detection Example p 2 p 1 Source: Data Warehousing and Data Mining Techniques for Cyber Security Figure 5. Advantages of the LOF approach 3.3. Unsupervised Support Vector Machines. Unlike 2 weeks data. 7 we records. T DoS teard R2L, exam U2R, by a buffe PROB port- Althou advance i unresolve In his cri tioned a n thetic sim using atta lected fro that the b 19

20 than 4, in this case 6. Data Mining Techniques Outlier Detection Local Outlier Factor Calculation Definition 5: (reachability distance of an object p w.r.t. object o) Let k be a natural number. The reachability distance of object Reachability p with respect Distance to object o is defined as reach-dist k (p, o) = max { k-distance(o), d(p, o) }. e given in Figure 1. This is a simng 502 objects. There are 400 obbjects in the cluster C 2, and two this example, C 2 forms a denser wkins' definition, both o1 and o2 jects in C 1 and C 2 should not be. r, we wish to label both o 1 and o 2 framework of distance-based out- k-distance Figure 2 illustrates bodu o is the a distance idea of reachability from the k-th distance closest with point k = 4. qin- tuitively, at most if object k-1 points p is far are away from o (e.g. p 2 in the figure), then the reachability closer than distance q between the two is simply their actual distance. However, if they are sufficiently close (e.g., p p 1 at least k points are 1 in the figure), NOT the actual farther distance than q is replaced by the o k-distance of o. The reason is that in so doing, the statistical fluctuations of d(p,o) for all the it is the diameter of a circle p's close to o can be significantly reduced. The strength of this with k closest neighbors smoothing effect can be controlled by reach-dist the parameter k (p 2, o) k. The higher k = 4 the value of k, the more similar the reachability distances for objects within the same neighborhood. 2 Source: LOF: Identifying Density-Based Local p Outliers ods, particularly with respect to s. These outliers are regarded as reach-dist k (p 1, o) = k-distance(o) 20

21 case explicitly but simply assume that there are no duplicates. (To Data Mining Techniques Outlier Detection deal with duplicates, we can base our notion of neighborhood on a k-distinct-distance, defined analogously to k-distance in definition 3, with the additional requirement that there be at least k objects with different spatial coordinates.) Local Outlier Factor Calculation Definition 7: ((local) outlier factor of an object p) Local Outlier Factor of Point p Local Reachability Density of p The (local) outlier factor of p is defined as lrd MinPts ( o) lrd MinPts ( p) o N LOF MinPts () p MinPts () p = N MinPts () p The outlier Definition factor 6: of (local object reachability p captures density the degree of an to object which p) we call p an outlier. The It local is the reachability average density of the of ratio p is of defined the local as reachability density of p and those of p s MinPts-nearest neighbors. It is easy to see that the lower p's local reachability reach-dist density MinPts is, and ( po, the ) higher the local reachability densities o of N p's MinPts-nearest neighbors are, the lrd higher is the MinPts ()1 p MinPts ( p) = LOF value of p. In the N MinPts following ( p) section, the formal properties of LOF are made precise. To simplify notation, we drop the subscript MinPts from reach-dist, lrd and LOF, if no confusion Intuitively, the local reachability density of an object p is the in- arises. verse of the average reachability distance based on the MinPtsnearest neighbors of p. Note that the local density can be if all the Proof 21 L t a T l 5.1 L based outliers, however, it is necessary to compare the densities of The different sets of objects, which means that we have to determine the In spon secti density of sets of objects dynamically. Therefore, we keep MinPts In insid parti as the only parameter and use the values reach-dist MinPts (p, o), for the bors clu o N MinPts (p), as a measure of the volume to determine the density jects bors in in the neighborhood of an object p. MinPts is a parameter be boun labe (k before) Lemm quite note To th re dist-mi clude dist-ma Let ε b Then 5.2 fo (i) (ii) Lem jects it holds

22 Outlier Detection LOF Bounds MinPts = 3 C Proof (Sketch): (a i max o N MinPts p d min d max i min by definition o 1 o N Min d min = 4 i max LOF MinPts (p) 4 Source: LOF: Identifying Density-Based Local Outliers d max = 6 i min LOF MinPts (p) 6 lrd( p) q direct N MinPts by definition o Figure 3: Illustration of theorem 1 1 q N giving rise to tighter bounds. In section 5.3, we will analyze in lrd( o)

23 Outlier Detection In particular, we hope to label o 2 as outlying, but label all objects in the cluster C 1 as non-outlying. Below, we show that for most objects in C 1 its LOF is approximately 1, indicating that they cannot be labeled as outlying. LOF Bounds Lemma 1: Let C be a collection of objects. Let reach-dist-min denote the minimum reachability distance of objects in C, i.e., reachdist-min = min {reach-dist(p, q) p, q C}. Similarly, let reachdist-max denote the maximum reachability distance of objects in C. Let ε be defined as (reach-dist-max/reach-dist-min 1). Then for all objects p C, such that: (i) all the MinPts-nearest neighbors q of p are in C, and (ii) all the MinPts-nearest neighbors o of q are also in C, Source: LOF: Identifying Density-Based Local Outliers it holds that 1/(1 + ε) LOF(p) (1 + ε). Proof (Sketch): For all MinPts-nearest neighbors q of p, reachdist(p, LOF(p) q) reach-dist-min. close to 1 Then if p is the in local the reachability same cluster density as Meaning: it s neighbors of p, as per (not definition an outlier) 6, is 1/reach-dist-min. On the other hand, reach-dist(p, q) reach-dist-max. Thus, the local reach- 23

24 Outlier Detection among the 3-nearest neighbors of p, we in turn find their minimum and maximum reachability distances to their 3-nearest neighbors. In the figure, the indirect min (p) and indirect max (p) values are marked as i min and i max respectively. LOF Bounds Theorem 1: Let p be an object from the database D, and 1 MinPts D. Then, it is the case that direct min ( p) LOF( p) indirect max ( p) directmin ( direct max ( p) indirect min () p... (minimum reachability distance from the ( ( ( ( ( closest MinPts neighbors Source: LOF: Identifying Density-Based Local Outliers directmax (... (maximum such reachability distance indirectmin/max(...( the same for the MinPts closest neighbors ( ( ( ( ( ( of the MinPts closest neighbors 24

25 Outlier Detection LOF Bounds pct = relative distance from the center pct is small for points with neighbors in the same cluster Ḻ -- outlier factor LOF LOF max : pct=10% LOF max : pct=5% LOF max : pct=1% LOF min : pct=5% LOF min : pct=1% 0.00 LOF min : pct=10% proportion direct/indirect Source: LOF: Identifying Density-Based Local Outliers Figure 4: Upper and lower bound on LOF depending on direct/ indirect for different values of pct = Figur 25

26 Outlier Detection LOF Bounds pct = relative distance from the center pct is small for point p with MaxPts neighbors in the same cluster no matter if p is in that cluster or not Thus the bounds are good for both cases Outliers: p is NOT in the same cluster as the closest neighbors LOF is large, bounds are tight Non-outliers LOF is close to 1, bounds are tight 26

27 Data Mining Techniques Outlier Detection sented in previous section , when k = 1. We outlier threshold that will serve to determine w point is an outlier or not. The threshold is based o training data and it is set to 2%. In order to co threshold, for all data points from training data mal behavior data) distances to their nearest nei computed and then sorted. All test data points D k (p)... distance from the point p to its k-th nearest neighbor distances to their nearest neighbors greater than outliers: n points old with are the detected largest as D k outliers. (p) values Other Data Mining Methods for Outlier Detection Distance to the k-th closest neighbor Nearest neighbor (NN) Mahalanobis-distance Based Outlier Same as above with Since k the = 1training data corresponds to normal be is straightforward to compute the mean and th deviation of the normal data. The Mahalanob normal data determined [ref] between in training the particular point p and the mea outliers: n points normal farthest data from is the computed mean μ as: of normal data Mahalanobis-distance Based Outlier Detection Mahalanobis distance: d M = ( p ) ( p ), (Σ... covariance where matrix the of normal is the data) covariance matrix of the nor T Similarly Network to Security the previous approach, MI-SIB, ZS 2011/12, the Lecture thresho

28 Data Mining Techniques Outlier Detection sented in previous section , when k = 1. We outlier threshold that will serve to determine w point is an outlier or not. The threshold is based o training data and it is set to 2%. In order to co threshold, for all data points from training data mal behavior data) distances to their nearest nei computed and then sorted. All test data points D k (p)... distance from the point p to its k-th nearest neighbor distances to their nearest neighbors greater than outliers: n points old with are the detected largest as D k outliers. (p) values Other Data Mining Methods for Outlier Detection Distance to the k-th closest neighbor Nearest neighbor (NN) Mahalanobis-distance Based Outlier Same as above with Since k the = 1training data corresponds to normal be is straightforward to compute the mean and th deviation of the normal data. The Mahalanob normal data determined [ref] between in training the particular point p and the mea outliers: n points normal farthest data from is the computed mean μ as: of normal data Mahalanobis-distance Based Outlier Detection Mahalanobis distance: d M = ( p ) ( p ), (Σ... covariance where matrix the of normal is the data) covariance matrix of the nor T Similarly Network to Security the previous approach, MI-SIB, ZS 2011/12, the Lecture thresho

29 considered as outlier. Therefore, the simple n Data Mining Techniques Outlier Detection neighbor approaches based on computing the distance in these scenarios. However, the example p 1 may b Advantage of LOF-based tected outlier using Detection only the distances to the n neighbor. On the other side, LOF is able to capture outliers (p 1 and p 2 ) due to the fact that it considers the sity around the points. Nearest Neighbor Approach points in C1 are far apart p2 is closer to C2 than that p2 will NOT be detected as an outlier but p1 will LOF Approach considers density of clusters both p1 and p2 WILL be detected as outliers p 2 p 1 Source: Data Warehousing and Data Mining Techniques for Cyber Security Figure 5. Advantages of the LOF approach 3.3. Unsupervised Support Vector Machines. U 29

30 MINDS Performance MINDS Performance 30

31 MINDS Performance Computational Complexity Neighborhoods around all data points: Calculate distances between all pairs of data points This is O (n 2 ) calculation computationally infeasible for millions of data points. Sampling a training data set from the observed data Compare all data points to this small set, Reduced complexity: O ( data size x sample size ) Additional benefit: anomalous behavior will not be seen often in the sample, so it will not be mistaken with normal behavior 31

32 MINDS Performance Performance of MINDS Worms October 10, 2002: MINDS detected two activities of the slapper worm that were not identified by SNORT since they were new variations of an existing worm code Scanning and DoS activities August 9, 2002: CERT/CC issued an alert for widespread scanning and possible denial of service activity targeted at the Microsoft-DS service on port 445/TCP August 13, 2002: Network connections related to such scanning were the top ranked outliers in MINDS The port scan module of SNORT failed (slow scanning) 32

33 MINDS Performance Performance of MINDS Policy Violations August 8 and 10, 2002: MINDS detected a machine running a Microsoft PPTP VPN server, and another one running a FTP server on non-standard ports. Both policy violations were the top ranked outliers since they are not allowed, and therefore very rare. 33

34 MINDS Performance Performance of MINDS Policy Violations February 6, 2003: unsolicited ICMP echo reply messages to a computer previously infected with Stacheldract worm (a DDoS agent) were detected by MINDS. The infected machine has been removed from the network, but other infected machines outside the local network were still trying to talk to the previously infected machine. 34

35 MINDS Performance Performance of MINDS vs. SNORT Scanning SNORT rule: 3 seconds window / more than 4 destination IPs for one source IP SNORT requires longer windows for slow scans MINDS detects fast and slow scans automatically MINDS detects outbound scans, SNORT does not by default Policy Violations SNORT requires a rule for each violation MINDS detect any rare unusual behavior 35

36 Mining Association Rules Mining Association Rules Summarizing Anomalous Connections Using Association Rules 36

37 Mining Association Rules Summarizing Anomalous Connections Using Association Rules Association Rules Implications X Y (for binary features X and Y) Example: {Breat, Butter} {Milk} Mining Association Rules Find all rules with specified support (percentage in data) And with specified confidence = estimate of P(Y X) Extracted rules The support of extracted rules is called Frequent Item Set 37

38 Mining Association Rules Summarizing Anomalous Connections Using Association Rules Example rule: {Bread, Butter} {Milk} Assume 10% transactions contain bread and butter 6% of the transactions contain bread, butter, and milk Support of the rule Percentage of data with the rule: 6% Confidence of the rule Estimate of P({Milk} {Bread, Butter}): 6% / 10% = 60% 38

39 Mining Association Rules Summarizing Anomalous Connections Using Association Rules Example rule: {Bread, Butter} {Milk} Support of the rule: 6% Confidence of the rule: 60% If requirement is 1% support and 50% confidence then the rule will be extracted Frequent Item Set: {Bread, Butter, Milk} 39

40 Mining Association Rules Summarizing Anomalous Connections Using Association Rules Association patterns specified as frequent item sets or association rules Summary of detected anomalous flows:x Humans can analyze the summary E.g. a frequent set for scanning: {srcip= X, dstport = Y } This is a candidate signature for a signature-based system Constructing a profile of normal network traffic: Identify sets of features that are found in normal traffic E.g. web browsing: {protocol=tcp, dstport=80, NumPackets=3... 6} 40

41 Mining Association Rules Challenges in Mining Association Rules Imbalanced class distribution Small support for attack, large for normal traffic Binarization and grouping of attribute values Supervised or unsupervised Pruning the redundant patterns Must be subsets with similar support Finding discriminating patterns Patterns that identify attacks and normal traffic Grouping the discovered patterns 41