Contribution of the French MetroSec

Transcription

1 Contribution of the French MetroSec project to traffic anomalies detection ti Philippe OWEZARSKI LAAS-CNRS Toulouse, France With the contribution of Patrice Abry, Pierre Borgnat, Nicolas Larrieu, Antoine Scherrer, Silvia Farraposo Colloque STIC, Paris, 7 novembre

2 Outline 4 Traffic characteristics and IDS 4 A non Gaussian and long memory model for Internet traffic with anomalies 4 Model validation with traffic traces (with and without anomalies) 4 Anomalies/DDoS attacks detection 8With the non Gaussian and long memory model 8Using deltoids 4 Ongoing g and Future work Colloque STIC, Paris, 7 novembre

3 Limits of current IDS 4 Signature based: 8 Create a Data Base with known attack signatures 8 But the Data Base is always incomplete 8 New attacks cannot be detected 4 Profile based 8 Issuing a stable profile for the analyzed system 8 Compare the actual behavior with the typical profile 8 Able to detect t new attack Even if profile based IDS are not appreciated, at we like its ability to detect new attacks But what are Profile based IDS lacks? Traffic monitoring, in characterization, ti analysis and modeling should help to define traffic profile Colloque STIC, Paris, 7 novembre

4 Known traffic characteristics 4 Non Gaussian, non Poisson statistics 4 Long Range Dependence (LRD), Strong correlations 4 Traffic can look different according to the granularity of observation 4 And Traffic is highly variable! Colloque STIC, Paris, 7 novembre

5 Profile based IDS issues Traffic profiles in IDS do not consider such variability False positive rate is high Impossible to fix reliable thresholds Temporal evolution of the number of TCP/SYN packets A traffic profile cannot be based only on some averages (non Gaussian) High level statistics are required Colloque STIC, Paris, 7 novembre

6 Marginal laws 4 Distributions of empirical probabilities LBL-TCP-3 Δ=4ms Δ=32ms Δ=256ms 4 Poisson model? Exponential law? Gaussian? 4 What aggregation level to select? Colloque STIC, Paris, 7 novembre

7 Traffic Correlation (SRD and LRD) Hurst parameter, H = Colloque STIC, Paris, 7 novembre

8 What model for a non Gaussian and long memory process? Colloque STIC, Paris, 7 novembre

9 Non Gaussian with LRD model Joint modelling of 1st and 2 nd orders statistics ti ti 4 Packet aggregated g count process: X Δ Δ( (k) or X Δ (k) = #pkt during [kδ, (k+1)δ] 4 Bytes aggregated count process: W Δ (k) W Δ (k) = #bytes during [kδ, (k+1)δ] 1st. PDFs of marginals as gamma laws Note: one fit for each Δ 2 nd. Covariance (or spectrum) with LRD Covariance of a farima model Colloque STIC, Paris, 7 novembre

10 Gamma distributions Γ = Γ β β α β β α α x x x exp ) ( 1, 1 ) ( β β β ) ( Shape parameter α : can model from Gaussian to exponential ; 1/ α distance to Gaussian Scale parameter β : multiplicative factor Colloque STIC, Paris, 7 novembre Scale parameter β : multiplicative factor

11 Long memory from a farima model 4 Long range dependence covariance is a non-summable power-law spectrum f XΔ (ν): f γ XΔ (ν) C ν, ν 0, with 0<γ<1 4 Farima = fractionnaly integrated ARMA 1. Fractional integration with parameter d LRD with γ=2d 2. Short range correlation of an ARMA(1, 1) parameters θ, φ f X Δ ( ν ) = σ 2 ε 1 e i2πν 2d 1 θe 1 φeφ e i2 πν 2 i2πν 2 Colloque STIC, Paris, 7 novembre

12 Monitoring platform ENST DAG LIP6 Jussieu QoSMOS ENST-B INT RIPE TTM Mont de Marsan ENS Lyon Pau EURECOM IUT GTR LAAS Colloque STIC, Paris, 7 novembre

13 Traces for validation Data Date (start time) T (s) Network link # Pkts (10 6 ) IAT (ms) Repository PAUG (11:25) 2620 LAN(100BaseT) ita.ee.lbl.gov/index.html LBL-TCP (14:10) 7200 WAN(100BaseT) ita.ee.lbl.gov/index.html AUCK-IV (13:00) WAN(OC3) wand.cs.xaikato.ac.nz/wand/wits CAIDA (10:00) 600 Backbone(OC48) /oc48/ UNC (16:00) 3600 WAN(100BaseT) www-dirt.cs.unc.edu/ts METROSEC-ref (18:30) 5000 LAN(100BaseT) METROSEC-ref (02:00) 9000 LAN(100BaseT) METROSEC (20:00) 09(20:00) 9000 LAN(100BaseT) DDoS METROSEC-FC (14:30) 1800 LAN(100BaseT) Colloque STIC, Paris, 7 novembre

14 DDoS and flash crowds database Id trace beginning Trace duration(s) attack beginning Attack duration (s) Throughp ut Packet size ratio (%) Generated with Iperf R 17: : I 9: : II 14: : III 16: : IV 10: : V 10: : A 14: : B 16: : C 10: : Flash crowds (generated by humans) FC-1 13: : FC-2 15: : Colloque STIC, Paris, 7 novembre

15 Γ α,β farima (φ, d, θ) model validation 4 Parameters estimation: 8 1st order: Instead of the usual moment based technique which estimates μ and σ 2, we use maximum likelihood based estimates for α and β. 8 2 nd order: LRD (long memory) estimated with a multiresolution analysis, characterized by d,, the long memory parameter measured on an aggregation range Δ for which the log scale diagram is linear. From this wavelet base estimation of d, we perform a fractional derivation of X Δ. This removes the long memory from the process so that only the ARMA component is left. φ and θ are easy to estimate with an iterative procedure based on the Gauss-Newton algorithm. Colloque STIC, Paris, 7 novembre

16 Γ α,β farima (φ, d, θ) model validation 4 To assess the validity of the model with actual traffic traces, we made a comparative analysis of : 8Actual traces time series 8Γ αβ α,β farima (φ, d, θ) ) time series produced by a numerical generator designed for this purpose Colloque STIC, Paris, 7 novembre

17 AUCK-IV: Γ α,β farima (φ, d, θ) fits marginals covariances Δ=10ms Δ=100ms Δ=400ms j=1 corresponds to 10 ms Colloque STIC, Paris, 7 novembre

18 METROSEC-ref1: Γ α,β farima (φ, d, θ) fits marginals covariances Δ=10ms Δ=100ms Δ=400ms j=1 corresponds to 10 ms Colloque STIC, Paris, 7 novembre

19 METROSEC-DDoS & FC: Γ α,β marginals fits DDoS attack Flash crowd Δ=2ms Δ=32ms Colloque STIC, Paris, 7 novembre

20 Logscale diagrams for METROSEC-DDoS & FC DDoS Flash Crowd During After Before Colloque STIC, Paris, 7 novembre

21 Estimated α and β as a function of log 2 Δ During After Before DDoS Flash Crowd α β Colloque STIC, Paris, 7 novembre

22 DDoS impact on traffic (1) 4 α = shape parameter, 1/α quantifies the gap with a Gaussian law 4 β = scale parameter decreases during DDoS attack DDoS attack accelerates the convergence towards a Gaussian distribution ibuti of traces, and decreases the fluctuation scale around the average e traffic Colloque STIC, Paris, 7 novembre

23 DDoS impact on traffic (2) 4 Histograms 8Regular traffic: 0 at the origin α small but the variance (then β) is large 8DDoS attack: Equals 0 on an interval containing 0 fast increase of α and dispersion (the β) small and blocked Colloque STIC, Paris, 7 novembre

24 Partial conclusion 4Model M for characterizing r Internet traffic which works with and without anomalies 4Some parameters change differently in the presence of a legitimate (flash crowd) or illegitimate (DDoS) anomaly How to use such model for an efficient and robust profile based IDS? Colloque STIC, Paris, 7 novembre

25 Detection principles 4 Select a reference window 4 Segment the trace into sliding windows of duration T 4 For a window at time I: 8Aggregated trace at scales Δ=2j, j=1,...,j 8Estimation of parameters : α Δ (I), β Δ (I) 8Compute the distance to the reference, between I and R: D(I) 8Selection of a threshold λ: if D(I) λ, anomaly Colloque STIC, Paris, 7 novembre

26 Selection of the best distance (Basseville 89) 4Quadratic distance on parameters J = = J j j j R I j I D )) ( ) ( ( 1 ) ( α α α = = J j j j R I J I D )) ( ) ( ( 1 ) ( β β β 4Divergence of Kullback-Leibler; p1 and p2 are 2 p.d.f. = dx x p x p x p x p p p DK )) ( ln ) ( )(ln ( ) ( ( ) ( giving a distance with one or two scales: = dx x p x p x p x p p p DK )) ( ln ) ( )(ln ( ) ( ( ), ( g g ), ( ) (,, ) (1 R I D p p DK I K Δ Δ Δ = ) ( ) ( ) 2 ( D Colloque STIC, Paris, 7 novembre ), ( ) (, ',, ', ) (2 ', R I D p p DK I K Δ Δ Δ Δ Δ Δ =

27 Ex. 1 : Denial of Service attack D α (I) D β (I) Colloque STIC, Paris, 7 novembre

28 Ex. 2: Multiplicative increase of traffic D α (I) D β (I) Colloque STIC, Paris, 7 novembre

29 Ex. 3: Comparison between distances KL 1D, j=4 KL 1D, j=7 D α KL 2D, j=4,7 Colloque STIC, Paris, 7 novembre

30 Statistical performance: ROC curves 4 ROC curves: detection probability according to the fixed probability of false alarms 4 P D =f(p FA ) or P D =f(λ), P FA =f(λ) Colloque STIC, Paris, 7 novembre

31 Statistical performance: detection proba. Method #A #B #C #R #I #II #III #IV #V DQM Dα P FA 10% DK K 16 (1D) DK K (1D) DK K 16,128 (2D) DQM Dα P FA 20% DK K 16 (1D) DK K 128 (1D) DK K 16,128 (2D) Colloque STIC, Paris, 7 novembre

32 Conclusion on anomalies/attacks detection 4 Parameters of the Γ αβ α,β farima (φ, d, θ) ) model change differently depending on the type of anomaly 4 Kullback- Leibler distance allows a robust detection of attacks, even when they represent less than 1% of the traffic (and is not sensitive to an artificial increase of the amount of traffic) BUT: it is not possible with this method to identify anomaly constituting packets / flows Colloque STIC, Paris, 7 novembre

33 Objectives 4 Define an approach to 8Detect 8Classify 8Identify traffic anomalies (One or more occurrences that change the normal flowing of data over a network) 4 Define a signature for each traffic anomaly, based on simple parameters must be easy to handled by network administrators must permit the design of IPS Colloque STIC, Paris, 7 novembre

34 The NAD Algorithm 4 Multi-scale concept 4 Tomography-based concept 4 Generic multi-criteria 8 Uses simple mathematical functions, as volume parameters, to detect anomalous flows Number of packets per unit of time Number of bytes per unit of time Number of new flows per unit of time 8 Uses IP features (addresses and ports) to identify the anomalies Colloque STIC, Paris, 7 novembre

35 The NAD Algorithm (2) Multi-Scale 600sec 300sec 300sec... 60sec 60sec 60sec 60sec 60sec 60sec 60sec 60sec 60sec 60sec 30sec 30sec 30sec 30sec 30sec 30sec 30sec 30sec 30sec 30sec 30sec 30sec 30sec 30sec 30sec 30sec 30sec 30sec 30sec 30sec Colloque STIC, Paris, 7 novembre

36 The NAD Algorithm (3) Tomography Colloque STIC, Paris, 7 novembre

37 Formal Definition 4 To detect an anomaly it must be responsible for a significant variation in one of the parameters deltoid based method Let, X = {x1,x2,...,xn}, xi = #{packet byte flows} and packet Δ = time X P = = pi pi _ granularit y { x1, x2,..., xn}, xi = {# packets # bytes # flows } { p1, p2,..., pn 1}, pi = xi + 1 xi E( p) + kσ, select < E( p) + kσ, reject / Δ Colloque STIC, Paris, 7 novembre

38 Some Types of Anomalies Port ID Port Scan Other type Network Scan Src IP DDoS Dst IP The distribution of points in plots can give a clue about the type of anomaly! Long flow Flash Crowd Dst IP Colloque STIC, Paris, 7 novembre

39 Flooding Attack Colloque STIC, Paris, 7 novembre

40 Signatures Denial of Service Colloque STIC, Paris, 7 novembre

41 Signatures Denial of Service n sp : n dp IP Source IP Source/Port IP Source/Port IP Source/Port n sp : 1 dp IP Source IP Source/Port IP Source/Port IP Source/Port Colloque STIC, Paris, 7 novembre

42 Flooding Attack Colloque STIC, Paris, 7 novembre

43 Signatures Network Scan IP Destination IP Destination IP Destination/Port Destinationn Port Colloque STIC, Paris, 7 novembre

44 Network Scan Colloque STIC, Paris, 7 novembre

45 NAD tool assessment Colloque STIC, Paris, 7 novembre

46 Contribution in anomalies detection 4 Experimental platform with monitoring and measurement capabilities 4 IDS assessment methodology 4 Its related database of traces with anaomalies 4Unfortunately not publicly available 4 Original anomalies detection, classification and identification algorithms 4Which proved to be efficient and accurate 4Which raised many interests : FT, WIDE, 4 Traffic generator Colloque STIC, Paris, 7 novembre

47 Ongoing and future Work 4 Performance comparison between our tools and several other detection tools (both research and commercial) 4 Fixing automatically threshold h 8Use honeypots results on botnets for this purpose 4 Using sketches on the Γ α,β farima (φ, d, θ), it is possible to detect the anomalies constituting packets (SIGCOMM LSAD 2007) Colloque STIC, Paris, 7 novembre

48 More information Colloque STIC, Paris, 7 novembre