Le Routage Robuste Réactif Une combinaison de techniques d ingénierie de trafic proactives et réactives pour traiter le trafic dynamique de réseau

Transcription

1 Le Routage Robuste Réactif Une combinaison de techniques d ingénierie de trafic proactives et réactives pour traiter le trafic dynamique de réseau Pedro Casas et Sandrine Vaton Brest, France, mars 2008 TELECOM Bretagne Département Informatique Universidad de la República Facultad de Ingeniería Uruguay

2 Outline 1 Introduction to the problem 2 A proactive approach: the Robust Routing 3 A reactive approach: Anomaly Detection/Localization 4 A combined approach: the Reactive Robust Routing 5 Conclusions and Perspectives

3 Outline 1 Introduction to the problem 2 A proactive approach: the Robust Routing 3 A reactive approach: Anomaly Detection/Localization 4 A combined approach: the Reactive Robust Routing 5 Conclusions and Perspectives

4 Routing Optimization (RO) in current network scenario Current network scenario: network convergence is a tangible reality heterogeneous services make network traffic uncertain and highly variable new kinds of network anomalies increase this traffic uncertainty

5 Routing Optimization (RO) in current network scenario Current network scenario: network convergence is a tangible reality heterogeneous services make network traffic uncertain and highly variable new kinds of network anomalies increase this traffic uncertainty Routing performance under all possible network situations: expected traffic variations: routing optimization for expected traffic unexpected traffic variations (Anomalies): minimize impact on other QoS services between anomalies detection and resolution

6 Routing Optimization (RO) in current network scenario Current network scenario: network convergence is a tangible reality heterogeneous services make network traffic uncertain and highly variable new kinds of network anomalies increase this traffic uncertainty Routing performance under all possible network situations: expected traffic variations: routing optimization for expected traffic unexpected traffic variations (Anomalies): minimize impact on other QoS services between anomalies detection and resolution Major challenge:...how to optimize routing for an unknown traffic demand?

7 RO in current scenario: a challenging task Sources of Demands Variation Daily Periodic Usage Patterns Unexpected Events Equipment Failures Network Attacks External Routing Changes Flash Crowds Spontaneous Services (P2P)

8 RO in current scenario: a challenging task Sources of Demands Variation Daily Periodic Usage Patterns Unexpected Events Equipment Failures Network Attacks External Routing Changes Flash Crowds Spontaneous Services (P2P)

9 RO in current scenario: a challenging task Sources of Demands Variation Daily Periodic Usage Patterns Unexpected Events Equipment Failures Network Attacks External Routing Changes Flash Crowds Spontaneous Services (P2P)

10 RO in current scenario: a challenging task Sources of Demands Variation Daily Periodic Usage Patterns Unexpected Events Equipment Failures Network Attacks External Routing Changes Flash Crowds Spontaneous Services (P2P)

11 RO in current scenario: a challenging task Sources of Demands Variation Daily Periodic Usage Patterns Unexpected Events Equipment Failures Network Attacks External Routing Changes Flash Crowds Spontaneous Services (P2P)

12 RO in current scenario: a challenging task Sources of Demands Variation Daily Periodic Usage Patterns Unexpected Events Equipment Failures Network Attacks External Routing Changes Flash Crowds Spontaneous Services (P2P)

13 RO in current scenario: a challenging task Sources of Demands Variation Daily Periodic Usage Patterns Unexpected Events Equipment Failures Network Attacks External Routing Changes Flash Crowds Spontaneous Services (P2P)

14 RO in current scenario: a challenging task Sources of Demands Variation Daily Periodic Usage Patterns Unexpected Events Equipment Failures Network Attacks External Routing Changes Flash Crowds Anomalies Spontaneous Services (P2P)

15 RO in current scenario: a challenging task Sources of Demands Variation Daily Periodic Usage Patterns Unexpected Events Equipment Failures Network Attacks External Routing Changes Flash Crowds Anomalies Spontaneous Services (P2P)

16 Additional source of uncertainty traffic demands are rarely available: direct traffic measurements (e.g. CISCO Netflow) seldom available for every ingress/egress links. direct traffic measurements causes router overloading. measurements are generally conducted at a higher level of aggregation, rendering the traffic process non-observable.

17 Examples of Variations in Real Data Correlated volume changes 8000 Link 47 Link 58 Link 104 Link 126 Link Load (unknown unit) Unidentifiable variations Time (min) (a) Traffic patterns in a large Tier-2 network.

18 How to tackle the problem? Proactive approach: Robust Routing Techniques consider traffic uncertainty within the routing optimization. Sources of Traffic Variation Expected variations and small load changes under normal operation Daily Traffic Patterns Unexpected Events Emerging Challenges External Routing Modifications Flash Crowd Events Equipment Failures Multi Hour Robust Routing Network Attacks (DoS/DDoS) Spontaneous Overlay Services (P2P) Reactive Robust Routing Anomaly Detection and Localization

19 How to tackle the problem? Reactive approach: Anomaly Detection and Localization anomaly detection and localization from simple measurements. parsimonious traffic modeling to overcome the non-observability problem. Sources of Traffic Variation Expected variations and small load changes under normal operation Daily Traffic Patterns Unexpected Events Emerging Challenges External Routing Modifications Flash Crowd Events Equipment Failures Multi Hour Robust Routing Network Attacks (DoS/DDoS) Spontaneous Overlay Services (P2P) Reactive Robust Routing Anomaly Detection and Localization

20 How to tackle the problem? Combined approach: Reactive Robust Routing robust routing for usual network operation. anomaly detection/localization for the unexpected events. robust routing reconfiguration to minimize network congestion. anomalies end detection (automation of the routing process). Sources of Traffic Variation Expected variations and small load changes under normal operation Daily Traffic Patterns Unexpected Events Emerging Challenges External Routing Modifications Flash Crowd Events Equipment Failures Multi Hour Robust Routing Network Attacks (DoS/DDoS) Spontaneous Overlay Services (P2P) Reactive Robust Routing Anomaly Detection and Localization

21 Outline 1 Introduction to the problem 2 A proactive approach: the Robust Routing 3 A reactive approach: Anomaly Detection/Localization 4 A combined approach: the Reactive Robust Routing 5 Conclusions and Perspectives

22 A Proactive Approach The Boy Scout motto:

23 A Proactive Approach The Boy Scout motto: Be Prepared!!! Stable Robust Routing robust Traffic Engineering techniques. consider traffic uncertainty in advance (robustness).

24 The Stable Robust Routing (SRR) Problem formulation Consider the following network scenario: Network topology: n nodes. L = {1,..., r} links with capacities in C = (c 1, c 2,...,c r ). N = {OD 1,.., OD m=n(n 1) } Origin-Destination traffic flows. Routing matrix R = {r l,k;l=1..r,k=1..m }, 0 r l,k 1. P(k) = {set of paths p for OD k }, k = 1..m. Traffic OD flows d = {d i,j;i,j=1..n }; d = {d k, k=1..m } Links traffic (aggregated ODs traffic) y = {y l, l=1..r } y(t) = R d(t) t.

25 The Stable Robust Routing Multipath Routing Optimization Given d, C, R and P(k), RO seeks to optimally balance d in P(k) to minimize some performance criterion:

26 The Stable Robust Routing Multipath Routing Optimization Given d, C, R and P(k), RO seeks to optimally balance d in P(k) to minimize some performance criterion: u max (C, d, R) = max l {1...r} k r l,k d k c l y = max l l {1...r} c l

27 The Stable Robust Routing Multipath Routing Optimization Given d, C, R and P(k), RO seeks to optimally balance d in P(k) to minimize some performance criterion: u max (C, d, R) = max l {1...r} k r l,k d k c l y = max l l {1...r} c l xp k, 0 xp k 1, fraction of d k in p P(k) xl k, 0 xl k 1, fraction of d k in l p

28 The Stable Robust Routing Multipath Routing Optimization Given d, C, R and P(k), RO seeks to optimally balance d in P(k) to minimize some performance criterion: u max (C, d, R) = max l {1...r} k r l,k d k c l y = max l l {1...r} c l xp k, 0 xp k 1, fraction of d k in p P(k) xl k, 0 xl k 1, fraction of d k in l p minimize u max subject to: P xp k 1 k N p P(k) P xp k xl k (l, k) (L, N) p P(k), l p P xl k.d k u max c l l L k N xp k, xk l 0 (l, k) (L, N), p P(k) u max 1

29 The Stable Robust Routing Traffic uncertainty set and routing optimization Traffic d is uncertain belongs to a polyhedral uncertainty set D: D = { d R m, R d y max, d 0 } minimize u max subject to: xp k 1 k N p P(k) xp k xl k k N, l L xl k.d k u max c l l L, d D p P(k), l p k N xp k, xl k 0 l L, p P(k), k N u max 1 Solved by a column and constraints generation method [BAK-05]. [BAK-05] W. Ben-Ameur and H. Kerivin, Routing of Uncertain Traffic Demands, Optimization and Engineering, 2005.

30 Trade off in the size of the uncertainty set D D A D B = { d R m, R d y 4:00 18:00 max, d 0 } = { d R m, R d y 18:00 4:00 max, d 0 } Historical Routing Robust Routing B Robust Routing A Maximum Link Utilization :00 7:00 9:00 11:00 13:0015:00 17:00 19:0021:00 23:00 1:00 3:00 Time (hours) Remark: a single stable routing scheme for long-time periods results in sub-optimal performance.

31 The Multi-Hour Robust Routing (MHRR) D t D 1 D 2 β 1 00:00 12:00 time β 2 time 24:00 β 3 Idea: divide the uncertainty set to reduce cost (adapt the set) and consider a SRR configuration for each sub-set.

32 The Multi-Hour Robust Routing (MHRR) D t D 1 D 2 β 1 00:00 12:00 time β 2 time 24:00 β 3 Idea: divide the uncertainty set to reduce cost (adapt the set) and consider a SRR configuration for each sub-set. Partitioning hyperplane α.d = β.

33 The Multi-Hour Robust Routing (MHRR) D t D 1 D 2 β 1 00:00 12:00 time β 2 time 24:00 β 3 Idea: divide the uncertainty set to reduce cost (adapt the set) and consider a SRR configuration for each sub-set. Partitioning hyperplane α.d = β. The optimal division is generally NP-complex.

34 The Multi-Hour Robust Routing (MHRR) D t D 1 D 2 β 1 00:00 12:00 time β 2 time 24:00 β 3 Idea: divide the uncertainty set to reduce cost (adapt the set) and consider a SRR configuration for each sub-set. Partitioning hyperplane α.d = β. The optimal division is generally NP-complex. However, when the direction (α) is known, it can be approximately solved [BA-07]: in our case, we take the time direction [CV-07]. [BA-07] W. Ben-Ameur, Between Fully Dynamic Routing and Robust Stable Routing, DRCN, [CV-07] P. Casas and S. Vaton, An Adaptive Multi-Temporal Approach for Robust Routing, EuroFGI WIPQoS&TC, 2007.

35 Some examples in Abilene Maximum Link Utilization Historical Routing 0.45 Stable Robust Routing A Stable Robust Routing B M H Robust Routing Maximum Link Utilization Historical Routing Stable Robust Routing A Stable Robust Routing B M H Robust Routing :00 23:00 1:00 3:00 5:00 7:00 9:00 11:00 13:00 17:00 19:00 21:00 Time (hours) (a) Expected daily behaviour 0 5:00 7:00 9:00 11:00 13:00 15:00 17:00 19:00 21:00 23:00 1:00 3:00 Time (hours) (b) Anomalous unexpected event Observation: the MHRR outperforms the SRR, but depends on a rough knowledge of the daily uncertainty set (we use it to improve routing for the expected traffic behavior in a robust fashion).

36 Outline 1 Introduction to the problem 2 A proactive approach: the Robust Routing 3 A reactive approach: Anomaly Detection/Localization 4 A combined approach: the Reactive Robust Routing 5 Conclusions and Perspectives

37 Anomaly Detection/Localization The Problem Objectives: detect and localize an abrupt change in traffic demand d(t) from link load measurements y(t) = R d(t). d(t) is seldom available. Conversely, link load measurements are highly spread. Problem: d(t) is non-observable directly from y(t). ill-posed problem r << m.

38 Anomaly Detection/Localization Proposed methodology [FNS-07] Stochastic parsimonious linear model for anomaly-free traffic demand d(t) The anomaly-free is considered as a nuisance parameter. Hypothesis testing to detect/localize an anomaly in the traffic residuals (after removing the modeled traffic) [FNS-07] L. Fillatre, I. Nikiforov and S. Vaton, Détection Localisation Séquentielle d Anomalies Volumiques dans un Réseau, GRETSI, 2007.

39 Anomaly Detection/Localization Stochastic Traffic model The order of increasing anomaly-free OD flows remains constant during long-time periods. The values of the ordered OD flows can be decomposed over a spline-basis with a small number of components. d(t) 1200 λ k(t) k time t (min) Small flows 140 Large flows Medium-size flows Approximation of real OD flows by the spline-based model

40 Anomaly Detection/Localization Stochastic Traffic model Model for the anomaly-free traffic: d(t) = Sµ(t) + ξ(t) ξ(t) is a white Gaussian noise with covariance matrix Σ(t). S = (s 1 s 2... s q ), is a splines basis that describes the traffic spatial distribution. µ(t) = (µ 1 (t)... µ q (t)) T R q, with q << m. the coefficients µ k (t) describe the anomaly-free intensity variations.

41 Anomaly Detection/Localization Stochastic Traffic model Anomaly-free model for y(t): y(t) = Hµ(t) + ζ(t) H = RS R r q, small n o columns easy to retrieve µ(t) from y(t). ζ(t) s covariance is estimated from samples. the anomaly-free traffic is eliminated by projecting y(t) into the null space of H.

42 Anomaly Detection/Localization Detection/localization of volume anomaly at time t 0 as a multiple hypothesis testing: H 0 = {the OD flows are anomaly-free} H j = t 0 =1.. Hj t 0, with H j t 0 = {the j-th OD flow presents an anomalous traffic from an unknown change time t 0 } compute (T,ν), T is the alarm time at which a ν-type change (ν {1, 2,...,m}) is detected and localized.

43 Anomaly Detection/Localization Detection/localization is conducted on the residuals obtained from link load observations, after filtering the anomaly-free traffic. Optimal recursive algorithm with well-established optimality properties in terms of detection delay and false alarm rate [IN-00]. Optimal trade-off between the worst case of the average detection delay (E(T )), the false alarm rate and the false localization probabilities. Multiple recursive detection and decision functions. [IN-00] I. Nikiforov, A Simple Recursive Algorithm for Diagnosis of Abrupt Changes in Random Signals, IEEE Trans. on IT, 2000.

44 Anomaly Detection/Localization Another example in Abilene gt(i, 0) Anomaly begins st(i) Alarm on OD flow 87 Level of alarm Time t (min) Time t (min) (a) Recursive detection functions (b) Decision functions Figure: Typical realizations of decision functions in Abilene.

45 Outline 1 Introduction to the problem 2 A proactive approach: the Robust Routing 3 A reactive approach: Anomaly Detection/Localization 4 A combined approach: the Reactive Robust Routing 5 Conclusions and Perspectives

46 The Reactive Robust Routing Combination of the reactive and the proactive approaches [CFS-08]. MHRR to handle usual-traffic and small changes in traffic demands in a robust and efficient way. Anomaly Detection/Localization algorithm to deal with unexpected events. Exploits the localization ability to compute an adapted SRR after the detection, based on an expansion of the uncertainty set [PLS-08]. Detects the end of the anomaly (if applicable) and takes back the MHRR configuration. [CFV-08] P. Casas, L. Fillatre and S. Vaton, Multi-Hour Robust Routing and Fast Load Change Detection for Traffic Engineering, IEEE ICC, [PLS-08] P. Casas, L. Fillatre and S. Vaton, Robust and Reactive Traffic Engineering for Dynamic Demands, EuroNGI, 2008.

47 Routing reconfiguration y i max + θr i,k y h max + θr h,k D : Before the anomaly D : After the anomaly y j max + θr j,k before the anomaly: D = {R. d y max }

48 Routing reconfiguration y i max + θr i,k y h max + θr h,k D : Before the anomaly D : After the anomaly y j max + θr j,k before the anomaly: D = {R. d y max } { θ = anomaly in OD flow k: d θ.δk = d + θ, δ k = (δ 1,k,.., δ m,k ) T, δ i,k = I i=k

49 Routing reconfiguration y i max + θr i,k y h max + θr h,k D : Before the anomaly D : After the anomaly y j max + θr j,k before the anomaly: D = {R. d y max } { θ = anomaly in OD flow k: d θ.δk = d + θ, δ k = (δ 1,k,.., δ m,k ) T, δ i,k = I i=k after the anomaly: D = {R. d y max + R. θ}

50 Routing reconfiguration y i max + θr i,k y h max + θr h,k D : Before the anomaly D : After the anomaly y j max + θr j,k before the anomaly: D = {R. d y max } { θ = anomaly in OD flow k: d θ.δk = d + θ, δ k = (δ 1,k,.., δ m,k ) T, δ i,k = I i=k after the anomaly: D = {R. d y max + R. θ} robust routing reconfiguration, using D as the uncertainty set (expansion of the uncertainty set).

51 Anomalies end detection The detection algorithm focuses in the abnormal OD flow k after the routing reconfiguration. Two simple hypotheses: H a H b : {the OD flow k is abnormal} : {the OD flow k presents a usual behavior} simple Neyman-Pearson s test (most powerful test for 2 simple hypotheses): (z(t)) = log f 0(z(t)) f k (z(t)) h > 0 H b The decision threshold h is fixed according to the prescribed false alarm probability.

52 Reactive Robust Routing evaluation Multi-Hour Robust Routing, adapted to the expected traffic demand. Anomaly detection/localization in OD flow k = 63 at time t = Robust routing reconfiguration. Detection of the anomaly s end at time t = Return to the MHRR configuration.

53 Reactive Robust Routing evaluation (z(t)) 80 (z(t)), k = k-th anomaly ends 20 Maximum Link Utilization 0.8 HR 0.7 SRR A SRR B MHRR 0.6 RRR replacements Time t (min) (a) Neyman-Pearson Test (z(t)) Time t (min) (b) Performance evaluation

54 Reactive Robust Routing evaluation 0.8 Maximum Link Utilization HR SRR A SRR B MHRR RRR Time t (min) between 20% and 50% of utilization improvement w.r.t. the stable robust routing approach

55 Outline 1 Introduction to the problem 2 A proactive approach: the Robust Routing 3 A reactive approach: Anomaly Detection/Localization 4 A combined approach: the Reactive Robust Routing 5 Conclusions and Perspectives

56 Conclusions of this work Robust TE techniques can not deal by themselves with traffic uncertainty, but they should be used in current traffic scenario. We have shown that reactive and proactive techniques can work together to improve the treatment of large volume anomalies. Decision theory offers powerful and promising techniques for the automation of diagnosis in the field of network anomaly detection (still in the very early stage).

57 Future directions exploit the linear parsimonious traffic model for further studies (d(t) becomes observable, many problems can be solved). use this traffic model for the traffic matrix estimation problem, tracking of traffic demands from simple link measurements, etc. consider other performance criterion for the optimization problem.

58 Thank You for Your Attention!! Remarks & Questions?