An Aggregation Technique for Traffic Monitoring. Kenjiro Cho, Ryo Kaizaki, and Akira Kato

Size: px

Start display at page:

Download "An Aggregation Technique for Traffic Monitoring. Kenjiro Cho, Ryo Kaizaki, and Akira Kato {kjc,kaizaki,kato}@wide.ad.jp"

Philomena Walton
8 years ago
Views:

1 An Aggregation Technique for Traffic Monitoring Kenjiro Cho, Ryo Kaizaki, and Akira Kato motivation for long-term monitoring flow-based monitoring needs predefined rules problems to track new protocols, DoS attacks an adaptive monitoring tool is needed! to use for trouble detection rough usage report in real-time for trouble-shooting should be able to provide detailed info desire to have tunable granularity flow-granularity time-granularity glance at coarse-grained traffic graph look into finer-grained graphs when necessary

2 aguri aggregation-based traffic profiler for near real-time, long-term, wide-area traffic monitoring adaptive spatially and temporally store fine-grained summary in real-time change granularity at display time aggregation to change granularity adaptive requires no rule an effective tool to detect anomalies (e.g., DoS attacks) to track long-term trend can be used for traffic control idea of aggregation-based profiling aggregate flows until volume becomes non-negligible report only aggregates to produce compact/adaptive summary.../.../ 1.1/ /16 1.1/ / / / / / / /

granularity at display time aggregation to change granularity adaptive requires no rule an effective tool to detect anomalies (e.g., DoS attacks) to track long-term

3 summary profile concise summary for operators 4 separate profiles src/dst address/protocol effective to detect DoS attacks reports byte-count used by each aggregate addresses are aggregatable also applicable to protocol/port to identify port ranges 32bit key by concatenating IPversion:proto:port spatial (flow) aggregation if resources (e.g., CPU, memory) are infinite profile all address occurences, then, aggregate approximation for real-time monitoring with finite resources use a variant of LRU when reclaiming a node, aggregate the counter to parent advantage counter values are never lost though resolution is reduced

4 temporal aggregation produce summary of summaries by reading its own outputs ex. 1-hour-long summary out of 6 1-minute-long summaries 2 types of summaries initial summary directly produced from non-aggregated source (e.g., from packet traces) derivative summary produced from summaries a summary with arbitrary resolution can be created from a set of archived summaries in different time scales with different aggregation threshold archiving and visualization summary output is in a plain text format easily processed by scripts archiving a script is periodically invoked to generate and archive summaries in different time scales e.g., hourly/daily/monthly/yearly summaries visualization scripts convert summaries to graphical images

5 application for traffic control once aggregates are identified and profiled, record can be used for traffic control many possible approaches e.g., rate-limiter at firewall aguritcm combines aggregation profiler with preferential packet dropping identifies aggregates whose traffic volume is more than fairshare probabilistically raises drop precedence for those aggregates related work MRTG[Oet98], RRDtool[Oet] automatically aggregate time-series data into average over time we combine temporal aggregation with spatial aggregation NeTraMet[Bro97], FlowScan[Pon] traditional flow-based monitoring tools require predefined rules have problems to cope with unknown protocols/dos attacks RED penalty-box[ff99], pushback[mbf+1] approaches from congestion control use RED drop history as samples similar to ours in the concept our approach traffic profiler to monitor and report network not only under congestion but all the time a network point to be protected is also to be monitored combined solution for performance and simplicity visible monitoring output could be advantageous to deployment

dropping identifies aggregates whose traffic volume is more than fairshare probabilistically raises drop precedence for those aggregates related work MRTG[Oet98], RRDtool[Oet] automatically aggregate

6 ± ÍÊÁ¹½º¼ Ë Ø Â Ò ¼ ½ ¼¼ ¼¼ ¾¼¼½ ±±ËØ ÖØÌ Ñ Ö ½¼ ½¼¼º¼¼±µ Ø ½¼ ¾ ¼º ±»½¼¼º¼¼±µ ¼º¼º¼º¼»¼ ½ ½º ±»½º ±µ ¼º¼º¼º¼»¾ ½ ½ ¾ ½º ±» º½ ±µ ½¾ º¼º¼º¼»½ ½ ¾½ ½º ±»½½º¼ ±µ ½ º¾ º¼º¼»½ ½ ¾¼ ½º ±µ ½ º¾ º¾½º¾½ ¾ ¾¾¼ ¾º ½±» º¼ ±µ ½ º¾ º½¾ º¼»½ ½¾ ½ ½º¾¼±µ ½ º¾ º½¾ º½ ¼ º ¾±µ ½ º¾ º¾¼¾º½¾ ½ ½ ½º ±»¾ º ±µ ½ ¾º¼º¼º¼» º½ ±»¾¼º¾ ±µ ½ ¾º½¼º¼º¼»½ ¼ º¼ ±»½ º½ ±µ ½ ¾º½¼º¼º¼»½ ½½ ¾ ½ ½½º¼ ±µ ½ ¾º½¼º½º¾ ¾¼ ¾ ½º ±µ ½ ¾º½¼º½ º½ ¾ ¾º ±» º ±µ ½ º º¼º¼»½½ ½ ½º ±»½º ±µ ½ º½½ º¼º¼»½ ½ ½ ¾º ±» º ±µ ½ º¼º¼º¼» ¾ ¼ ¾º ±µ ½ º º½¾º º ±» ¾º ¾±µ ½ º º¼º¼»½¾ ¾ ¾ ¾º ±» º¼ ±µ ½ º½¼ º½ º¼»¾¼ ¼ º½ ±µ ½ º½¼ º½ º½ ¾ ¼ º½ ±µ ½ º½¼ º½ º½ ¼¾¾ ¼ º ±µ ½ º½¼ º½ º½ ¾ ½ ½ ½ º¼ ±µ ½ º½¼ º½ º½ º¾ ±µ ½ º½¼ º½ º½ ½ ¾¾ ½º ±µ ½ º½¼ º½ º½ ½½½ ½ ½º¼ ±» º ±µ ½ ¾º¼º¼º¼» ¼ º ¾±» º ¼±µ ½ º¼º¼º¼» ½ ¾ ½º ±µ ½ º½¼ º¾ ½º ¾ ¾ ¼ ¾º¾ ±µ ½ º½ ¼º¾½ º¾ ¾ ¾ ¾º ±»¾º ±µ ¾¼ º¼º¼º¼» Ø ¾º ¾± ½ ½½»½ µ ±ÄÊÍ aguri implementation a user program on UNIX input modules summary input/pcap interface/aguritcm input create a 4 tuple (tree, key, prefixlen, count) profiler engine does aggregation-based profiling summaries summary input plot generator plot output tcpdump file captured packets aguritcm in kernel pcap input aguritcm input tree-based profiler next period aggregate output control HUP signal summary output input module profiler engine summary output (1) ±± Ò Ì Ñ Ë Ø Â Ò ¼ ½ ¼¼ ¼ ¾¼¼½ ± Ú Ê Ø ½ º¼ Å Ô

¾º½¼º½ º½ ¾ ¾º ±» º ±µ ½ º º¼º¼»½½ ½ ½º ±»½º ±µ ½ º½½ º¼º¼»½ ½ ½ ¾º ±» º ±µ ½ º¼º¼º¼» ¾ ¼ ¾º ±µ ½ º º½¾º º ±» ¾º ¾±µ ½ º º¼º¼»½¾ ¾ ¾ ¾º ±» º¼ ±µ ½ º½¼ º½ º¼»¾¼ ¼ º½ ±µ ½ º½¼ º½ º½ ¾ ¼ º½ ±µ ½ º½¼ º½

7 ½¼ ¼ ½¼¼º¼¼±µ Ô ÔÖÓØÓ ÖÔÓÖØ ¼º¼ ±»½¼¼º¼¼±µ ¼»¼ ¼ ¼ ¾ ¼ ¾ ¾º ±» º ±µ ¼» ¼ ½ ¾ ½º ±» º½ ±µ ¼»¼ ¼ º½½±» º ±µ ¼» ½ º¾ ±µ ¾¼ ¾ ¾½ º ±µ ¼ ½¼ ¾ ½º¼½±»½º¼½±µ ½¼¾» ½ ½ ½º ¾±»¾º ±µ ½¾ ¼» ½ ¼ ½ ½º ¾±»½º ¾±µ ½¾ ¼» ½ ¾ ½º ±»½º ±µ ½» º ±» º ±µ ¾¼» ½ ¼¼ ½º ±µ ½¾ ¾ ½º¾¾±»½º¾¾±µ ¾»½ ½½½ ½º¼ ±µ ½ Ø ¼º ¼± ½¼»½ ¼ µ ±ÄÊÍ summary output (2) aggregation algorithm any approximation introduces excess aggregation errors (1) part of counter value could be aggregated to ancestors (2) entry close to the aggregation threshold could be removed unavoidable for derivative summaries but any approximation can detect non-negligible entries our algorithm PATRICIA tree variable key length well understood a fixed number of nodes a variant of LRU for leaf management aggregation exemption threshold

aggregation errors (1) part of counter value could be aggregated to ancestors (2) entry close to the aggregation threshold could be removed unavoidable for derivative summaries but any

8 1-day dst address plot graph 3 dst address 25 2 Traffic (Mbps) :: 6:: 9:: 12:: Time 15:: 18:: 21:: 4/13 total.../ / / / / / / / / / / / /7 1-day src protocol plot graph 35 ip:proto:srcport 3 25 Traffic (Mbps) /1 total 4:6:8 4:/3: 4:6:6144/6 4/11 4/13 4:6:/3 4:6:49152/2 4:17:53 4:6:/ 4/14 4/15 Time 4/16 4:6:248/5 4:6:/9 4:6:124/6 4:6:/5 4/18 4:6:32768/1 4:6:3128 4:6:2 4:6:/11 4/19 4/2

../7 1-day src protocol plot graph 35 ip:proto:srcport 3 25 Traffic (Mbps) 2 15 1 5 4/1 total 4:6:8 4:/3: 4:6:6144/6 4/11 4/13

9 zooming into April 17th DoS attack found! 1 ip:proto:srcport Traffic (Mbps) :: 6:: 9:: 12:: Time 15:: 18:: 21:: 4/18 total 4:6:8 4:/3: 4:1: 4:6:/ 4:6:124/8 4:6:128/8 4:6:1536/8 4:6:/9 4:6:1792/8 4:6:/5 4:6:/3 4:6:372/8 4:6:188 4:6:1472/1 4:17:27648 traffic density graph

10 Ð Ò Ø Ó Ô Ø Ó Ö Ö Ø Ô µ ØÖ ½ ¾ ¾¼º ¾Å ½ ½ ½ ¾½º½¾Å ¾ ¾½ ¾ ¾¾º Å ¼ ½ ½ ¼ ¾ º Å ¼¼ ½ ½ ¼ ¾ º¾ Å ¾ º ¾Å ½½ ¼ ½ º½¼Å ¼ È ¾Ì ½ Û Ö ½¾ ½¾ È ¾Ì ¾ Û Ö ¾½ ¾½ evaluation trace-driven evaluation 2 traces from WIDE backbone #1: trans-pacific link #2: domestic link to IX ¾ ¾ ½ º ½Å ¼¼ aggregation accuracy accuracy isn t important but need a quantitative measure to evaluate the algorithm performance trade-off between accuracy and performance distortion index quantify difference of 2 subtrees traditional methods in graph theory can t be used distortion index isn t perfect but usable distance k k i j i j T1[k] T2[k] ½¾ Û Ö ½¾ ½¾ ¾

isn t important but need a quantitative measure to evaluate the algorithm performance trade-off between accuracy and performance distortion index quantify

11 ¼¼ ½ Ü ¼¼ ¼ Ü ¼ ½ Ü ¼Ü ¼ Ü¾ Ü ¼ ØÝÔ ½ Øµ ¾Ò µ ¾Ò µ Ö µ Ö µ Òºµ Ö ¼º¼ ¼º¼ ¼º¼ ½ ¼º¼ ¼º¼ ½ Ø ¼º¼ ¼º¼ ¾ ¼º¼ ½¾ ¼º¼ ¼º¼ ½ Ö ¼º¼ ¼º¼¼ ¼º¼¾½¼ ¼º¼¾¼ ¼º¼¾½ ¾ Ø ¼º¼ ¼º¼½½ ¼º¼½ ¼ ¼º¼¾¼¾ ¼º¼¾¼ ¾ distortion with varying tree size trace #1 (src) (dst) sec (LRU) 5sec (LRU) 1sec (LRU) 6sec (LRU/AE) 5sec (LRU/AE) 1sec (LRU/AE) sec (LRU) 5sec (LRU) 1sec (LRU) 6sec (LRU/AE) 5sec (LRU/AE) 1sec (LRU/AE).2.2 distortion index.15.1 distortion index the number of nodes in a tree the number of nodes in a tree trace #2 (src) (dst) sec (LRU) 5sec (LRU) 1sec (LRU) 6sec (LRU/AE) 5sec (LRU/AE) 1sec (LRU/AE) sec (LRU) 5sec (LRU) 1sec (LRU) 6sec (LRU/AE) 5sec (LRU/AE) 1sec (LRU/AE).2.2 distortion index.15.1 distortion index the number of nodes in a tree the number of nodes in a tree distortion in summary generations effect of repeated aggregation

15.1 distortion index.15.1.5.5 32 64 128 256 512 124 248 496 8192 32 64 128 256 512 124 248 496 8192 the number of nodes in a tree the number of nodes in a tree trace #2 (src) (dst).3.25 6sec (LRU) 5sec (LRU) 1sec (LRU) 6sec (LRU/AE) 5sec (LRU/AE) 1sec (LRU/AE).

12 performance with varying tree size 3 trace #1 trace #2 25 throughput (pps) the number of nodes in a tree evaluation results a variant of LRU produces decent summaries heuristic is effective when nodes are insufficient 128 or 256 nodes per tree is ok even for backbone traffic performance is good enough 25Kpps with 256 nodes (19Kpps required to process 64B packets at 1Mbps) (PC router performacne: 8Kpps) algorithm is insensitive to traffic variations, period length, summary generations

insufficient 128 or 256 nodes per tree is ok even for backbone traffic performance is good enough 25Kpps with 256 nodes (19Kpps

13 application for traffic control targeted for a protection measure against DoS attacks diffserv components as building blocks local mechanism combine profiler with diffserv marker traffic profiling and marking in kernel neither class configuration nor classifier rule is needed aguritcm profiler based marking if aggregate uses more than fairshare degrades the drop precedence level RIO preferentially drops packets under long-term congestion profile info aguritcm input interface forwarding RIO dropper on output queue output interface

rule is needed aguritcm profiler based marking if aggregate uses more than fairshare degrades the drop precedence level RIO

14 Ô marking algorithm counter values are aged at the end of profiling period to smooth out marking probability fairshare (f:fairshare, c:count, n:number of aggregates) È marking probability Ò µ ¼ ÓØ ÖÛ aguritcm implementation diffserv traffic conditioner module for ALTQ preliminary tests show resilience in the face of misbehaving flows rough fairness among aggregates test config aggregate-1 (TCP) aggregate-2 (TCP) aggregate-3 (UDP) S1 S2 S3 1baseTX (1Mbps) R 1baseT (1Mbps) D1 D2 D3

traffic conditioner module for ALTQ preliminary tests show resilience in the face of misbehaving flows rough fairness

15 aguritcm effects without traffic control (UDP monopolizes bandwidth) aggregate 3 aggregate 2 8 aggregate 1 Traffic (Mbps) Time (sec) with aguritcm (no starvation/improved fairness) 8 6 aggregate 3 aggregate 2 aggregate 1 Traffic (Mbps) Time (sec) conclusion aguri is an adaptive traffic profiler for near real-time, long term, wide area monitoring has been used for monitoring WIDE backbone since Feb 21 aguritcm combines profiler with marker a protective measure against DoS attacks preliminary test results look promising needs further investigation and parameter tuning aguri is available from

real-time, long term, wide area monitoring has been used for monitoring WIDE backbone since Feb 21 aguritcm combines profiler with marker a protective measure against

HowPros and Cons of Owning a Home-Based Business

HowPros and Cons of Owning a Home-Based Business ÄØ Ø ÊÚ ÓÒ ÅÖ ¾½ ¾¼¼½ ÓÑÑÒØ ÏÐÓÑ ÖÑ Ò ÅÒÖÐ ÁÒÒØÚ ØÓ ÅÒÔÙÐØ Ø ÌÑÒ Ó ÈÖÓØ Ê ÓÐÙØÓÒ Ú ÀÖ ÐÖ ÌÖÙÒ ÓÖ ËÓÒÝÓÒ ÄÑ Ï ØÒ º ÕÙØ º ÖÓÚØ Ëº ÒÒ Åº ÖÒÒÒ Àº Ó º ÓÛÖÝ ÈºÙÐÖ Êº ÀÒÐ Âº ÀÖ ÐÖ º ÄÑÒÒ Åº ÅØÐÐ ÁºÈÒ ºÊ ÑÙ Ò