網路流量量測與分析 楊竹星 國立中山大學資訊工程系

Transcription

1 路 流 量 量 立

2 Outline Introduction NetFlow Netflow Overview Netflow Architecture Netflow Formats Netflow Feature Acceleration Netflow Deployment AAA Our solutions System Architecture Enhance Flow-Tools Protocol Using Dynamic Ports Conclusion and Future Work

3 Introduction Goals Service providers must have access to in-depth info about their networks A complete view of current use Understand the behavior of their networks Network Problem Determination and Analysis Network security attack detection and prevention Detailed network usage history reports Analytical tools to analyze and predict usage trends Plan for network deployment and expansion Etc. Usage-based Billing, SLA monitoring

4 Introduction Challenges Capturing Characteristics How to capture traffic characteristics from high-speed, high volume networks (Mbps Gbps Tbps)? Analysis How to analyze and generate data needed quickly? Evolving network applications Streaming media (Windows Media, Real, Quicktime) P2P traffic Network Security Attacks Log Generation & Storage What kind of information to save to perform various/long-term analysis? How to minimize storage requirements?

5 Tools Taxonomy IN OUT 2 Data Collect RTFM RMON Netflow SNMP PacketDump Analysis Tools cflowd Flow-tools Flowscan Panoptis MINDS Traffic Engineering, User Monitoring, Billing. DDOS, Virus, Worms

6 Data Collection SNMP Data Simple Network Management Protocol (SNMP) Router CPU utilization, link utilization, link loss, Collected from every router/link every few minutes Applications Detecting overloaded links and sudden traffic shifts Measuring link utilization Advantage Open standard, available for every router and switch Disadvantage Coarse granularity, both spatially and temporally Version consistency

7 Data Collection Flow-Level Traces Flow monitoring (e.g., Cisco Netflow) Measurements at the level of sets of related packets Set of packets that belong together Source/destination IP addresses and port numbers Same protocol, ToS bits, Same input/output interfaces at a router (if known) Number of bytes and packets, start and finish times Applications Computing application mix and detecting DoS attacks Measuring the traffic matrix for the network Advantages Medium-grain traffic view, supported on some routers Disadvantages Not uniformly supported across router products Large data volume, and may slow down some routers

8 Data Collection Packet-Level Traces Packet monitoring IP, TCP/UDP, and application-level headers Collected by tapping individual links in the network Applications Fine-grain timing of the packets on the link Fine-grain view of packet header fields Advantages Most detailed view possible at the IP level Disadvantages Expensive to have in more than a few locations Challenging to collect on very high-speed links Extremely high volume of measurement data

9 Business Requirements How do I efficiently track network and application resource usage? How do I know if my customers are adhering to usage policy agreements? How do I account and bill for resources being utilized? How do I effectively plan to allocate and deploy resources most efficiently? How do I track customers to enhance marketing customer service opportunities?

10 Accounting What For? Network monitoring Network planning Security analysis Application monitoring and profiling User monitoring and profiling Traffic engineering Peering agreements Usage-based billing Destination sensitive billing

11 Accounting vs. Billing Steve SAP Accounting Application Billing Application Src Add Dest Add User Resource Steve SAP

12 Accounting Why? Baselining, Performance Network monitoring Application monitoring User monitoring Trends, statistics Deviation from normal History

13 Accounting Why? Network Design Capacity planning Traffic engineering Source Rome POP Paris POP ISP2 ISP3 Dest. Munich POP London POP

14 Accounting Why? Peering Agreements ISP

15 Outline Introduction NetFlow Netflow Overview Netflow Architecture Netflow Formats Netflow Feature Acceleration Netflow Deployment AAA Our solutions System Architecture Enhance Flow-Tools Protocol Using Dynamic Ports Conclusion and Future Work

16 NetFlow Enables Traffic Analysis and Monitoring for Network Planning Usage-Based Billing Router Feature Acceleration NetFlow statistics empowers users with the ability to characterize their IP data flows The who, what, where, when, and how much IP traffic questions are answered

17 NetFlow s Value NetFlow enables IP traffic flow analysis without probes Offers a rich data set to be mined for network management, traffic engineering, and valueadded service offerings (i.e. marketing data, personal NMS data) Increasing margins on existing Cisco infrastructure is possible and economical with NetFlow usage based billing

18 Flow-Based Analysis Seven Keys Define a Flow: 1. Source Address 2. Destination Address 3. Source Port 4. Destination Port 5. Layer 3 Protocol 6. TOS Byte (DSCP) 7. Input Interface NetFlow Data Exported

19 Outline Introduction NetFlow Netflow Overview Netflow Architecture Netflow Feature Acceleration Netflow Formats Netflow Deployment AAA Our solutions System Architecture Enhance Flow-Tools Protocol Using Dynamic Ports Conclusion and Future Work

20 NetFlow Components IOS Netflow FlowCollector Netflow Data Analyzer RMON Probe Network Planning Accounting/Billing Data Switching Data Export Data Aggregation Data Collection Data Filtering Data Aggregation Data Storage File System Management Data Presentation NFC Control and Configuration Partner Applications

21 NetFlow Component: IOS IOS RMON Probe Data Switching Data Export Data Aggregation

22 NetFlow Cache Tracks Flows A Flow is defined by Seven Characteristics: Source/Destination IP address pair Source/Destination application port pair IP Protocol Input Physical Interface Index IP Type of Service (ToS) byte Flows are unidirectional NetFlow is enabled on a per input-interface basis

23 NetFlow Feature Acceleration NetFlow Accelerates NetFlow Policy Routing (NPR) Router-based network data encryption Access Control Lists (ACL) RSVP In the future Network Address Translation (NAT) Committed Access Rate (CAR) Web Cache Control Protocol (WCCP) Others Availability of such acceleration will be announced on a feature-by-feature basis

24 NetFlow Data Record Usage Packet Count Byte Count Source IP Address Destination IP Address From/To Time of Day Start Timestamp End Timestamp Source TCP/UDP Port Destination TCP/UDP Port Port Utilization QoS Input Interface Port Output Interface Port Type of Service TCP Flags Protocol Next Hop Address Source AS Number Dest.. AS Number Source Prefix Mask Dest.Prefix Mask Application Routing and Peering

25 Router Based Aggregation AS Prefix Matrix Protocol Type Source Prefix Dest. Prefix

26 NetFlow Components: FlowCollecter IOS Netflow FlowCollector RMON Probe Data Switching Data Export Data Aggregation Data Collection Data Filtering Data Aggregation Data Storage File System Management

27 NetFlow FlowCollector Flow record reception Data volume reduction Filtering Aggregation Flexible thread language Flat file, binary, and/or compressed file storage File cleanup Solaris and HP-UX NetFlow FlowCollector Flow Consumer Applications

28 FlowCollector Aggregation Schemes Over 20 aggregation schemes From Call Detail Records for billing To AS information for statistics Many combinations in-between

29 Highlighted New Features in FlowCollector 3.0 Support for RBA export data 8 additional aggregation schemes Improved disk space management Configuration and Control API Autonomous Message Notification High availability process monitoring on hosting workstation

30 NetFlow Components: Data Analyzer IOS Netflow FlowCollector Netflow Data Analyzer RMON Probe Network Planning Accounting/Billing Data Switching Data Export Data Aggregation Data Collection Data Filtering Data Aggregation Data Storage File System Management Data Presentation NFC Control and Configuration Partner Applications

31 Network Data Analyzer NetFlow FlowCollectors NetFlow FlowAnalyzer Graphical display of NetFlow data Consumes from NetFlow FlowCollector(s) Time-based analysis & data sorting Histograms, Bar Charts, Piecharts Spreadsheet data export

32 Highlighted Features in Network Data Analyzer Search operations Address to Address transactions Address to Subnet transactions Subnet to Subnet transactions Address away from Address/Subnet transactions Multiple router or dataset selection DetailASMatrix aggregation & drilldown DNS address and AS number to name translation

33 Highlighted Features in Network Data Analyzer NetFlow Collector Control Traffic Matrix Statistics (TMS) Data Collection Control and Analysis View router-based aggregation schema data Router control for NetFlow and TMS

34 NetFlow FlowCollector Flow record reception Data volume reduction Filtering Aggregation Flat file, binary, and/or compressed file storage File cleanup Solaris and HP-UX NetFlow FlowCollector Applications

35 Network Data Analyzer NetFlow FlowCollectors NetFlow FlowAnalyzer Graphical display of NetFlow data Consumes from NetFlow FlowCollector(s) Time-based analysis ands data sorting Configure routers and FlowCollectors Histograms, bar charts, and pie charts Spreadsheet data export

36 Outline Introduction NetFlow Netflow Overview Netflow Architecture Netflow Formats Netflow Feature Acceleration Netflow Deployment AAA Our solutions System Architecture Enhance Flow-Tools Protocol Using Dynamic Ports Conclusion and Future Work

37 NetFlow Cache Tracks Flows A Flow is defined by Seven Characteristics: Source/Destination IP address pair Source/Destination application port pair IP Protocol Input Physical Interface Index IP Type of Service (ToS) byte Flows are unidirectional NetFlow is enabled on a per input-interface basis

38 Netflow Formats Version 1 Initial Version Not commonly used Version 8 Router based aggregation Available in 12.0(3)T, 12.0(3)S Version 5 Superset of Version 1 Added AS accounting Datagram Sequencing Commonly used Version 9 Configurable Flow Record Templates Version 7 Cat5K NFFC Only Not available in IOS Versions 2,3,4 and 6 were experimental

39 Cache Management & Data Export Header Sequence number Record count Version number Flow Record Flow Record NetFlow Cache Flow cache manager expires flows No traffic/long life/tcp flags/cache full/etc. Intelligent cache aging ensures cache entries are always available Distributed NetFlow Cache on VIPs Router exports groups of expired flows every second Export uses UDP datagrams with sequence numbers

40 Cache Management & Export NetFlow Cache Flow Entries Flow 1 Flow 2 Flow 3 Flow expired Cache full Timer expired Export Buffer UDP To Collector

41 Flow Management Rules for expiring NetFlow cache Entries Flows which have been idle for a specified time are expired and removed from the cache. (This is configurable) Long lived flows are expired and removed from the cache. Flows are expired after 30min, by default. As the cache becomes full the cache is intelligently purged. TCP connections which have been closed. That is, a FIN/RST has been received.

42 Data Export When does NetFlow export data? Flow datagrams are exported once per second, OR When a complete UDP datagram of flows is available Netflow Version Version 1 Version 5 Version 7 Version 8 Version 9 Number of Flow Records per Export Packet 24 flow records 30 flow records 27 flow records Variable Variable

43 Version 1 Version 1 is the initial NetFlow format supported on 11.1, 11.2, 11.3, 12.0 On by default No reason to use v.1 unless supporting a legacy collection system.

44 Version 1 Header Format Generic Time Netflow Version Number of flow sysuptime UTC time in sec Residential nanoseconds

45 Version 1 Flow Format Usage Packet Count Byte Count Source IP Address Destination IP Address From/To Time of Day Start sysuptime End sysuptime Source TCP/UDP Port Destination TCP/UDP Port Port Utilization Input ifindex Output ifindex Next Hop Address Application QoS Type of Service TCP Flags Protocol Routing

46 Version 1 Configuration router (config-if)#ip route-cache flow router (config)#ip flow-export destination router (config)#ip flow-export version 1 router (config)#ip flow-export source loopback 0 Show commands sh ip cache flow sh ip flow export

47 Version 5 Header Format Generic Time Check Netflow Version Number of flow sysuptime UTC time in sec Residential nanoseconds Flow Sequence From Engine Type Engine Id 0 for RP, 1 for VIP/LC VIP/LC slot number Added from version 1

48 NetFlow Data Record (V5) Usage Packet Count Byte Count Source IP Address Destination IP Address From/To Time of Day Start Timestamp End Timestamp Source TCP/UDP Port Destination TCP/UDP Port Port Utilization QoS Input Interface Port Output Interface Port Type of Service TCP Flags Protocol Next Hop Address Source AS Number Dest.. AS Number Source Prefix Mask Dest.Prefix Mask Application Routing and Peering

49 Version 5 Configuration router (config-if)#ip route-cache flow router (config)#ip flow-export destination router (config)#ip flow-export version 5 <peer-as origin-as> Optional configuration router (config)#ip flow-export source loopback 0 router (config)#ip flow-cache entries < > router (config)#ip flow-cache timeout

50 Version 5 Show Commands router#sh ip cache flow router#sh ip cache verbose flow router#sh ip flow export router#sh ip flow acceleration

51 Version 8 Router Based Aggregation, i.e. version 8 Enables router to summarize NetFlow data Reduces NetFlow Export data volume Decreases NetFlow Export bandwidth requirements Making collection easier

52 Version 8 Supported from 12.0(3)T, 12.0(3)S and 12.1 Onboard aggregation, the router maintains extra Netflow cache(s), just for accounting. Still needs the main cache (version 5) When flows expire from the main cache, they are added to each enabled aggregation cache Several aggregations can be enabled at the same time

53 Aggregations Currently 5 aggregations: ProtocolPort, AS, SourcePrefix, DestinationPrefix, Prefix 6 extra aggregations available in IOS 12.0(15)S, Targeted for 12.2(1)T, containing the TOS Requires the new Netflow Collector 3.5 or above Default Values: cache size: 4096 entries active/inactive timeouts: 30 min/15 sec

54 Version 8 Header Format Generic Time Check Netflow Version Number of flow sysuptime UTC time in sec Residential nanoseconds Flow Sequence Export from Engine Type Engine Id 0 for RP, 1 for VIP/LC VIP/LC slot number What Aggregation Added from version 5

55 Version 8 Export NetFlow Main Cache Flow Entries Flow 1 Flow 2 Flow 3 Flow expired Cache full Timer expired Aggreg. Cache Export V5 Record Export v5 UDP To Collector Not Necessary Flow expired Cache full Timer expired AS-Matrix Prefix-Matrix... Cache full Timers expired Export V8 Record UDP To Collector

56 Version 8 Export Don t export version 5 and version 8 at the same time. No sense! If you export multiple aggregations, maybe worth to export only v5 and aggregate locally on the Netflow Collector. This will reduce the export bandwidth requirement

57 Version 8 Export The default timeouts for version 5 active/inactive timeouts: 30 min/15 sec The default timeouts for version 8 active/inactive timeouts: 30 min/15 sec Must wait AT LEAST 30 seconds for a version 8 export packet (for UDP) The aggregation cache will contains less entries than the main cache. But these entries will be updated more often.

58 Version 8 Configuration router (config)# ip flow-aggregation cache as router (config-flow-cache)# export destination router (config-flow-cache)# enabled router (config)# ip flow-aggregation cache protocolport router (config-flow-cache)# export destination router (config-flow-cache)# cache entries 8192 router (config-flow-cache)# enabled Note the 2 different export ip addresses/ports

59 Version 8 Show Command router#sh ip cache flow aggregation as IP Flow Switching Cache, bytes 2 active, 4094 inactive, 13 added 216 ager polls, 0 flow alloc failures SrcIf SrcAS DstIf DstAS Flows Pkts B/Pk Active Se0/0 0 Se0/ Se0/0 0 Null Note: you must choose peer-as or origin-as router (config)# ip flow-export version 5 <peer-as origin-as> So that the main cache populates the BGP AS So that the aggregation cache will contain the populated BGP AS

60 Version 9 Simply 3 rd party application development Reduce time cycle for new features Future Proofing New approach from v1-v8 Available in IOS version *

61 Version 9 New Concepts Template FlowSet Data FlowSet Options Template

62 Version 9 Packet Format Packet Header Template FlowSet Data FlowSet Data FlowSet Template FlowSet Data FlowSet Packet Header Format same as Version 5 Template FlowSet & Data Flowsets are intermingled in the same packet Template FlowSet defines proceeding Data FlowSets Data FlowSets contain the actual export data.

63 Version 9 Template FlowSet FlowSet ID = 0 Length Template ID Field Count Field 1 Type Field 1 Length Field 2 Type Field 2 Length... Field N Type Field N Length FlowSet ID - Length - Template ID - Field Count - Field Type - Field Length - The FlowSetID is used to distinguish a template record from a data record. Always zero. Refers to the total length of the FlowSet. It is the combined length of all the FlowSets in the record. Router generates a unique FlowSet ID s to reflect the Netflow data it will be exporting. ID s start at 256, are reserved. Defines the number of fields in a template record. Type of field is vendor specific. Cisco supplied values are consistent over all platforms. Length of all the above in bytes.

64 Version 9 Data FlowSet FlowSet ID = TemplateID Length Record 1 Field 1 Value Record 1 Field 2 Value Record 1 Field 3 Value. Record 1 Field N Value Record 2 Field 1 Value Record 2 Field 2 Value. Record 2 Field N Value FlowSet ID = Template ID Length Record N Field N The FlowSet ID maps to a (previously received) template ID. The collector application uses this to make sense of the data it receives. Defines the length of Data FlowSet. The type and length of the fields have been previously defined. This are actual data collected in the router.

65 Version 9 Options Template FlowSet ID = 0 Length Reserved Template ID = 1 Option Scope Length Option Length Scope 1 Field Scope 1 Field Length... Option 1 Field Type Option 1 Field Length FlowSet ID = 0 - Length - Reserved Template ID - Option Scope Length - Options Length - Scope 1 Field Type - Scope 1 Field Length - Option 1 Field Type - Option 1 Field Length - The FlowSet ID is used to distinguish template records from data records Defines the total length of the FlowSet ID s are reserved by Netflow Version 9. An ID of 1 defines an Options Template. This value in bytes, defines the length of any of the scope fields that follow. Defines the length of the options field. 0x0001 System 0x0002 Interface 0x0003 Line Card 0x0004 Netflow Cache 0x Template Length of the Scope field. Defines the type of field in the options record Length of the field

66 Netflow Formats - Summary Version 1 Initial Version Not commonly used Version 8 Router based aggregation Available in 12.0(3)T, 12.0(3)S Version 5 Superset of Version 1 Added AS accounting Datagram Sequencing Commonly used Version 9 Configurable Flow Record Templates Version 7 Cat5K NFFC Only Not available in IOS Versions 2,3,4 and 6 were experimental

67 Version 1,5,8 Platform Support Cisco IOS Release Version Supported netflow export version(s) Supported Cisco Hardware Platforms 11.1 CA, 11.1CC v1, v , RSP , 11.2P v1 7200, 7500, RSP P v1 Route Switch Module (RSM), 11.2(10)P and later 11.3, 11.3T v1 7200, 7500, RSP v1, v5 1720, 2600, 3600, 4500, 4700, AS5800, 7200, ubr7200, 7500, RSP7000, RSM 12.0T, 12.0S v1, v5 1720, 2600, 3600, 4500, 4700, AS5800, 7200, ubr7200, 7500, RSP7000, RSM, MGX 8800 RPM, BPx 8600 From 12.0(3)T From 12.0(3)S * : from 12.0(4)T v1, v5, v8 1400*, 1600*, 1720, 2500*, 2600, 3600, 4500, 4700, AS5800, AS5300, 7200, ubr7200, 7500, RSP7000, RSM, MGX8800 RPM, BPX (4)XE v1,v5,v (6)S and later v

68 Outline Introduction NetFlow Netflow Overview Netflow Architecture Netflow Formats Netflow Feature Acceleration Netflow Deployment AAA Our solutions System Architecture Enhance Flow-Tools Protocol Using Dynamic Ports Conclusion and Future Work

69 Netflow - Not a Switching Path In the past (before CEF), Netflow was a switching mechanism. But we faced complications and performance problems When CEF was written, the Netflow code was rewritten to do only the accounting job. No switching anymore. Netflow runs now on the top of CEF to store accounting statistics. We still look into the FIB for adjacencies, encapsulation info, route, As a consequence the Netflow switching name was changed to Netflow services

70 Netflow Acceleration An API used by the other IOS features Needs 12.0(3)T Reserve extra space in the Netflow cache for state information from other features. Apply the feature processing on the first packet versus every packets. Information from the first packet is used to be build the cache entry, accessed by subsequent packets from the same flow Access Control Lists is accelerated by default, nothing to configure

71 Netflow Acceleration Depending on the train 12.0S, 12.0ST, 12.1 or 12.2, Netflow accelerates Ip accounting RSVP Crypto encrypt and decrypt Policy Routing WCCP inbound redirection Cisco Applications and Services Architecture Future: CAR, NAT, etc...

72 NetFlow Feature Acceleration NetFlow Accelerates NetFlow Policy Routing (NPR) Router-based network data encryption Access Control Lists (ACL) RSVP In the future Network Address Translation (NAT) Committed Access Rate (CAR) Web Cache Control Protocol (WCCP) Others Availability of such acceleration will be announced on a feature-by-feature basis

73 Netflow Bypasses the Access-list Y First packet in flow? N ACL acceleration Y Create an Netflow entry Forward the packet with CEF Pass the ACL? N Create an Netflow entry with output i/f null Discard the packet Lookup entry in netflow cache Y Update the Netflow entry stats Output i/f is null? Go through the ACL Maybe deny packet N Update the Netflow entry stats Forward the packet with CEF

74 Netflow Acceleration router (config)# ip flow-cache feature-accelerate % Flow Feature Acceleration will be enabled after either the next reboot or flow switching is turned off on all interfaces. Show commands sh ip flow acceleration sh ip cache verbose flow NetFlow cache size is increased bytes by entries instead of 64 Can accelerate up to 7 features per flow

75 Acceleration - Netflow Policy Routing ip cef ip flow-cache feature-accelerate interface ethernet0/0/1 ip route-cache flow ip policy route-map test route-map test permit 10 match ip address 1 set ip precedence priority set ip next-hop set ip next-hop verify-availability route-map test permit 20 match ip address

76 Acceleration - Netflow Policy Routing The first packet will go through the route-map and the access-list A Netflow cache entry will be created with extra information for policy routing (for example the next hop) Subsequent packets of the same flow will bypass the route-map access-list checks Note that the acceleration doesn t change the switching path!

77 Performance (Approximate Number) Enabling Netflow version 5 on a router increases the cpu utilization by 20 to 25 % The Neflow export increases the cpu utilization by 5 % Enabling Neflow version 8 increases the cpu utilization by 2 to 5%, depending on the number of aggregations enabled With a multiple of 6% for multiple aggregations Netflow is done in hardware on the cat6000 supervisor

78 Outline Introduction NetFlow Netflow Overview Netflow Architecture Netflow Feature Acceleration Netflow Formats Netflow Deployment AAA Our solutions System Architecture Enhance Flow-Tools Protocol Using Dynamic Ports Conclusion and Future Work

79 Where to Collect the Traffic: Edge vs. Core Edge Core Communication pattern Flow duplication CPU impact Data compression Data reduction (filter) Data aggregation

80 Where to Deploy Netflow? On the edges of the network All routers because Netflow accounts incoming traffic only For billing, on the aggregation routers because some Line Cards only support sampled Netflow For accounting, capacity planning, on the aggregation routers or the router. Sampled netflow could be sufficient

81 Where to Deploy Netflow? For BGP information, on the BGP peering routers Can monitor one link, egress and ingress, but should be on a MPLS PE-CE link. Basic principles: Don t account your exported data Avoid a flow duplication design. Netflow Collector doesn t do flow de-duplication. Done by partner tools export export traffic

82 Netflow and Security There is no authentication mechanism between the routers and the collector The collector is only interpreting received UDP packets, without any checks Make sure your Data Communication Network is secure, including the collector machine Potential problem: someone sending wrong accounting information to the collector with a router stolen IP address

83 How Many Netflow Collector? In theory, one NFC per POP or Aggregation Router (7x00 router) For VPNSC (MPLS VPN environment), we advice one NFC per PE Basic principles: Check your Sun capabilities NFC sizer calculater. Reduce the number of routers per NFC if needed. Rule of thumb: 10 routers per NFC

84 Deployment Tricks Enable the ifindex persistence if accounting per interface Look at the router cpu (<60%) and memory before enabling Netflow Check the export link bandwidth Use a dedicated export lan If you export too much traffic: go for the aggregations, don t export version 5 go for sampled if on a GSR increase the aggregations timers Access-lists still account the traffic

85 What to Collect: Level of Collection Details Link statistics or traffic details: SA, DA Application details (port numbers) QoS Time stamps Routing and peering Header or payload Layer 2 or Layer 3 information Data export: push or pull model Collection interval and history Consider the generated data volume

86 What to Collect: The Two Extremes... S N M P N e t F l o w Usage Time of Day Port Utilization QoS Packet count Byte count Start sysuptime End sysuptime Input ifindex Output ifindex Type of service TCP flags Protocol Source IP address Destination IP address Source TCP/UDP port Destination TCP/UDP port Next hop address Source AS number Dest. AS number Source prefix mask Dest. prefix mask From/To Application Routing and Peering

87 What to Collect: Full Collection vs. Sampling Processing every packet might not scale up to very high-speed interfaces Amount of collected data might be huge It might take longer to process the data than to generate it Network Management traffic might fully utilize the available bandwidth Packet sampling can help to overcome those issues

88 What to Collect: 1 in n Sampling Sampling Interval: 1 in 2 Packets Missed Flows: 1 out of 5 (15 %) Sampling Interval: 1 in 5 Packets Missed Flows: 2 out of 5 (35%)

89 What to Collect: Sampling Best Practices Sampling for monitoring is fine Continuously sampling might be OK even for billing purposes Carefully determine the sampling rate Sampling algorithms: 1 in n (deterministic, random, hash-based) Filter, expressions Time based Trajectory sampling Sampling White Paper: work in progress

90 IP Accounting/Billing Many Different Flavors! Flat-rate billing doesn t always scale Competitive pricing models can be created with usage-based billing Usage-based billing considerations Time of day Within my network or off Application Distance-based QoS/CoS Bandwidth usage Transit or peer Data transferred Traffic class (i.e. going through a secure tunnel, high-speed link, or special arrangement)

91 User Definition Users (IP Address, Name, etc.) User 1 User 2 User 3 User 4 User 5 User 6 User 7 Departments Dept. 1 Dept. 2 Dept. 3 Dept. 4 Dept. 5 Customers Co. 1 Co. 2 Co. 3 Co. 4 Co. 5 Co. 6 Co. 7 Reporting can be offered at any level Customers can self-manage all sub-levels Orange and blue can be sold at a premium

92 Which Aggregations to use on a Router? AS Protocol-Port Source-Prefix Destination-Prefix Prefix Source Prefix Source Prefix Mask Destination Prefix Destination Prefix Mask Source App Port Destination App Port Input Interface Output Interface IP Protocol Source AS Destination AS First Timestamp Last Timestamp # of Flows # of Packets # of Bytes

93 Which Aggregation to use on a Router? AS- TOS Protocol-Port- TOS Source-Prefix- TOS Destination-Prefix- TOS Prefix-TOS Prefix-Port Source Prefix Source Prefix Mask Destination Prefix Destination Prefix Mask Source App Port Destination App Port Input Interface Output Interface IP Protocol Source AS Destination AS TOS First Timestamp Last Timestamp # of Flows # of Packets # of Bytes

94 Network Data Analyzer NetFlow FlowCollectors NetFlow FlowAnalyzer Graphical display of NetFlow data Consumes from NetFlow FlowCollector(s) Time-based analysis ands data sorting Configure routers and FlowCollectors Histograms, bar charts, and pie charts Spreadsheet data export

95 Open API s Enable Third Parties to Leverage NetFlow Cflowd - ANS, BBN and CAIDA Traffic accounting port, AS, network and pure flow matrices NeTraMet/NetFlowMet - by Nevil Brownlee IETF s Realtime Traffic Flow Measurement (RTFM) smurfind - Walter Prue USC/ISI Real time DOS attack warnings

96 End-to-end Coverage Health Reports Service Level Reports Report for Thu 1/15/98 Trend Reports Auto Range: Custom From: 09/04/ :00 AM 01/15/ /13/1997 Baseline: 6 weeks (02/04/98 to 03/17/98) Created : 05/15/98 12:00:16 09/13/1997 Exceptions Reports Router & LAN Stats. WAN Stats. Access Stats. NetFlow Collector RMON Probes SAA Agent Ping MIB Element & L2/L3/Access Stats. Traffic Flow Stats. Response Time/ Availability Stats.

97 Concord and NetFlow Report for Thu 1/15/98 Report for Thu 1/15/98 Report for Thu 1/15/98 Concord Workstation NetFlow Collector Benefits Within Cisco IOS, Lower cost of entry than RMON/RMON2 probes Leverages large installed base of Cisco routers and switches NetFlow enabled Reports Router Link, LAN, router utilization Application mix Communicating pairs NetFlow enabled L3 Switch

98 Cisco NetFlow support Gather high volume NetFlow data Router Router InfoVista NetFlow Agents InfoVista Web Access Server Combine it with other InfoVista data Router Router Données InfoVista Server InfoVista Client Router InfoVista NetFlow Agents InfoVista Client Analyze traffic flows by source and destination autonomous system, average packet size and used protocols

99 Cisco NetFlow support End-User Benefits: A Service Provider can optimize its existing connections with other autonomous systems, plan new connections, and proactively identify problem areas. An Enterprise can use this information to identify network use patterns and to plan the evolution of its network infrastructure. Destination Autonomous System Source Autonomous Systems Packet distribution by source AS Automatic resolution of Autonomous System name

100 Outline Introduction NetFlow Netflow Overview Netflow Architecture Netflow Formats Netflow Feature Acceleration Netflow Deployment AAA Our solutions System Architecture Enhance Flow-Tools Protocol Using Dynamic Ports Conclusion and Future Work

101 Description RADIUS and TACACS+ accounting allows data to be sent at the start and end of services, indicating the amount of resources such as time, packets, bytes, etc. used during the session AAA is used for login purposes in general Dial-in Telnet and ssh PPP

102 RADIUS and TACACS+ Comparison Remote Authentication Dial In User Service Standards-based clientserver protocol (IETF) UDP-based (fast) Recommended for high performance Only password field encrypted Shared key, never sent in clear over the network User authentication to network access/services Terminal Access Control Access Control System Rich feature set: allows command authorization and accounting Cisco proprietary (but supported by other vendors) TCP-based (reliable) Full packets are encrypted Shared key, never sent in clear over the network User authentication to network devices

103 AAA: Principles Incoming and outgoing packets/bytes of an incoming call (no dial out accounting) Each of the call can generate start and stop records Each call reports 2 logs: Accounting request start with start time Accounting request stop with stop time and full accounting AA Accounting is an improved logging system, but AAA is not used primarily for accounting Adequate for billing because we have the username Supported on all switching paths

104 RADIUS Interaction RADIUS Server User Dials NAS Accept Call Pre-Auth Pre-Auth Access Request Pre-Auth Access Accept Call Connects Accept User User Auth Access Request Access Accept User Connects Call Disconnects User Acctg User Acctg Accounting Request (START) Accounting Ack Accounting Request (STOP) Accounting Ack

105 RADIUS Accounting Attributes, RFC Acct-status-type 41 Acct-delay-time 42 Acct-input-octets 43 Acct-output-octets 44 Acct-session-id 45 Acct-authentic 46 Acct-session-time 47 Acct-input-packets 48 Acct-output-packets 49 Acct-terminate-cause 50 Acct-multi-session-id 51 Acct-link-count

106 Configuration aaa group server radius MyAdmin server auth-port 1645 acct-port 1646 server auth-port 1645 acct-port 1646! aaa accounting exec OpsAcctg start-stop group MyAdmin aaa accounting network DialAcctg start-stop group MyAdmin! interface Group-Async0 ppp accounting DialAcctg! line vty 0 4 accounting exec OpsAcctg

107 Radius Extension for VoIP RADIUS enhanced for voice applications to allow calls to create a call detail record (CDR) The Acct-session-id field is utilized: <session id>/<call leg setup time>/<gateway id>/<connection id>/<call origin>/<call type>/<connect time>/<disconnect time>/<disconnect cause>/<remote ip address> Acct-Session-Id = "25/ UTC Thu Feb /ch1- gw.mwest/fc82262c 9CD C285F4/originate/VoIP/ UTC Thu Feb / UTC Thu Feb /10 / "

108 AAA Possible Applications Network Monitoring AAA Network Planning Security Analysis X Application Monitoring User Monitoring X Traffic Engineering Peering Agreement Usage-Based Billing X Destination Sensitive Billing

109 Outline Introduction NetFlow Netflow Overview Netflow Architecture Netflow Formats Netflow Feature Acceleration Netflow Deployment AAA Our solutions System Architecture Enhance Flow-Tools Protocol Using Dynamic Ports Conclusion and Future Work

110 路 流 量 量 Network Device Flow Generator Flow Capturer Flow Analyzer Scalability Data Store Presenter Web Site User Interface Web browser raw packet Flow information Network Characteristics analyzed data System design for Flow Capture Flow Analyzer Distributed, load-balancing architecture for scalability Traffic Analysis & Data Reduction Presentation & Reporting

111 Data Source ISP 6509 ISP 6509 TANet 6509 ISP 6509 ISP Collector Analyzer

112 System Architecture Cache creation Data export Data processing Aggregation Data presentation Flow-Analyzer Flow-Collector Collection Filtering Aggregation Storage

113 Flow Tools NetFlow 理 Open Source Flow-Tools 理 理 NetFlow 料

114 Components in Flow-tools Flow-capture Raw File Flow-cat Flow-filter Flow-stat Reports

115 Components in Flow-tools Flow-tag Group flows by common prefixes, AS Flow-merge Merge flow files in order chronological Flow-split Split flow files into smaller files based on size, time, or tags Flow-report Combine functions of flow-filter/flow-tag/flow-stat

116 Main Data Structure used in Flow Tools Hash + Link-List XOR-Folding Hash 16 Bit Hash Value 2^16 Buckets Extensively Used on calculating Top N s IP IP Pair AS Pair Bucket Bytes Next(4 byte) Prefix(4 byte) Counters

117 Enhanced Flow-Tools -- Summary Use Adaptive Hash Buckets 18 Use Different Hash Functions 不 Hash 1.3% Replace 2 nd level hash by Judy Array 度 98 RAM 1/28

118 Adaptive Hash Bucket Size Data source: TANet 流 量

119 Enhanced Flow-Tools - Summary Use Adaptive Hash Buckets 18 Use Different Hash Functions 不 Hash 1.3% Replace 2 nd level hash by Judy Array 度 98 RAM 1/28

120 Use Different Hash Functions -> ISP, Backbone, 兩 流 量 Key=Hash(Source_IP), 19bit hash value Flows: 7,170,567 Time to Calculate Hash Value < 2sec IP : 44,7830, Bucket : 52, % free buckets if hash is uniform

121 Use Different Hash Functions XOR Folding Lower 19bits FNV-XOR 32 bit FNV hash, xor-folding to 19bits FNV-Lazy mod mapping 32 bit FNV hash, mod 2^19 MyHash1

122 料, ISP, 兩 流 量 Report Format: (Source IP, flows, bytes, packets) Flows: 7,170,567 Time to Calculate Hash Value < 2sec IP: 447,830, Buckets: 524,288(2^19) Free Buckets & Average Chain Length 理 狀 : 14.59% 1.00 XOR Folding : 41.30% 1.45

123 Average Chain Length

124 Free Buckets 理

125 Enhanced Flow-Tools -- Summary Use Adaptive Hash Buckets 18 Use Different Hash Functions 不 Hash 1.3% Replace 2 nd level hash by Judy Array 度 98 RAM 1/28

126 Performance of Original Flow Tools,ISP, 兩 流 量 Flows: 1,214,748 IP: 301,342 Report Format ip-source-address-destination-count M RAM

127 Two Level Hash (Original Flow Tools) Bucket Bytes Next(4 byte) Prefix(4 byte) Counters Bucket Bytes Next(4 byte) Prefix(4 byte) Mask(1 byte) Pad(3 byte)

128 Inefficient 2 nd level hash 連 IP 數 數 數 FTP IP 度 link-list 64 entry 若 Source IP 數 量 Destination IP 數 量 浪

129 Inefficient 2 nd level hash Bucket Bytes Next(4 byte) Prefix(4 byte) Counters Bucket #IP: st Hash: 65536* *108=31MB 2 nd Hash: *(256*8+64*12)=805MB Bytes Next(4 byte) Prefix(4 byte) Mask(1 byte) Pad(3 byte)

130 Replace 2 nd level hash by Judy Array Bucket Bytes Next(4 byte) Prefix(4 byte) #IP: st Hash: 65536* *108=31MB 2 nd Hash: *12=3.4MB Judy Array(min 12 bytes) Index(0..2^32-1) Map To IP 4 Bytes Mask(1 byte) Pad(3 byte)

131 Performance Speedup <-ISP, 理 RAM Original MB Judy Array 8 35MB

132 路

133 Motivation Well-known: telnet/ssh/smtp/pop3/news

134 Protocol FTP Active Mode Passive Mode edonkey BitTorrent

135 FTP Active

136 FTP Passive

137 edonkey Protocol Search File 2 Index Server 1 Register IP & Port Upload File List Peer A 3 Bi-direction File Transfer Peer B

138 edonkey Protocol Register & Upload File List Default TCP 4661 File Search Query Default UDP 4665 File Transfer & Control Message Default TCP 4662 emule Extention to Control Message Default UDP 4672, 4673

139 BitTorrent Protocol Peer A Tracker IP File Hash Tracker 2 Ask Peer IP List of Known File Hash 3 2 Download/Upload Status Peer B Bi-direction File Transfer 1 Web Server Tracker IP File Hash Download Meta info file(some Video.torrent)

140 BitTorrent Protocol Tracker Default TCP 6969 HTTP GET/POST Peer Default TCP BitTorrent Peer Protocol

141 流 NetFlow Raw Data Top N IP? Y? Y N? 離 Outbound 流 量 量 N Protocol Y 列 byte Source IP 立 連 Protocol 列 IP:Port Protocol

142 流 NetFlow Raw Data 離 Outbound 流 量 Byte 數 Source IP Heuristic Protocol:Port 立 連 Protocol 列 IP:Port Protocol 量

143 Well-known: telnet, ssh, smtp, pop3, news

144 Ongoing Work Support for various applications Streaming services Other P2P services Distributed, load-balancing architecture for scalability parallel or distributed architecture subdivide monitoring system into several functional components efficient load sharing between each sites Considerations for small storage requirements Significant aggregation based on the ingress point Local reduction of the data should be effective

145 145