Netflow for Accounting, Analysis and Attack

Transcription

1 Netflow for Accounting, Analysis and Attack Andy Chien Consulting System Engineer 1

2 Agenda Introduction Platforms Versions Accounting and Analysis MPLS Environment Accounting and Analysis BGP and Autonomous Systems Accounting and Analysis Multicast Options Attack Security Features and Applications Scaling Features and Options Eport Collector, NAM and Partners Evolving NetFlow Deployment and Direction Q&A 2

3 Introduction RST _05_2004_c Cisco Systems, Inc. All rights reserved. 3

4 The Value of NetFlow Services REQUIREMENT TECHNOLOGY SOLUTION Who are the top talkers? How much Traffic per user/group? Traffic per application? Traffic on-net / off-net How many users are active on the network at any given time? Where Does the traffic come from? Does it go to? When was it transmitted? Security attacks? Source Address Destination Address Source Port Destination Port Layer 3 Protocol Type DSCP Input Logical Interface BGP Net Hop TOS MPLS Label MPLS Label Type (LDP, BGP, VPN, ATOM, TE Tunnel MID-PT) Network Monitoring Network Planning Security Analysis Application Monitoring User Monitoring Traffic Engineering Peering Agreement Usage-Based Billing Destination-Sensitive Billing 4

5 What Is a Flow? Defined by Seven Unique Keys: Source IP address Destination IP address Source port Destination port Layer 3 protocol type TOS byte (DSCP) Input logical interface (ifinde) A flow is unidirectional! Eported NetFlow Data 5

6 NetFlow Principles Inbound traffic only (with some eceptions) Unidirectional flow Accounts for both transit traffic and traffic destined for the router Works with Cisco Epress Forwarding (CEF) or fast switching Not a switching path Supported on all interfaces and Cisco IOS software platforms Provides the subinterface information in the flow records 6500/7600 enables NetFlow on all interfaces by default 6

7 How does the NetFlow Cache work? NetFlow Cache 7 identifiers Other data Flow identifiers Flow data update Flow identifiers Flow data Flow identifiers Flow data Eported Data 7

8 NetFlow Architecture IPv4/v6 Traffic SNMP NetFlow MIB NetFlow Enabled Devices Network (IP, MPLS) Source address Destination address Source port Destination port Layer 3 protocol type DSCP Input logical interface BGP net hop TOS MPLS label MPLS label type NetFlow Eport (v9) Packets: 1. Templates 2. Data Records NetFlow Collector (various) Applications: Performance Security Billing 8

9 NetFlow Processing Order Features and Services Pre- Processing Post- Processing Packet Sampling Filtering IPv4 Multicast MPLS IPv6 Aggregation schemes Non-key fields lookup Eport 9

10 NetFlow Eport Format Version 5 Usage Packet count Byte count Source IP IP Address address Destination IP IP Address address From/To Time of Day Start sysuptime End sysuptime Source TCP/UDP port Destination TCP/ UDP port Application Port Utilization QoS Input ifinde Output ifinde Type of service TCP flags Protocol Net Hop address Source AS number Dest. AS number Source prefi mask Dest. prefi mask Routing and Peering Blue key field Black value field Red lookup field 10

11 NetFlow Cache Eample 1. Create and update flows in NetFlow cache Srclf SrclPadd Dstlf DstlPadd Protocol TOS Flgs Pkts Src Port Src Msk Src AS Dst Port Dst Msk Dst AS NetHop Bytes/ Pkt Active Idle Fa1/ Fa0/ A2 / A2 / Fa1/ Fa0/ / / Fa1/ Fa0/ A1 / A1 / Fa1/ Fa0/ / / Epiration Inactive timer epired (15 sec is default) Active timer epired (30 min (1800 sec) is default) NetFlow cache is full (oldest flows are epired) RST or FIN TCP Flag Srclf SrclPadd Dstlf DstlPadd Protocol TOS Flgs Pkts Src Port Src Msk Src AS Dst Port Dst Msk Dst AS NetHop Bytes/ Pkt Active Idle Fa1/ Fa0/ A2 / A2 / Aggregation No Yes 4. Eport version Non-Aggregated Flows Eport Version 5 or 9 5. Transport protocol Eport Packet Header Payload (Flows) e.g. Protocol-Port Aggregation Scheme Protocol A2 11 Pkts SrcPort DstPort 00A2 Bytes/Pkt 1528 Aggregated Flows Eport Version 8 or 9 11

12 NetFlow Configuration Commands Per interface e.g. ip flow-eport version 5 e.g. ip flow-eport destination Default is interface with best route to collector. Recommendation: configure loopback interface. Selects the aggregation cache Sets the seconds an inactive flow will remain in the cache before epiration. 15 seconds is default Sets the minutes an active flow will remain in the cache before epiration. 30 minutes is default Sets the maimum number of flow entries in the cache. The default varies dependent on platform. 12

13 NetFlow Show Commands Shows NetFlow statistics Shows NetFlow statistics for the configured aggregation scheme Shows eport statistics Clears NetFlow statistics Clears eport statistics 13

14 show ip cache flow router_a#sh ip cache flow IP packet size distribution (85435 total packets): IP Flow Switching Cache, bytes 2728 active, 1368 inactive, added ager polls, 0 flow alloc failures Active flows timeout in 30 minutes Rates and Duration Inactive flows timeout in 15 seconds last clearing of statistics never Protocol Total Flows Packets Bytes Packets Active(Sec) Idle(Sec) Flows /Sec /Flow /Pkt /Sec /Flow /Flow TCP-X TCP-other Total: Flow Details Packet Sizes # of Active Flows SrcIf SrcIPaddress DstIf DstIPaddress Pr SrcP DstP Pkts Et0/ Se0/ AEE Et0/ Se0/ D Et0/ Se0/ CB

15 show ip cache verbose flow router_a#sh ip cache verbose flow IP packet size distribution (23597 total packets): IP Flow Switching Cache, bytes 1323 active, 2773 inactive, added ager polls, 0 flow alloc failures Active flows timeout in 30 minutes Inactive flows timeout in 15 seconds last clearing of statistics never Protocol Total Flows Packets Bytes Packets Active(Sec) Idle(Sec) Flows /Sec /Flow /Pkt /Sec /Flow /Flow TCP-other Total: Source Mask and AS Destination Information Flow Rate and Duration ToS Byte and TCP Flags SrcIf SrcIPaddress DstIf DstIPaddress Pr TOS Flgs Pkts Port Msk AS Port Msk AS NetHop B/Pk Active Et0/ Se0/ FA7 / / Et0/ Se0/

16 Platforms RST _05_2004_c Cisco Systems, Inc. All rights reserved. 16

17 Comprehensive Platform Support ASIC ASIC / 4700 AS5300/ / 7300/ 7400/ 7500/ Si C 4500 ASIC C 6500/ 7600 ASIC

18 Cisco Catalyst 6500 and 7600 Series Switches Hybrid: Cisco Catalyst OS on PFC/supervisor and Cisco IOS software on MSFC Native Cisco IOS software: PFC/supervisor and the MSFC both run a single bundled Cisco IOS software image Eport is centrally via the supervisor and MSFC, each linecard has its own hardware NetFlow cache and forwarding table, i.e. distributed platform Hybrid Native 12.1E Native 12.2SX MSFC v5 v5 v5, v8* Sup1a V7, v8 v7 N/A Sup2 V7, v8 v5, v7 v5, v7, v8 Sup720 v5, v7, v8 v5, v7 v5, v7, v8 *No NetFlow Support on MSFC with Sup1a 18

19 Catalyst 6500/7600 Supervisor Supervisor 1 1 st st Packet 2 nd Packet Supervisor 1: MSFC When destination has no adjacency in FIB the 1st packet goes to MSFC for ARP request; This packet is not counted by the supervisor2 If NetFlow is enabled on the MSFC2, the MSFC2 accounted packets will have DstIf = Null (by limitation) Supervisor 2 99% of traffic goes through the supervisor 2 3 rd Packet 4 th Packet 19

20 MLS Best Design NFC Vlan1 Eport Supervisor 2 Eport MSFC2 Vlan14 MLS-Enabled and Eport v7 from the SUP2 Eport v5 from the MSFC2 And Eport in the sc0 vlan 20

21 Catalyst 6500/7600 Versions and Features Cisco IOS software release 12.1(13)E1 PFC2 Source/destination interface information (Hybrid 6.3(6)) PFC2 Source/destination AS information PFC2 Support for V5 NetFlow data eport (Hybrid 7.5(1)) IP Net hop Sampled NetFlow is available on PFC in Cisco IOS Cisco IOS software release 12.2(14)SX Version 8 in native mode PFC3b (Sup720) cards ToS byte Multicast traffic Hybrid Catalyst OS 7.2(1) L2 switched traffic (vlan to vlan y) support (doesn t require MSFC) Hybrid Catalyst OS 7.3(1) Destination and source IfInde enabled by default 21

22 Catalyst 6500/7600: Native Cisco IOS Mode mls flow ip full -> flow mask mls nde src_address version 7 -> version 7 eport source OR mls nde sender -> NDE enable + NDE from the PFC uses the source configured from the MSFC!!!!! interface vlan 1 ip address ip route-cache flow interface FastEthernet 3/2 ip address ip route-cache flow ip flow-eport source vlan1 -> version 5 eport source ip flow-eport version 5 ip flow-eport destination > both for version 5 and 7 eport 22

23 Catalyst 6500/7600 Switched Traffic New L2 switched traffic (vlan to vlan y) support in Hybrid Catalyst OS 7.2(1); It doesn t require MSFC; Note: Not yet available in native mode set mls bridged-flow-statistics enable/disable <vlan> Destination and source IfInde enabled by default, support in Hybrid 7.3(1) set mls nde {destination-inde source-inde} {enable disable} 23

24 Catalyst 4000/4500 NetFlow NetFlow services card in Supervisor 4: 12.1(13)EW supports version 5 without interface tracking 12.1(19)EW supports version 5 (with interface tracking) and version 8 NetFlow services card in Supervisor 5: 12.2(18)EW supports Version 5 and 8 Prior card was NetFlow Feature Card (NFFC) (now end of sale) 24

25 Cisco Series Internet Routers: NetFlow support Engine 0 software support Engine 1 software support Engine 2 support in ASICs, however there is a significant performance impact if running many other features concurrently Engine 3 support in ASICs Engine 4 not supported Engine 4+ support in ASICs 25

26 Versions RST _05_2004_c Cisco Systems, Inc. All rights reserved. 26

27 NetFlow Versions NetFlow Version Comments 1 Original 5 Standard and Most Common Specific to Cisco C6500 and 7600 Series Switches Similar to Version 5, but Does Not Include AS, Interface, TCP Flag and ToS Information Choice of 11 Aggregation Schemes Reduces Eport Resource Usage Fleible, Etensible File Eport Format to Enable Easier Support of Additional Fields and Technologies e.g. MPLS, Multicast, BGP Net Hop, and IPv6 27

28 Version 8: Aggregation Flow Format AS- TOS Protocol- Port-TOS Source- Prefi-TOS Destination- Prefi-TOS Prefi- TOS Prefi- Port Source Prefi Source Prefi Mask Destination Prefi Destination Prefi Mask Source App Port Destination App Port Input Interface Output Interface IP Protocol Source AS Destination AS TOS First Timestamp Last Timestamp # of Flows # of Packets # of Bytes 28

29 Version 8: Configuration!"# $ %& %& '(& )* )*'(& * * * '(& * *+'(& &* &*'(& Note Do Not Eport Version 5 at the Same Time: ip flow-eport version 5 29

30 Version 9 Why Just Another Version? Previous formats (versions 1, 5, 7, and 8) were fied format and infleible 1) Cisco needed to build a new version each time a customer wanted to eport new fields 2) Partners had to reengineer to support the new eport format Solution: Build a Fleible and Etensible Eport Format! Whitepaper 30

31 NetFlow v9 Principles Version 9 is an eport format Still a push model Send the template regularly (configurable) Independent of the transport protocol, currently UDP, but ready for reliable transport protocols e.g. TCP, SCTP, Advantage: we can add new technologies and data types quickly e.g. MPLS, IPv6, BGP Net Hop, Multicast, 31

32 NetFlow v9: Eample for Template Definition Template A Flow Set ID (0 for Template) Length of Template Structure 1001 (Template ID) 3 (# of Fields) SRC_AS_NUMBER 2 DST_AS_NUMBER 2 L4_PROTOCOL 2 Template B Flow Set ID (0 for Template) Length of Template Structure 1002 (Template ID) 4 (# of Fields) SRC_IP_PREFIX 4 SRC_AS_NUMBER PACKET_COUNT 2 BYTE_COUNT

33 NetFlow v9: Eample for 1 Eport Packet As Defined in the Previous Slide Packet Header Template B ID: (# of Records in FlowSet) Template A ID: Record 1 Record 2 Record 1 Data for Template B Data for Template A 33

34 NetFlow v9 Eport Packet To Support Technologies Such As MPLS or Multicast, This Eport Format Can Be Leveraged to Easily Insert New Fields Flows from Interface A Flows from Interface B Header (Version, # Packets, Sequence #, Source ID) Template FlowSet Template Record Template ID #1 (Specific Field Types and Lengths) Template Record Template ID #2 (Specific Field Types and Lengths) Data FlowSet FlowSet ID #1 Data Record (Field Values) Data Record (Field Values) Data FlowSet FlowSet ID #2 Data Record (Field Values) Option Template FlowSet Template ID (Specific Field Types and Lengths) Option Data FlowSet FlowSet ID Option Option Data Data Record Record (Field Values) (Field Values) The header follows the same format as prior NetFlow versions so Collectors will be backward compatible Templates are associates to data records by matching ID numbers Each data record represents one flow If eported flows have different fields, they require a separate template record (e.g. BGP net-hop cannot be combined with MPLS Aware NetFlow records) 34

35 NetFlow v9 Eport Format Eample of Eport Packet Right after NetFlow Configuration: Header (Version, # Packets, Sequence #, Source ID) Template FlowSet Template Record Template ID (Specific Field Types and Lengths) Template Record Template ID (Specific Field Types and Lengths) Template Record Template ID (Specific Field Types and Lengths) Template Record Template ID (Specific Field Types and Lengths) Option Template FlowSet Template ID (Specific Field Types and lengths) Option Data FlowSet FlowSet ID Option Data Record (Field Values) Option Data Record (Field Values) Eample of Eport Packets Containing Mostly Flow Information: Header (Version, # Packets, Sequence #, Source ID) Data FlowSet FlowSet ID Data Record (Field Values) Data Record (Field Values) Data Record (Field Values) Data Record (Field Values) Data Record (Field Values) Data Record (Field Values) Data FlowSet FlowSet ID Data Record (Field Values) 35

36 NetFlow Version 9 Configuration Configuring Version 9 Eport for the Main Cache!"# "# $, -.!"# "#./ Configuring Version 9 Eport for an Aggregation Scheme!"# "#! "#! "#$ Eport Versions Available for NetFlow Flows &0)1*! "#$ 232 Eport Versions Available for Aggregated.3. NetFlow Flows! "#. 36

37 IETF: IP Flow Information Eport (IPFIX) Working Group IPFIX is an effort to: Define the notion of a standard IP flow Devise data encoding for IP flows Consider the notion of IP flow information eport based upon packet sampling Identify and address any security privacy concerns affecting flow data Specify the transport mapping for carrying IP flow information (IETF approved congestion-aware transport protocol) NetFlow version 9 has been selected as a basis for the IPFIX protocol 37

38 IETF: Packet Sampling WG (PSAMP) PSAMP agreed to use IPFIX (NetFlow version 9) for eport PSAMP is an effort to: Specify a set of selection operations by which packets are sampled Describe protocols by which information on sampled packets is reported to applications Note: NetFlow is already using some sampling mechanisms 38

39 NetFlow Features supported with Version 9 Multicast NetFlow Availability: Major Release 12.3(1) and 12.2(18)S Ingress Accounting of replicated multicast packets Egress Per user accounting of multicast packets MPLS Aware NetFlow Availability: Release 12.0(26)S Label and prefi eport information BGP Net Hop Availability: Releases 12.0(26)S, 12.2(18)S, and 12.3 Edge to Edge Traffic Matri BGP traffic destination information NetFlow for IPv6 Availability: Release 12.3(7)T Eport IPv6 source and destination information 39

40 Accounting and Analysis MPLS Environment RST _05_2004_c Cisco Systems, Inc. All rights reserved. 40

41 NetFlow MPLS Features Overview Traditional NetFlow (IP to MPLS) MPLS-Aware NetFlow (MPLS to MPLS) Egress MPLS NetFlow (MPLS to IP) MPLS IP PE P PE IP Traffic Flow MPLS-aware NetFlow Cisco IOS software releases 12.0(24)S, 12.2(18)S, and 12.3(1) Egress MPLS NetFlow accounting Cisco IOS software releases 12.0(10)ST and 12.1(5)T 41

42 MPLS-Aware NetFlow (v9) IP MPLS Traffic Flow IP Enable on MPLS interfaces Tracks ingress traffic NetFlow version 9 only Option of IP and MPLS output or MPLS aggregation (top label aggregation) Supported in Cisco IOS software releases 12.3(1), 12.2(18)S and 12.0(26)S (24)S on the

43 MPLS Aware NetFlow Flow Keys Key fields (uniquely identify the flow) Source IP address Destination IP address IP protocol Input ifinde Source application port Destination application port DSCP Up to 3 incoming MPLS labels of interest with eperimental bits and end-of-stack bit Positions of the above labels in the packet label stack Additional eport fields Flows Packets Bytes Timestamps (sysuptime) IP Net Hop Output interface Accumulation of TCP Flags NetFlow version 5 fields of the underlying IP packet (TCP flags, etc ) Type of the top label: LDP, BGP, VPN, ATOM, TE Tunnel MID-PT, unknown The forwarding equivalent class (FEC) mapping to the top label 43

44 MPLS Aware NetFlow Flow Keys (AToM) Key fields (Uniquely Identify the flow) Input ifinde Up to 3 incoming MPLS labels of interest with eperimental bits and end-of-stack bit Positions of the above labels in the packet label stack Additional eport fields Packets Bytes Timestamps (sysuptime) Output interface Type of the top label: LDP, BGP, VPN, ATOM, TE Tunnel MID-PT, unknown Issue: the Forwarding Equivalent Class mapping to the top label (= ). Redesign planned. 44

45 MPLS-Aware NetFlow Top Label Aggregation Fields Key fields (uniquely identifies the flow) Input ifinde The top incoming MPLS labels with eperimental bits and end-of-stack bit Additional eport fields Flows Packets Bytes Timestamps (sysuptime) IP Net Hop Output interface Accumulation of TCP Flags Type of the top label: LDP, BGP, VPN, ATOM, TE tunnel MID-PT, unknown The FEC mapping to the top label Supported in 12.0(25)S 45

46 MPLS-Aware NetFlow Configuration ip flow-cache mpls label-positions [label-position-1 [label-position-2 [label-position-3]]] [noip-fields] [mpls-length] no-ipfields labelposition-n mplslength Position of an MPLS Label in the Incoming Label Stack; Label Positions Are Counted from the Top of the Stack, Starting with 1 Controls the capture and reporting of MPLS flow fields. If the no-ip-fields keyword is not specified, the following IP related flow fields are included: Source IP address Destination IP address Transport layer protocol Source application port number Destination application port number IP type of service (ToS) TCP flag (the result of a bitwise OR of TCP Controls the Reporting of Packet Length; If the mpls-length Keyword Is Specified, the Reported Length Represents the Sum of MPLS Packet Payload Length and the MPLS Label Stack Length; If the mpls-length Keyword Is Not Specified, Only the Length of the MPLS Packet Payload Is Reported 46

47 Cisco Series Internet Routers MPLS-Aware NetFlow (v9) Engines 0, 1, 2, and 3 Up to 3 labels and IP packet header fields Engine 4 Not supported Engine 4+ 1 label and IP packet header field MPLS-Aware NetFlow supported in Cisco IOS software release 12.0(24)S MPLS-Aware NetFlow top label aggregation supported in Cisco IOS software release 12.0(25)S 47

48 Egress MPLS NetFlow For accounting of MPLS Layer 3 VPN traffic, i.e. the MPLS to IP traffic coming from the core Enable on IP interface Tracks egress traffic NetFlow version 5 and version 8 IP MPLS Traffic Flow IP Can be enabled on sub-interfaces All other NetFlow commands still apply Supported in 12.0(10)ST, 12.1(5)T and 12.0(22)S router(config-if)#tag-switching ip flow egress 48

49 Output Sampled NetFlow New Enable on IP interface Tracks egress traffic Tracks both MPLS to IP and IP to IP Only supported on engine 3 (IP Service Engine (ISE)) line card Supported in 12.0(24)S (12.0(26)S added input interface) IP MPLS Traffic Flow IP router(config-if)#ip route-cache flow sampled [input output] 49

50 MPLS Aware NetFlow The Core Traffic Matri AS1 AS2 AS3 AS4 AS5 Customers CE CPE PE PE PE PoP P P MPLS Core P P P PoP PE PE PE Customers CE CPE Server Farm 1 Server Farm 2 Internal Traffic: PoP to PoP, the PoP being the CPE or CE Eternal traffic matri PoP to BGP AS 50

51 Accounting and Analysis BGP and Autonomous Systems RST _05_2004_c Cisco Systems, Inc. All rights reserved. 51

52 BGP Autonomous System NetFlow Enabled AS 101 AS 102 AS 103 AS 104 Configuring Peer-AS Source AS = AS 103 Destination AS = AS 105 AS 105 Router(config)#ip flow-eport version 5 peer-as AS 106 Note: The AS Fields Will Remain Empty unless You Configure It Eplicitly with peer-as or origin-as 52

53 BGP Autonomous System NetFlow Enabled AS 101 AS 102 AS 103 AS 104 Configuring Origin-AS Source AS = AS 101 Destination AS = AS 106 AS 105 Router(config)#ip flow-eport version 5 origin-as AS 106 Note: The AS Fields Will Remain Empty unless You Configure It Eplicitly with peer-as or origin-as 53

54 BGP net-hop NetFlow-Enabled Here AS 1 AS 2 AS 3 Router1 Router2 Router3 Router4 Router5 Traffic Flow The IGP resolved net hop is Router3, so IP net-hop is Router3 The BGP net-hop is Router 5 (by IOS default configuration) If neighbor a.b.c.d net-hop self is configured (disables BGP net-hop calculation) then BGP net-hop is Router 4 54

55 BGP net-hop Details Supported only in version 9 eport For traffic engineering/analysis (traffic matri) and possible billing applications. What is the Net Hop IP address of my BGP Traffic? Eported fields include all version 5 fields, including IP Net Hop Adds 16 bytes to each NetFlow flow record (goes from 64 bytes to 80 bytes), while CPU increase is negligible Edge to Edge traffic matri for engineering/analysis and possible billing applications Supported in Cisco IOS Software Releases 12.0(26)S, 12.2(18)S, and 12.3(1) 55

56 NetFlow Version 9 Configuration Configuring Version 9 Eport!"# "# $, -.!"# "#. Configuring Version 9 Eport with BGP Net-Hop!"# "#.$ 45*67 %& %&!"# "#. 56

57 NetFlow BGP Net-Hop TOS Aggregation New Key fields (uniquely identify the flow) Origin AS Destination AS Inbound interface DSCP Additional eport fields Flows Packets Bytes Timestamps (sysuptime) Net BGP hop Output interface Note IP Net-Hop is not included Available now in releases 12.0(26)S, 12.2(18)S and 12.3(1) Note: Not supported on the 4k/6k/10k 57

58 NetFlow BGP Net Hop TOS Aggregation Remote Traffic (Different AS) Source Local Traffic (Within AS) 58

59 NetFlow Collector 5.0 BGP Features New NFC collects NetFlow records as a passive BGP peer to receive the full BGP table from each eporting router Allows BGP attribute correlation to flow records Fields include: BGP AS path BGP Net Hop (if not provided via router) BGP community (in NFC 5.1) Stay tuned! 59

60 Accounting and Analysis Multicast Options RST _05_2004_c Cisco Systems, Inc. All rights reserved. 60

61 Multicast NetFlow Three Types of NetFlow Implementations for Multicast Traffic: 1. Traditional NetFlow 2. Multicast NetFlow Ingress 3. Multicast NetFlow Egress 61

62 Multicast: Traditional NetFlow Traditional NetFlow Configuration 1: (S, G) - ( , ),///8 NetFlow Collector Server Eth 0,89///,.,89///,...- Eth 1 Eth 3 Eth 2 Flow Record Created in NetFlow Cache Srclf SrclPadd Dstlf DstlPadd Protocol TOS Flgs SrcPort SrcMsk DstPort DstMsk NetHop Bytes Packets Active Idle Eth Null A2 /24 00A2 / There is only one flow per NetFlow configured input interface Destination interface is marked as Null Bytes and Packets are the incoming values Note: C 6500/7600 Accounts for Multicast Traffic in This Way in PFC3b (Sup720) 62

63 Multicast NetFlow Ingress (v9) Multicast NetFlow Ingress Configuration 1: (S, G) - ( , ),///8 NetFlow Collector Server Eth 0,89///,.,89///,...- Eth 1 Eth 3 Eth 2 Flow Record Created in NetFlow Cache Srclf SrclPadd Dstlf DstlPadd Protocol TOS Flgs SrcPort SrcMsk DstPort DstMsk NetHop Bytes Packets Active Idle Eth Null A2 /24 00A2 / There is only one flow per NetFlow configured input interface Destination interface is marked as Null Bytes and Packets are the outgoing values 63

64 Multicast NetFlow Egress (v9) Multicast NetFlow Egress Configuration (S, G) - ( , ) NetFlow Collector Server 1:, 1:8 1:,///8 Eth 0 Eth 1 Eth 3,89///,.,89///,...- Eth 2 Flow Records Created in NetFlow Cache Srclf SrclPadd Dstlf DstlPadd Protocol TOS Flgs SrcPort SrcMsk DstPort DstMsk NetHop Bytes Packets Active Idle Eth Eth A2 /24 00A2 / Eth Eth A2 /24 00A2 / Eth Eth A2 /24 00A2 / There is one flow per Multicast NetFlow Egress configured output interface One of the 7 Key fields that define a unique flow has changed from source interface to destination interface Bytes and Packets are the outgoing values 64

65 Multicast NetFlow: Summary Supported via NetFlow version 9 eport format Performance: Ingress vs. Egress Multicast NetFlow Ingress and traditional NetFlow will have similar performance numbers Multicast NetFlow Egress will have performance impact that is proportional to the number of interfaces on which it is enabled (include input interfaces) Availability Cisco IOS software releases 12.0(27)S, 12.2(18)S and 12.3(1) Not supported in Cisco C6500/7600 series switches Do not currently support the tracking of multicast traffic via NetFlow due to current ASIC limitation Will have this support in a future supervisor 65

66 Attack Security Features and Applications RST _05_2004_c Cisco Systems, Inc. All rights reserved. 66

67 How to Identify a Security Attack? Suddenly highly-increased overall traffic in the network Higher CPU and memory utilization of network devices Unepectedly large amount of traffic generated by individual hosts Increased number of accounting records generated Multiple accounting records with abnormal content, like one packet per flow record (e.g. TCP SYN flood) A changed mi of traffic applications, e.g. a sudden increase of unknown applications An increase of certain traffic types and messages, e.g. TCP resets or ICMP messages An increasing number of ACL violations 67

68 What Does a DoS Attack Look Like? Potential DoS Attack on Router Estimated: 660 pkt/s Mbps ;# < &1 &1* &* &%& )1 )1* )* 8.,.8/,//.99.,. /8/8/8,2 8.,.8/,//888,8.,. /8/8/8,99 8.,.8/,//,2,9.,. /8/8/8,2. 8.,.8/,//,-...,. /8/8/8,- 8.,.8/,//- 9.,. /8/8/88,2 8.,.8/,//,--..,. /8/8/8,28, 8.,.8/,//8,2.,. /8/8/8,-, 8.,.8/,//,,, -.,. /8/8/8,2. 8.,.8/,//8.,8..,. /8/8/8, )%& **= *= 4>*=,,,,,,,,, < < < < < < < < < < < Typical DoS Attacks Have the Same (or Similar) Flow Entries: Input Interface (SrcIf) Destination IP (DstIf) 1 Packet per flow (Pkts) Bytes per packet (B/Pk) 68

69 NetFlow: Mitigating Attacks Method 1 Cost Saving 1. sh ip cache flow command to find top volume flows 2. Identify source of attack 3. Write access list to block 4. Monitor via show ip cache flow and null entry in DestIf field to show it s being blocked 5. Note you can configure prefi-port aggregation and use sh ip cache flow aggregation prefi port Method 2 Most Effective Arbor Networks leverages NetFlow to provide a quicker response and more sophisticated solution than described above 69

70 Tracing DoS Attack with NetFlow 1/2 1. To show high rate flows router# show ip cache flow include (K M) 2. To show all flows to one destination leverage router# sh ip cache flow include <destination> eample: #,. /8/8/8 < &1 &1* &* &%& )1 )1* )* )%& **= *= 4>*= 8.,.8/,//.99.,. /8/8/8,2, 8.,.8/,//888,8.,. /8/8/8,99, 8.,.8/,//,2,9.,. /8/8/8,2., 8.,.8/,//,-...,. /8/8/8,-, < < < < < < < < < < < 3. To look for known attack signatures e.g. if we know of an attack using UDP port 666 (He 029A) we run router# show ip cache flow inc 029A 70

71 Tracing DoS Attack with NetFlow 2/2 Enable NetFlow on relevant routers/switches Victim router1#sh ip cache flow include <destination> Se1 <source> Et0 <destination> (lot of more flows to the same destination) The Flows Come from Serial 1 router1#sh ip cef s1 Prefi Net Hop Interface / Serial1 Find the Upstream Router on Serial /30 attached Serial1 Continue on This Router 71

72 Flow Chart to Identify a High Number of Flows Collect Continuously OR Start Collection Every X Minutes Count Flows During Time Interval Z IF Number of Flows > N n End y Alarm! 72

73 DoS Attack Eample: Arbor Networks Configure NetFlow Eport to Arbor DoS Collector(s) Service Provider C Service Provider A Service Provider B X 1. Profile: Baseline Traffic Patterns in the Network 2. Monitor: Analyze Traffic for Anomalies 3. Detect: Forward Anomaly Fingerprints to Controllers 4. Trace: Trace the Attack to Its Source 5. Filter: Recommends Filters (X) IDS Firewall Customer Web Server 73

74 NetFlow-Based Traffic Characterization 74

75 DoS: Technical Alternatives after NetFlow ACLs Manual Performance impact Unicast Reverse Path Forwarding (urpf) Automate with BGP Only stops non-eisting sources MQC Policing: Automate via QPPB (QoS Policy Propagation with BGP) Performance impact 75

76 DoS: Administrative Alternatives after NetFlow If source address of flow is not spoofed (falsified): Use Routing table for prefi from which IP source comes ( show ip route <source ip> and/or show ip cef <source ip> ) For source IP or source/peer AS use Internet Routing Registry (IRR: Europe whois.ripe.net, Asia-Pac whois.apnic.net, USA and rest whois.arin.net) direct site contact (abuse@domain) If source address of flow is spoofed (falsified): Trace packet flow back through the network using NetFlow Find upstream ISP via NetFlow incoming interface on edge router Upstream ISP needs to continue the tracing 76

77 NetFlow MIB and Top N Talkers New Snapshot of current live NetFlow cache via SNMP Administration and configuration of NetFlow using the MIB interface NetFlow MIB cannot be used to retrieve all flow information, but provides useful traffic snapshots: Packet size distribution Number of bytes eported per second Number of flows This is targeted at Denial of Service (DoS) attacks, security monitoring and remote locations where eport to a local NetFlow collector is not possible NetFlow MIB available in release 12.3(7)T Top N Talkers Top N Flows based on various NetFlow field values MIB and CLI support 12.2(25)S and 12.3(11)T 77

78 How Cisco IT Uses NetFlow Characterize IP traffic and account for how and where it flows Total avoidance of SQL slammer worm Transitioned from managed DSL service to internet VPN Detection of unauthorized WAN traffic Validation of QoS parameters and BW allocation Analysis of VPN traffic and tele-commuter behavior Calculating total cost of ownership for applications Use of NetFlow Security Monitoring WAN Aggregation and Edge Core routers and Nat Gateway NMS and Usage Network Traffic Analysis by Application with BGP; Anomaly Detection Arbor Networks Network Traffic Analysis by Application, for Capacity Planning Using NetQOS Collection of Historical Data, Useful for Forensics and Diagnostics with Flow Tools 78

79 Powerful Insight into Tunnels with NetFlow NetFlow Totals Tunnel Packets into One Flow Non-Tunnel Router Tunnel Head Tunnel Midpoint Tunnel Tail Non-Tunnel Router Tunnel Traffic NetFlow Accounts for Packets Prior to IPSec Tunnel Enable Here: NetFlow Accounts for Both the Tunnel and Post-Tunnel Flows NetFlow Accounts for Packets Prior to IPSec Tunnel NetFlow lets you break out both pre and post encryption Support for both GRE and IPSec encryption Tested with 12.3 images Paper at under Technical Documents 79

80 Scaling Features and Options RST _05_2004_c Cisco Systems, Inc. All rights reserved. 80

81 Scaling Memory Utilization A NetFlow cache entry (a single flow) is 64 bytes Platform w/ 64MB DRAM 7200 w/ 128MB DRAM 7500 w/ 64MB DRAM 7500 w/ 128MB DRAM C6500/7600 Sup1/PFC1 C6500/7600 Sup2/PFC2 C6500 / 7600 Sup720/PFC3b w/ 64MB DRAM w/ 128MB DRAM Configuration: Default NetFlow Cache Size (Entries) 4k 4k 4k 64k 128k 64k 128k 32k 32k 256k 64k 64k router(config-if)#ip flow-cache entries <number> 81

82 Scaling Sample Traffic Deterministic vs. Random Sampling Deterministic Sampled NetFlow: Sampling 1 out of 8 Packets NetFlow Always Chooses 8 th Packet for Eport NetFlow Always Chooses 8 th Packet for Eport Eport Flow Random Sampled NetFlow: Sampling 1 out of 8 Packets NetFlow Randomly Chooses 5 th Packet for Eport NetFlow Randomly Chooses 2 nd Packet for Eport Eport Flow 82

83 Sampled NetFlow Details Deterministic Cisco C 6500/7600 series switches (12.1(13)E) Cisco series internet routers (12.0(11)S and 12.0(14)ST) Random (select packet to eport per statistical principles) Cisco IOS Software Releases 12.0(26)S, 12.2S(18), and 12.3(1)T Cisco 800, 1700, 1800, 2600, 2800,3600, 3700, , and 7500 Series Routers Random sampling Cisco Series 12.0(28)S Cisco Series deterministic sampling today Time-based Cisco C 6500/7600 Series Random and Time based sampling 12.1(13)E 83

84 Catalyst 6500/7600 Sampled NetFlow Support for both time and (packet-based) deterministic sampling Sampling rate is configurable only for the whole bo Accuracy of NetFlow on the platform comes from tuning the aging timers correctly A way of minimizing packet loss, is using DFC (Distributed Forwarding Card) cards, spreading the incoming packet load evenly onto different VLANs on different cards Available now in release 12.1(13)E 84

85 Cisco Series Internet Routers Sampled NetFlow Engine Full NetFlow Supported Supported Sampled NetFlow Supported Supported Supported Supported Supported Supported Not Supported Despite ASIC Support in Engine 2, 3 and 4+ Linecards Full NetFlow Still Inflicts a Heavy Burden on Memory and Therefore Sampled NetFlow Is Preferred 85

86 Sampled NetFlow CPU Reduction and 3620: CPU Impact Reduced by at Least 75% with Sampling Rate of 100 and 82% with Sampling Rate of Full NetFlow Load [%] :100 Sampling 1:1000 Sampling No NetFlow Samples 86

87 Configuring NetFlow on a Subinterface New Receive NetFlow information only on the specific subinterface(s) of interest Reduces CPU and memory impact on router as well as eport traffic and collector sizing needs Router(config-if)#ip flow ingress New ip flow ingress command is easier to distinguish between egress NetFlow commands Same ip flow ingress command can now be used to configure NetFlow on the main interface Available now in releases 12.2(14)S and 12.2(15)T Note: NetFlow Has Always Eported Subinterface Information 87

88 NetFlow Multiple Eport Destinations New Two identical streams of NetFlow data are sent to the two destination hosts (collectors); currently the limit is two destinations router(config)#ip flow-eport destination router(config)#ip flow-eport destination Main and aggregation caches supported Available now in releases 12.0(19)S, 12.0(19)ST, 12.2(2)T and 12.2(14)S Available in C 6500/7600 in Cat 8.3 and 12.2(14)S on MSFC3 and Sup720 88

89 NetFlow Input Filters: Overview New Flow filter prevents flows from entering NetFlow cache Increases scalability and decreases CPU usage Filters are based on Quality of Service (QoS) modular QoS Command Line Interface (MQC) class maps User can match flows from a certain port/source with ACL Define traffic class (match ACL) and flow sampling per match Available in release 12.0(27)S, 12.3(4)T, 12.2S(25) Packets Traffic Filter High Importance Traffic Filter Low Importance Sample 1:1 from Server B Sample 1:100 from Subnet A 89

90 NetFlow Input Filters: Details New Pre-filters traffic prior to NetFlow processing Modular QoS command line (MQC) provides the filtering mechanism for NetFlow classification by: IP source and destination addresses Layer 4 protocol and port numbers Incoming interface ToS byte (includes DSCP and IP precedence) MAC address Layer 2 information (such as Frame Relay DE bits or Ethernet 802.1p bits) Network-Based Application Recognition (NBAR) Ability to sample filtered data at different rates, depending on how interesting the traffic is Available now in release 12.3(4)T 90

91 NetFlow Input Filters: Eample New Packets VOIP Tight Filter for Traffic of High Importance 1:1 Sampling VPN Moderately-Tight for Traffic of Medium Importance 1:100 Sampling NetFlow Cache Best Effort Default Wide Open Filter for Traffic of Low Importance 1:10000 Sampling 91

92 NetFlow Performance Paper Tests Tests Ran: Access lists (ACLs) 200 and 500 lines 0, 1, and 2 NetFlow data eport destinations Initial performance after enabling V8 Aggregation vs. v5 Configuring AS origin or peer Policy Based-Routing (PBR) Full NetFlow vs. 1:100 sampled NetFlow Platforms: 2600, 3600, 7200 NPE-400 and NSE-1, 7500 RSP8 VIP4-80 with CEF and dcef, Engine 1 Linecard dcef 92

93 NetFlow Performance Paper Conclusions Additional CPU utilization Number of Active Flows 10,000 45,000 65,000 Additional CPU Utilization <4% <12% <16% NetFlow data eport (single/dual) No significant impact NetFlow v5 vs. v8: Impact depending on the number of aggregations enabled. (Additional 6% for multiple aggregations) NetFlow feature acceleration: >200 lines of ACLs and/or Policy Based-Routing (PBR) NetFlow vs. sampled NetFlow on the Cisco series internet routers: 23% vs. 3% (65,000 flows, 1:100) 93

94 Performance Testing NetFlow Version 9 Similar CPU and throughput numbers result from configuration of both NetFlow version 5 and 9 CPU is slightly higher immediately following initial boot up or configuration Caused by sending template flowsets to collector BGP Net-Hop performance is almost identical to v5 results, however MPLS-aware NetFlow is a bit more 94

95 NetFlow Performance Summary Enabling NetFlow version 5 and eporting increases the CPU utilization by around 15% (with a ma of 20% depending on the platform) Enabling NetFlow version 8 increases the CPU utilization by 2 to 5% above version 5, depending on the number of aggregations enabled NetFlow is implemented in hardware on the Cat6500/7600 supervisor; only the eport takes CPU cycles NetFlow version 9: similar results as version 5 Memory usages is 64 bytes per flow; so to have room for 64,000 flows 4 MB of DRAM is required 95

96 Technical Advice: Reducing Performance Impact Reduce CPU and Memory Impact on the Router, Collector, and Network: Aging timers Sampled NetFlow Leverage distributed architectures (VIP, Linecards) Flow masks (only C 6500/7600) Enable on specific subinterface Aggregation schemes (v8 on router or on collector) Filters (router or collector) Data compression (collector) Increase collection bucket sizes (collector) Collector and router can be placed on the same LAN segment (network) 96

97 Eport Collector, NAM and Partners RST _05_2004_c Cisco Systems, Inc. All rights reserved. 97

98 NetFlow Infrastructure Cisco Cisco and Partners Partners Network Planning RMON/NAM Accounting Billing Router: Cache creation Data eport Aggregation Collector: Collection Filtering Aggregation Storage Applications: RMON Application Data processing Data presentation 98

99 NetFlow Collector (NFC) Overview What Does NFC Do? Collect eported NetFlow records from network elements Filter Aggregate flow records Integrate eternal data into records, e.g. adding BPG attributes Map ranges of values from one or more fields to user-defined strings Compresses records (optionally) Stores NetFlow accounting records (ASCII or binary) Web-based GUI (NFC 5.0) to sort, graph, eport, filter, and drill down on report data Eport e.g..csv eport to MS Ecel 99

100 NFC 5.0 Features What Is New in NFC 5.0? Web-based user interface XML configuration Report generator MPLS/VPN PE-PE traffic reports BGP peer for attribute correlation Interface name mapping DNS lookup MPLS/EXP support Self-describing header Generic field mapping Ma burst rate support V5 sampled NetFlow header support Enhanced logging IPv6 support Platform Requirements: Solaris 8/9 HP-UX 11i Linu Red Hat Enterprise Japanese Linu version will be fully supported in NFC 5.1 Note: 2-4 GB RAM and Dual Processors Recommended 100

101 NFC 5.0 Key Features: Web-Based Interface NFC Reports Provide the User with the Ability to Sort, Graph, Eport, Filter, and Drill Down on Report Data 101

102 NetFlow on the Network Analysis Module (NAM) NetFlow collection and analysis combined Instant results NAM offers powerful combination of NetFlow and RMON (mini-rmon, RMON1, RMON2, HCMON, SMON, and DSMON) RMON2 can provide additional application level visibility (L5-7) ART Application Response Time MIB Packet decoding Detail analysis of traffic of interest RMON/NetFlow Support in NAM GUI Applications RMON and NF Hosts RMON and NF Conversations RMON and NF Voice RMON VLAN RMON ART RMON DiffServ RMON Portstats RMON 102

103 NetFlow with NAM Web-Based GUI Bar Charts, Pie Charts, Usage, etc Troubleshooting Drill Down Setting Alarm Thresholds 103

104 NetFlow Partners Traffic Analysis Collection Flow-Tools Denial of Service Billing 104

105 Evolving NetFlow Deployment and Direction RST _05_2004_c Cisco Systems, Inc. All rights reserved. 105

106 NetFlow Deployment: Level of Collection Details Link statistics or traffic details: SA, DA Application details (port numbers) QoS Time stamps Routing and peering Layer 2 or Layer 3 information Eport all details or aggregated data records Data eport: push or pull model Collection interval and history Consider the generated data volume 106

107 NetFlow Deployment: Rules of Thumb Monitor the router CPU (<60%) and memory before enabling NetFlow For optimized eport: aggregate on router/switch rather than on the collector If eporting version 8 on router don t also eport another version (5, 7, or 9) Eport via a dedicated interface/vlan for easier troubleshooting and management Keep collector on LAN interface 1 hop away: Avoid drops WAN interfaces have less bandwidth to afford NetFlow eport creates ~1% to 1.5% of the interface throughput that NetFlow is enabled on Enable the ifinde persistence if accounting per interface 107

108 NetFlow Deployment: Considerations Edge Core Edge Edge NetFlow positives: Interface is key field Full NetFlow and sampled NetFlow options Account for all CE/end user traffic Edge considerations: IP addressing pre or post NAT Collectors: a) # required b) locations c) aggregating all data Core NetFlow positives: Collectors can be centrally located TCP flags tracking on IP addressing pre or post NAT Core considerations: Sampled NetFlow recommended Amount of collection information Missing flows at the edge? 108

109 NetFlow and IPv6 New Collects IPv6 flow records Based on NetFlow version 9 Support or both ingress and egress traffic Full NetFlow i.e. non-sampled Data eport is still IPv4 Available in release 12.3(7)T 109

110 NetFlow Roadmap Scalability & Fleibility Enhancing Cisco Technologies with Flow Accounting Optimizing Data for Flow Processing Standardization Nov 2003 Dec 2003 Jan 2004 Feb 2004 Mar 2004 Apr 2004 May 2004 Jun 2004 Jul 2004 Aug 2004 Sep 2004 Oct 2004 Nov 2004 Dec 2004 Jan 2005 Feb 2005 Mar (Rls2)T Input Filter 12.0(27)S Input Filter Targeting 12.3(2)T NetFlow MIB & Top Talker NetFlow IPv6 Targeting 12.2(25)S NetFlow MIB & Top Talker Input Filter Targeting 12.3(11)T Egress NetFlow Targeting 12.2(Rls6)S Egress NetFlow Targeting 12.4(Rls1)T Security Eports Targeting 12.2(Rls7)S Fleible Flow Definition Reliable Eport Security Eports MIB Phase 2 110

111 NetFlow Collection Engine Roadmap : Committed/Delivered : Not committed : Patch CYQ2-04 CYQ3-04 CYQ4-04 CYQ1-05 CYQ2-05 CYQ3-05 CYQ4-05 CYQ1-06 CYQ2-06 CYQ3-06 Web-Based Interface VPN PE-PE traffic statistic DNS and Interface name look up Mac Burst Rate QOS TOS byte( include Ep field) Top N summary IPv6 Random Sampling NetFlow Passive BGP Peering AS path Radar 1) Correlation Module Cafeteria style Aggregation PE-PE; CE-PE and PE-CE Traffic statistic for CE-PE; PE-CE and PE-PE Soap interface with ISC Interface via flat file 2) BGP field keys (case AS path changes) 3) NFCs synchronization 4) Integration with Janko and Japan partners 5) Japanese Linu version Radar High Availability Security features Fleible Flow Definition NetFlow Egress Limited trend analysis Phase two Integration with & subset of NMTG CS 5.0 Cisco IT requirements Radar Integration NAM Blade Security Features Integration with CS Cisco IT requirements NFC CC on 07/27/ SCTP Prototype only Web GUI Enhancement Radar Navigation trees (sort node... Etc ) handle boundary conditions (out-of-memory etc..) Fleible Flow - FNF Enhancement NFC5.0 configuration (filters; NetFlow Egress aggregators; v9 field format; schedule report;..etc..) Limited trend GUI report for VPN PE-PE report analysis Phase one Enhancement of GUI Per Boeing requests Integration with & FleM License subset of NMTG CS BGP attribute available 5.0 Migration tools Leverage in NMTG CS 4.0 Net Gen of Appliance 6.0 (Sub-set of NMTG CS) Target: FCS Dec,

112 Upcoming New Features: NetFlow Product Update Future NetFlow Security Enhancements (Q2CY2005) New eports and show commands for security monitoring Fleible Flow Keys (Q3CY2005) Allow user defined flow keys and aggregation with v.9 Reliable and Congestion Aware Eport (Q2CY2005) SCTP protocol NetFlow eport NBAR and NetFlow Integration (Radar) Application flow information eport 112

113 SCTP Reliable Transport Future Flows may be sent in reliable, unreliable or partial reliable mode SCTP connection to collector and multiple streams per connection Supported with Version 9. Templates may be sent reliably Congestion Awareness, retransmission and queuing Releases 12.4(2nd)T, 12.2S(Rls7) Send Queue Template Flow Set (reliable) Data Flow Set (unreliable) Template Flow Set Data Flow Set (no congestion) Collector Data Flow Set (under congestion, potentially dropped) 113

114 NetFlow Security Enhancement Future New show commands to understand and parse NetFlow data For Eample, show flows on port X to destination Y show ip flow top <N> <aggregate-field> <sort-criteria> <match-criteria> show ip flow top 10 destination-address packets interface ser0 port-range 100 to 135 New Flow eport fields including Source Mac, TTL, Packet length, ICMP type, and more Also will be available in 12.2(rls7)S 114

115 Fleible Flow Keys Future Today Fied 7 Keys Source IP address Destination IP address Source port Destination port Layer 3 protocol type TOS byte (DSCP) Input logical interface Define Any Keys for a Flow Source IP address Destination IP address Source port Destination port Advantages User Defined Flow information Isolate and account for Specific Information Increase performance reduced data in flow Reduced Cache Decrease Flow Eport BW New Flow keys possible on ingress and Egress 115

116 Addressing The Business Needs with NetFlow Accounting: Primary Cisco accounting technology; Current economic environment drives need to costjustify, and charge for IT network rollout/service provider premium services Analysis: Key Cisco IOS network management feature for Performance and Fault Management Traffic matri: Primary technology for building core traffic matrices Attack: Primary technology for identifying denial of service attacks 116

117 NetFlow Summary NetFlow is a mature Cisco IOS feature (in Cisco IOS since 1996) NetFlow provides input for Accounting, Performance, Fault, Security, and Billing Applications Cisco has IETF and industry leadership NetFlow v9 eases the eporting of additional fields A lot of new features have been added Stay tuned for more 117

118 References NetFlow Cisco Network Accounting Services Comparison of Cisco NetFlow versus other available accounting technologies nwact_wp.htm Cisco IT Case Study business.cisco.com/prod/tree.taf%3fasset_id=106882&it= &public_view=true&kbns=1.html Cisco NetFlow Collector/Analyzer rtrmgmt/inde.htm A complete white paper nfwhite.htm 118

119 Q & A 119

120 2004, Cisco Systems, Inc. All rights reserved. 120