Catalyst 6500 Architecture 2
Session Goal To provide you with a thorough understanding of the Catalyst 6500 switching architecture, packet flow, forwarding engine functions, and key feature operations. 3
Agenda Chassis and Power Supplies Supervisor Engine and Switch Fabric Architectures Module Architectures Layer 2 Forwarding IP Unicast Forwarding NetFlow Access Control Lists Packet Walks 4
Catalyst 6500 E-Chassis Family 6509-V-E 6513-E 6509-E 6506-E 6504-E 6503-E 7 Chassis Members From 3 Slot to 13 Slot 5
Catalyst 6500 E-Series Chassis Inside the Chassis FABRIC BUS Linecard Slots Supervisor 32/720/2T Slots Linecard Slots 6
Catalyst 6500 Switch Backplanes Classic (32Gb) BUS Backplane 720Gb / 2Tb Crossbar Backplane DBUS RBUS EOBC CROSSBAR Linecard Linecard Linecard Linecard Linecard Data Bus (DBUS) allows L/C to forward data to Supervisor for forwarding decision Results Bus (RBUS) returns forwarding result from Supervisor back to L/C Ethernet Out of Band Channel (EOBC) provide out of band management between Supervisor and LC Crossbar is a matrix of N channels to provide a data path between linecards Sup720 supports 18 channels at 8G/20G per channel (speed autodetected) Sup2T supports 26 channels at 20G/40G per channel (speed autodetected) 7
Catalyst 6500 Linecard Slot Support Slot 6503/ 6503-E 6504-E 6506/ 6506-E 6509/ 6509-E 6509- NEBS-A 6509-V-E 6513 6513-E 1 Dual Dual Dual Dual Dual Dual Single Dual 2 Dual Dual Dual Dual Dual Dual Single Dual 3 Dual Dual Dual Dual Dual Dual Single Dual 4 - Dual Dual Dual Dual Dual Single Dual 5 - - Dual Dual Dual Dual Single Dual In order to take advantage of the dual fabric channels in slots 1 8 of the 6513-E chassis, the Supervisor 2T is required. 6 - - Dual Dual Dual Dual Single Dual 7 - - - Dual Dual Dual Single Dual 8 - - - Dual Dual Dual Single Dual 9 - - - Dual Dual Dual Dual Dual 10 - - - - - - Dual Dual With any version of the Supervisor 720, the 6513-E fabric channel distribution Is the same as the 6513. 11 - - - - - - Dual Dual 12 - - - - - - Dual Dual 13 - - - - - - Dual Dual 8
Power Supply Redundancy The Catalyst 6500 can utilize two power supplies to work in either Combined or Redundant Mode Redundant Mode Catalyst 6500 Combined Mode Catalyst 6500 50% 50% 83% 83% Use the Cisco Power Calculator on cisco.com to determine which supplies and which mode of operation is needed for your system. Power Supply 1 Power Supply 2 Each power supply operates at ~50% capacity Neither supply operates at >60% or <40% capacity If one fails, the second supply can power the system on its own This is the default and recommended configuration for the power supplies Power Supply 1 Power Supply 2 Each power supply provides up to 83% of its capacity The total system power available is 167% of the capacity of a single supply If one fails, the second supply may not be able to power the system on its own - this could result in devices or linecards being shut down This is not the recommended mode for production 11
Agenda Chassis and Power Supplies Supervisor Engine and Switch Fabric Architectures Module Architectures Layer 2 Forwarding IP Unicast Forwarding NetFlow Access Control Lists Packet Walks 12
Catalyst 6500 Supervisors Supervisor 720: Some Facts Supervisor 720 Quick Facts Integrated 720Gbps Switch Fabric Supervisor 720 3A / 3B / 3BXL Supervisor 720-10G 3C / 3CXL Integrated Policy Feature Card 3 (PFC3) supporting hardware acceleration for select features Integrated Multilayer Switch Feature Card 3 (MSFC3) supporting two CPUs for Layer 2 and Layer 3 functionality IPv6 unicast and multicast forwarding support in hardware Virtual Switching System (VSS) support with Sup720-10G models All uplinks can be active in systems with redundant Supervisors (more information Cisco in Public the notes) 13
Supervisor 720 3A / 3B / 3BXL Classic BUS EOBC Rbus Dbus Crossbar Fabric Channels L2 CAM Switch Fabric Layer 2 FWD Engine FIB TCAM FIB Table QOS ACL RP Flash SP Flash RP DRAM SP DRAM RP SP MET Fabric / Replication Port Port Layer 3 FWD Engine Security ACL Counters Netflow TCAM Netflow Table Netflow Stats Adjacency Adj Stats MSFC3 1G 1G 1G Policy Feature Card (PFC3) 14
Supervisor 720-10G 3C / 3CXL Classic BUS Crossbar Fabric Channels EOBC Rbus Dbus Fabric RP SP Switch Fabric MET 20Gbps Fabric / Replication L2 CAM Layer 2/3 FWD Engine FIB TCAM FIB Table QOS ACL Security ACL Counters Netflow TCAM Netflow Table Netflow Stats RP Flash RP DRAM SP Flash SP DRAM MSFC3 10G Port Quad Port PHY 1G 1G Port 1G 10G Adjacency Adj Stats Policy Feature Card (PFC3) 15
Catalyst 6500 Supervisors Supervisor 2T: Some Facts Supervisor 720 Quick Facts Integrated 2-Tbps Switch Fabric Integrated Policy Feature Card 4 (PFC4) supporting hardware acceleration for select features Supervisor 2T PFC4 / PFC4XL Integrated Multilayer Switch Feature Card 5 (MSFC5) supporting a single CPU for L2 and L3 functionality Connectivity Management Processor (CMP) for improved management capability One external compact flash slot (power controlled by IOS) All uplinks can be active in systems with redundant Supervisors (more information Cisco in Public the notes) 16
Supervisor 2T PFC4 / PFC4XL Classic BUS Crossbar Fabric Channels EOBC Rbus Dbus Fabric DRAM CPU Flash MSFC5 Switch Fabric MET 10G Port Fabric / Replication 1G 20Gbps Quad Port PHY 1G Port 1G 10G L2 NetFlow CAM LIF Table PFC4 FIB TCAM FIB Table CL2 TCAM QOS ACL Security ACL Layer 2/3 L3/4 Forwarding FWD Engine Engine LIF DB LIF Stats CL1 TCAM Counters Netflow TCAM Netflow Table Netflow Stats Adjacency Adj Stats L2 Forwarding Engine L2 CAM (128K) Policy Feature Card (PFC4) FIB ADJ RPF Table ACE Counter 17
Supervisor Chassis Requirements Chassis Supervisor 720s All E-Series All non-e Series E-Fans for E-Series Supervisor 2Ts Only E-Series E-Fans for E-Series E-Fan cannot be used in non-e Series Chassis Fan2 cannot be used in E-Series Fan Trays Power Supplies Fan2 for non-e Series 2500W AC / DC or greater With Supervisor 2T and 6513-E, only Supervisors are allowed in the Supervisor Slots 3-Slot : 1 and 2 Supervisor Slots 4-slot : 1 and 2 6-slot : 5 and 6 9-slot : 5 and 6 With Supervisor 720 and 6513-E, the fabric channel distribution is the same as with Supervisor 720 and 6513. 13-slot : 7 and 8 18
Catalyst 6500 Supervisors Switch Fabric The Supervisor 720 and Supervisor 2T support a Switch Fabric which offers each connected linecard a set of discrete communication paths into the switch backplane Linecard Slot #9 Linecard Slot #8 Linecard Slot #1 Linecard Slot #2 Linecard Slot #7 Linecard Slot #6 Data Flows Linecard Slot #3 Linecard Slot #4 Supervisor Slot #5 19
Catalyst 6500 Supervisor 720 The 720Gbps Switch Fabric Switch Fabric - Integrated 720Gbps Switch Fabric - Provides backplane interconnects between linecards - Fabric Traces are distributed across each linecard slot - Each Fabric Trace can run at 8Gb/sec OR 20Gb/sec 20
Catalyst 6500 Supervisor 2T The 2Tbps Switch Fabric - Integrated 2Tbps Switch Fabric Switch Fabric - 26 Channels to support the 6513-E - Provides backplane interconnects between linecards - Fabric Traces are distributed across each linecard slot - Each Fabric Trace can run at 20Gb/sec OR 40Gb/sec 21
Catalyst 6500- Checking Fabric Utilization 6509E#show platform hardware capacity fabric Switch Fabric Resources Bus utilization: current: 25%, peak was 75% at 19:28:31 UTC Mon Feb 2 2012 Fabric utilization: Ingress Egress Module Chanl Speed rate peak rate peak 1 0 20G 10% 50% @13:49 06Jan12 20% 50% @13:49 06Jan12 1 1 20G 20% 50% @13:49 06Jan12 10% 50% @13:49 06Jan12 2 0 20G 0% 1% @20:30 13Jan12 0% 1% @20:46 06Jan12 2 1 20G 0% 1% @20:47 16Jan12 0% 1% @16:52 06Jan12 3 0 20G 20% 40% @13:49 06Jan12 0% 0% @13:49 06Jan12 6 0 20G 0% 1% @17:44 06Jan12 0% 1% @00:36 08Jan12 8 0 8G 0% 3% @16:33 12Feb12 50% 100% @13:49 06Jan12 22
Catalyst 6500 Multilayer Switch Feature Card MSFC Serves as Control Plane for 6500 Supervisors 720 and 32 have Two CPU s SP and RP SP serves as L2 control plane RP serves as L3 control plane MSFC3 Supervisor 2T has One CPU Single CPU performs L2 and L3 functions CMP on MSFC5 provides CPU, file system, and boot management MSFC5 Local Bootflash holds IOS images Only SP Bootflash holds Native IOS images for Supervisor 720 Config held in NVRAM 23
Catalyst 6500 Supervisor 2T MSFC5: Connectivity Management Processor (CMP) The Connectivity Management Processor (CMP) supports new capabilities that will aid Network Administrators in managing the system: CPU Image Recovery - TFTP boot of the system CPU File Transfer - Image on USB device or TFTP Remote CPU Reset - Hard or Soft reset CPU Console Logging - Record CPU console log for troubleshooting USB Support - USB serial console access 24
Catalyst 6500 Policy Feature Card PFC3 PFC Serves as Data Plane for 6500 Two primary s L2 and L3 TCAM s used for high speed lookup into Forwarding (FIB), ACL (Security and QoS) and Netflow Tables PFC3 48Mpps Maximum Forwarding PFC4 60Mpps Maximum Forwarding PFC4 Common features supported in hardware by PFC3 and PFC4 include: IPv4 - IPv6 - MPLS - Multicast - Policing - Classification - RACL - VACL - PACL - GRE - Tunneling - URPF - Control Plane Policing - and more Features introduced by the PFC4 include: Flexible NetFlow - ACL Dry Run - ACL Hitless Commit - Cisco TrustSec VPLS - Egress NetFlow - IPv6 urpf - Roles Based Access Control 512K Multicast Routes Improved EtherChannel Hash and more 26
Agenda Chassis and Power Supplies Supervisor Engine and Switch Fabric Architectures Module Architectures Layer 2 Forwarding IP Unicast Forwarding NetFlow Access Control Lists Packet Walks 29
Catalyst 6500 Classic Module Architecture Rbus Dbus EoBC Port Linecard Ingress and Egress packet queuing and scheduling is done in the Port All other functions (Lookups, Policing, Replication, etc) are performed on the Supervisor There is no connection to the Switch Fabric Packets destined to fabric-attached modules utilize the Supervisor s switch fabric connection 30
Catalyst 6500 CEF256 Module Architecture Rbus Dbus EoBC 8Gb Fabric Channel to Switch Fabric Fabric Replication Port Port Port Port Dbus Rbus Linecard CEF256 provides connection to Bus and Switch Fabric Ingress and Egress packet queuing and scheduling is done in the Port Can use either Bus or Fabric for data transmission Local replication for multicast and SPAN replication 31
Catalyst 6500 CEF720 Module Architecture 20Gbps Fabric Channel Dbus Rbus EoBC 20Gbps Fabric Channel Fabric and Replication Port Port Centralized Forwarding Card Fabric and Replication Port Port Linecard CEF720 has no local forwarding Uses CFC card to forward Packet header to Supervisor over BUS for forwarding lookup Ingress and Egress packet queuing and scheduling is done in the Port Data sent over fabric channel to destination linecard 33
Catalyst 6500 dcef720 Module Architecture 20Gbps Fabric Channel 20Gbps Fabric Channel EoBC Port Fabric and Replication.. Port Distributed Forwarding Card L2 FWD L3 FWD Port Fabric and Replication.. Port Linecard dcef720 uses DFC3 / DFC4 for local forwarding DFC3 / DFC4 contains same hardware and logic as PFC3 / PFC4 on Supervisor Module has no connection to Dbus or Rbus Ingress and Egress packet queuing and scheduling is done in the Port 34
Catalyst 6500 dcef2t Module Architecture 40Gbps Fabric Channel 40Gbps Fabric Channel EoBC FABRIC INTERFACE FIRE FIRE Distributed Forwarding Card L2 FWD L3 FWD FIRE FIRE Linecard PORT PORT PORT PORT PORT PORT PORT PORT CTS CTS CTS CTS CTS CTS CTS CTS dcef2t uses DFC4 for local forwarding and other operations (ACL, NetFlow, QoS, MPLS, etc) Ingress and Egress packet queuing and scheduling is done in the Port Linecard has no connection to Rbus or Dbus CTS s provide wire-rate encryption / decryption 35
Catalyst 6500 Module Architecture Centralized Forwarding Cards (CFC) The Centralized Forwarding Card (CFC) provides BUS connectivity for the CEF720 linecards The CFC is available only for certain CEF720 modules: WS-X6704-10GE WS-X6724-SFP WS-X6748-SFP WS-X6748-GE-TX The CFC provides the connection to the Dbus and Rbus The CFC is used to communicate with the Supervisor when centralized forwarding is used 36
Catalyst 6500 Module Architecture Distributed Forwarding Card 3 (DFC3) The DFC3 provides local forwarding lookups and feature enforcement (ACL, QoS, MPLS, NetFlow, etc) for the module to incrementally boost overall switch performance - if installed on a CEF720 linecard, it replaces the CFC The DFC3 supports forwarding rates up to 48Mpps The DFC3 stores a local copy of the forwarding table, as well as Security and QoS ACL s that are centrally defined The DFC3 IS field upgradeable and is supported only with Sup720 Three different versions of the DFC3 are supported DFC3A DFC3B/DFC3BXL DFC3C/DFC3CXL 37
Catalyst 6500 Module Architecture Distributed Forwarding Card 4 (DFC4) The DFC4 is an option for CEF720 linecards - it is used to provide local forwarding lookups and feature enforcement (ACL, QoS, MPLS, NetFlow, etc) for the module to incrementally boost overall switch performance - if installed on a CEF720 linecard, it takes the place of the CFC The DFC4 supports forwarding rates up to 60Mpps The DFC4 also stores a local copy of the forwarding tables, as well as Security and QoS ACL s that are centrally defined The DFC4 is located underneath a protective cover that protects the daughtercard from getting damaged when the linecard is inserted or removed from a chassis The DFC4 IS field upgradable Two different versions of the DFC4 are supported DFC4-A / AXL DFC4-E / EXL 38
Catalyst 6500 Module Architecture DFC3/4 Interoperability with PFC3/4 DFC3s work only with PFC3s, and DFC4s work only with PFC4s. When mixing DFCs and PFCs of different capabilities, the lower common denominator is in effect: Example 1 : A PFC3BXL on the Supervisor with a DFC3B on the module will result in the PFC3BXL running in PFC3B mode. Result : The larger FIB and NetFlow tables of the XL will not be used as they will need to be programmed to match the smaller tables sizes of the non-xl. Example 2: A PFC3C on the Supervisor with a DFC3B on the module will result in the PFC3C running in PFC3B mode. Result : The VSS capability of the PFC3C will be disabled when it runs in PFC3B mode since PFC3B mode does not support VSS. Mixing of different PFCs in the same chassis is not supported. When inserting a module with a lower level DFC than the PFC on the Supervisor, the system must be reloaded for the PFC to reprogram itself to the lower mode. 39
Catalyst 6500 Module Architecture Centralized Forwarding Modes of Operation When utilizing Centralized Forwarding, the backplane will operate in one of three modes these modes are determined by the combination of linecards installed in the chassis, from which module the traffic is sourced from and to which module the traffic is destined. Mode Description Illustration FLOW THROUGH Between non fabric modules and between a non fabric and a fabric enabled linecard Throughput 15 Mpps (@ 64 byte frames) Bandwidth 16 Gbps of bandwidth shared throughout Data Bus frame size is variable; min of 4 cycles (64B Data) on the DBus for every frame +1 wait cycle Data H Data H Bus TRUNCATED Between fabric linecards when a non fabric linecard is in the chassis. Throughput 15 Mpps (@ 64 byte frames); independent of frame size for CEF256 and CEF720 Bandwidth 16 G shared for classic; 8 G per CEF256; 20 G/channel CEF720 Data Bus frame size is variable; min of 4 cycles (64 Bytes Data) on the Data Bus for every frame. Data H D H Bus COMPACT When only ALL fabric enabled linecards in a chassis Throughput 30 Mpps (@ any frame size) Bandwidth 8 G CEF256; 20 G/channel CEF720 Data Bus frame size is constant (compact header); 2 cycles (32 B Data) on the DBus for every frame + no wait cycle D H D H D H D H Bus 41
Agenda Chassis and Power Supplies Supervisor Engine and Switch Fabric Architectures Module Architectures Layer 2 Forwarding IP Unicast Forwarding NetFlow Access Control Lists Packet Walks 42
Catalyst 6500 Internals L2 Forwarding Steps Frame received Layer 2 Table Source MAC Lookup Destination MAC Lookup Layer 2 Table Learn Yes New MAC? Router MAC? Yes L3 forwarding Layer 2 Table No No Update entry Layer 2 Table Known MAC? No Yes L2 forwarding L2 flooding
Catalyst 6500 Internals Layer 2 Table Structure The PFC has an integrated CAM Table that supports 4096 rows * X pages = MAC address space MAC Table Table MAC A B C D E F Port 1 2 3 4 5 6 PFC 16, 24, or 32 pages 4096 rows MAC Table PFC3B/BXL = 16 pages (64K entries) PFC3C/CXL = 24 pages (96K entries) PFC4/XL = 32 pages (128K entries) 44
Catalyst 6500 Internals Layer 2 Forwarding Operation Frame 16, 24, or 32 Pages VLAN MAC PFC Hash 0000.2222.7777 20 MAC Table Row HIT!!! 0000.1111.cccc 10 0000.dddd.a112 30 0000.bbbb.ac1c 30 1. Hash result identifies the starting Page and Row in MAC table 2. Lookup key (VLAN + MAC) compared to contents of indexed line on each page, sequentially 3. Destination lookup: Match returns destination interface(s), Miss results in Flood 4. Source lookup: Match updates age of matching entry, Miss installs new entry in table MAC Table 4096 Rows 45
Displaying the Layer 2 Table 6513E.SUP2T.SA.2#show mac address-table Legend: * - primary entry age - seconds since last seen; n/a - not available; S - secure entry; R - router's gateway mac address entry; D - Duplicate mac address entry Displaying entries from active supervisor: vlan mac address type learn age ports ----+----+---------------+-------+-----+----------+----------------------------- * 192 00d0.0053.bc00 dynamic Yes 5 Gi7/3 R 205 0024.c4dc.d740 static No - Router R 20 0024.c4dc.d740 static No - Router * 192 0014.5e31.4220 dynamic Yes 65 Gi7/3 * 60 00d0.2bfc.23f5 dynamic Yes 30 Gi5/14 * 192 00e0.1e5d.e9ff dynamic Yes 30 Gi7/3 46
Catalyst 6500 Internals EtherChannel Combines multiple physical interfaces into ONE logical interface EtherChannel Load Sharing Deterministic PFC3 algorithm supports 8 results (3 bits) PFC4 algorithm supports 256 results (8 bits) Load Sharing is by flow and NOT per packet EtherChannel can be configured for L2 and L3 interfaces 47
EtherChannel Power-of-2 Ports PFC3 Flow Distribution Frame 1 2 3 4 5 6 7 8 EtherChannel Hash 3 bit result E/Chan Bundle Link1 Link2 Link3 Link4 Link5 Link6 Link7 Link8 2 Links 50% 50% -- -- -- -- -- -- 3 Links 37.5% 37.5% 25% -- -- -- -- -- 4 Links 25% 25% 25% 25% -- -- -- -- 5 Links 25% 25% 25% 12.5% 12.5% -- -- -- 6 Links 25% 25% 12.5% 12.5% 12.5% 12.5% -- -- 7 Links 25% 12.5% 12.5% 12.5% 12.5% 12.5% 12.5% -- 8 Links 12.5% 12.5% 12.5% 12.5% 12.5% 12.5% 12.5% 12.5% Even Distribution for Flows is for those cases highlighted in RED 48
EtherChannel Power-of-2 Ports PFC4 Flow Distribution Frame 1 2 3.. 256 EtherChannel Hash 8 bit result E/Chan Bundle Link1 Link2 Link3 Link4 Link5 Link6 Link7 Link8 2 Links 50% 50% -- -- -- -- -- -- 3 Links 33.6% 33.2% 33.2% -- -- -- -- -- 4 Links 25% 25% 25% 25% -- -- -- -- 5 Links 20.4% 19.9% 19.9% 19.9% 19.9% -- -- -- 6 Links 16.8% 16.8% 16.8% 16.8% 16.4% 16.4% -- -- 7 Links 14.5% 14.5% 14.5% 14.5% 14% 14% 14% -- 8 Links 12.5% 12.5% 12.5% 12.5% 12.5% 12.5% 12.5% 12.5% Even Distribution for Flows is for those cases highlighted in RED 49
Agenda Chassis and Power Supplies Supervisor Engine and Switch Fabric Architectures Module Architectures Layer 2 Forwarding IP Unicast Forwarding NetFlow Access Control Lists Packet Walks 52
Catalyst 6500 IP Unicast Forwarding Note This session covers IP Unicast forwarding. There is a dedicated Breakout Session at Cisco Live for IP Multicast Forwarding with the Catalyst 6500: BRKARC-3322 Catalyst 6500 IP Multicast Architecture 53
Catalyst 6500 Interface Management Supervisor 720 Supervisor 2T 4K VLAN POOL VLANs L3 Ports SVI Tunnels CoPP Etc 16K Bridge Domains VLAN 1 4K VLAN 1 4K VLAN 1 4K 128K Logical Interfaces SVI CoPP L3 Ports Tunnels Etc VLANs used for both L2 bridging and L3 routing L3 interfaces internally consume VLANs from the 4K VLAN pool Separate L2 bridging and L3 routing Break the 4K VLAN barrier Allows VLAN reuse on a per port basis Massive scale of L3 interfaces 54
Catalyst 6500 PFC3/DFC3 Lookup Process Netflow TCAM 5 L3 Engine 4 FIB TCAM & SSRAM Netflow Table 7 4 Security ACL TCAM Netflow Statistics Adjacency Statistics 8 4 6 QoS ACL TCAM Adjacency Table 3 8 IP Packet Parse 1 L2 Engine 2 IP Packet Parse 8 L2 MAC Table 55
Catalyst 6500 PFC4/DFC Lookup Process Input Forwarding Engine Lookup Architecturally, the PFC/DFC4 is almost the same as the PFC/DFC3 What changes is the Dual-Cycle Input (IFE) and Output (OFE) Processing Here we perform the Input Forwarding Engine (IFE) pass... IFE process: Packet Header GV IF RP CL1 1.IF: Get Port and Ingress LIF QoS info 2.RP: Src FIB Lookup, Source QoS PO CL2 3.CL1: Ingress ACL TCAM Lookup 4.CL2: Select Ingress Class and Policy 5.NF: Ingress NetFlow lookup RI PL L3 NF 6.L3: Dst FIB Lookup, Dst QoS L2 Engine L3 Engine 7.PL: Apply Ingress Policing and Marking 56
Catalyst 6500 PFC4/DFC Lookup Process Output Forwarding Engine Lookup Architecturally, the PFC/DFC4 is almost the same as the PFC/DFC3 What changes is the Dual-Cycle Input (IFE) and Output (OFE) Processing Here we perform the Output Forwarding Engine (OFE) pass... OFE process: RBUS Result GV IF RP CL1 1.IF: Get Egress LIF QoS info 2.CL1: Egress ACL TCAM lookup 3.CL2: Select Egress Policy and Class PO CL2 4.NF: Select NF Egress Policy and Class 5.PL: Apply Egress Policing and Marking RI PL L3 NF 6.RI: Generate RBUS result L2 Engine L3 Engine 57
Catalyst 6500 IP Unicast Forwarding Layer 3 Forwarding on PFC Routing Protocols receive routing updates from the network... Routing Protocols OSPF, EIGRP, ISIS, BGP, etc Control Plane (RP) Holds routing tables in Routing information Base (RIB) from Static Routes and all running Routing Protocols FIB (on PFC/DFC) FIB & ADJ tables are used by EARL to perform L3 lookups & forwarding Hardware CEF Loads FIB into PFC & distributes to DFC s Hardware Based CEF Process Software CEF Takes RIB and builds a Forwarding Information Base (FIB) containing IP/mask prefixes 1. FIB lookup based on destination prefix (longest-match) 2. FIB Hit returns Adjacency pointer 3. Adjacency contains Rewrite (next-hop) information 4. ACL, QoS & NetFlow lookups occur in parallel, and effect final result 58
Catalyst 6500 IP Unicast Forwarding Layer 3 Forwarding on PFC Located on the PFC are the FIB and Adjacency Table The FIB contains: L3 entries are arranged logically from MOST to LEAST specific (based on /mask) Overall FIB hardware shared by: IPv4 Unicast IPv4 Multicast IPv6 Unicast IPv6 Multicast MPLS The Adjacency Table: L2 Re-Write information and / or pointers for replication Hardware adjacency table also shared among protocols 172.20.45.1 10.1.1.100 MASK (/32) 10.1.3.0 10.1.2.0 MASK (/24) 10.1.0.0 172.16.0.0 MASK (/16) 0.0.0.0 MASK (/0) FIB TCAM IF, MACs, MTU IF, MACs, MTU IF, MACs, MTU IF, MACs, MTU Adjacency Table 59
Catalyst 6500 Internals Layer 3 Forwarding on PFC Assuming a lookup was performed for a packet with a destination of 10.1.5.2 /24, then the following would occur 1 Packet 172.20.45.1 10.1.1.100 2 Key Gen 3 Lookup Key HIT! MASK (/32) 10.1.3.0 10.1.2.0 MASK (/24) 10.1.0.0 172.16.0.0 MASK (/16) 4 Load-Sharing Hash 5 6 IF, MACs, MTU IF, MACs, MTU IF, MACs, MTU IF, MACs, MTU 7 Adjacency Table 0.0.0.0 MASK (/0) FIB TCAM 60
Supervisor FIB TCAM Resources IPv6 and IPv4 multicast require 2 entries MPLS and IPv4 only one XL PFCs = 1M entries Non-XL PFCs = 256K entries By default TCAM is allocated as seen in the table NON-XL PFCs XL PFCs IPv4, MPLS 192k 512k IPv6, Multicast 32k 256k SUP720-3BXL Example 6509E#sh mls cef maximum-routes FIB TCAM maximum routes : ======================= Current :- ------- IPv4 + MPLS - 512k (default) IPv6 + IP Multicast - 256k (default) Changing default (requires Reboot!) 6509E(config)#mls cef maximum-routes? ip number of ip routes ip-multicast number of multicast routes ipv6 number of ipv6 routes mpls number of MPLS labels 62
Displaying IPv4 Forwarding Summary 6509E#show platform hardware capacity forwarding <snip> L3 Forwarding Resources FIB TCAM usage: Total Used %Used 72 bits (IPv4, MPLS, EoM) 196608 28 1% 144 bits (IP mcast, IPv6) 32768 7 1% detail: Protocol Used %Used IPv4 28 1% MPLS 0 0% EoM 0 0% IPv6 1 1% IPv4 mcast 3 1% IPv6 mcast 3 1% <snip> Adjacency usage: Total Used %Used 1048576 171 1% 63
Displaying Hardware IPv4 Prefix Entries 6509E#show platform hardware cef Codes: decap - Decapsulation, + - Push Label Index Prefix Adjacency 68 255.255.255.255/32 receive 75 10.10.1.1/32 receive 76 10.10.1.0/32 receive 77 10.10.1.255/32 receive 78 10.10.1.2/32 Gi1/1, 0030.f272.31fe 3200 224.0.0.0/24 receive 3201 10.10.1.0/24 glean 3202 10.100.0.0/24 Gi1/1, 0030.f272.31fe 3203 10.100.1.0/24 Gi1/1, 0030.f272.31fe 3204 10.100.2.0/24 Gi1/1, 0030.f272.31fe 3205 10.100.3.0/24 Gi1/1, 0030.f272.31fe < > 64
Finding the Longest-Match Prefix Entry 6509E#show platform hardware cef 171.1.1.0 Codes: decap - Decapsulation, + - Push Label Index Prefix Adjacency 6509E#show platform hardware cef lookup 171.1.1.0 Codes: decap - Decapsulation, + - Push Label Index Prefix Adjacency 3531584 171.0.0.0/8 Vl192,00d0.0053.bc00 6500E#show platform hardware cef ipv6 lookup FF00:: Codes: + - Push label Index Prefix Adjacency 512 FF00::/8 glean 65
IPv4 CEF Load Sharing Up to 16* hardware load-sharing paths per prefix Use maximum-paths command in routing protocols to control number of load-sharing paths IPv4 CEF load-sharing is per-ip flow 10.10.0.0/16 via Rtr-A via Rtr-B Per-packet load-balancing not supported Load-sharing based on Source and Destination IP addresses by default Unique ID in PFC3 and PFC4 prevents polarization A B Configuration option supports inclusion of L4 ports in the hash mls ip cef load-sharing full 10.10.0.0/16 Unique ID not included in hash in full mode 66
Load-Sharing Prefixes and Paths 6509E#show platform hardware cef lookup 10.100.20.1 Codes: decap - Decapsulation, + - Push Label Index Prefix Adjacency 3222 10.100.20.0/24 Gi1/1, 0030.f272.31fe Gi1/2, 0008.7ca8.484c Gi2/1, 000e.382d.0b90 Gi2/2, 000d.6550.a8ea 6509E#show platform hardware cef exact-route 10.77.17.8 10.100.20.199 Interface: Gi1/1, Next Hop: 10.10.1.2, Vlan: 1019, Destination Mac: 0030.f272.31fe 6509E#show platform hardware cef exact-route 10.44.91.111 10.100.20.199 Interface: Gi2/2, Next Hop: 10.40.1.2, Vlan: 1018, Destination Mac: 000d.6550.a8ea 67
Agenda Chassis and Power Supplies Supervisor Engine and Switch Fabric Architectures Module Architectures Layer 2 Forwarding IP Unicast Forwarding NetFlow Access Control Lists Packet Walks 68
Catalyst 6500 NetFlow Netflow is a process designed to collect information about traffic flows that pass through the switch - Netflow collection of flow records is a hardware process while the exporting of flow records to an external collector is a control plane process Netflow Netflow Collection Server Data Flow Exported Netflow Record 69
Catalyst 6500 NetFlow PFC3 Flow Masks The Catalyst 6500 supports the following flow masks - these are used to identify which pieces of information in the header will be used as input into generating a key for flow lookups 70
Catalyst 6500 NetFlow TCAM Lookup on PFC3 2 Packet 1 Flow Key Hash Key 3 Hash Function 4 Compare Mask HIT! Mask Key Key Key Key Key Key Key Key Key Key Key 5 Compare NetFlow Table Index 128K/256K entries 6 Result Flow Data Flow Data Flow Data Flow Data Flow Data Flow Data Flow Data Flow Data Flow Data Flow Data Flow Data Flow Data HIT! 7 Statistics 128K/256K rows Netflow TCAM Netflow Table Key 128 entries Alias CAM 71
Catalyst 6500 NetFlow NetFlow Export Process Supervisor Netflow Export Netflow Data EOBC WS-X6748-GE-TX w\dfc4 Netflow Data WS-X6908-10G-2T\2TXL Netflow Data Netflow Collector Direct Export supported with Supervisor 720 and : WS-X6708-10GE-3C/3CXL WS-X6716-10x-3C/3CXL Direct Export supported with Supervisor 2T and : WS-X6716-10x upgraded with DFC4-E / DFC4-EXL WS-X6816-10x-2T/2TXL WS-X6908-10G-2T/2TXL WS-X6904-40G-2T/2TXL 73
Catalyst 6500 NetFlow PFC4 Key Enhancements The PFC4 can do everything the PFC3 can do and adds these new capabilities: Increased Support for NetFlow Entries Up to 1M NetFlow entries (512K for Ingress and 512K for Egress) can now be stored in PFC4XL. Improved NetFlow Hash The hash efficiency is improved to 99%, allowing a greater percentage of the NetFlow table to be utilized. Egress NetFlow Provides support for collecting flow statistics for packets after they have had ingress processing applied to them. Sampled NetFlow in Hardware Allows users to to have NetFlow records created based on a sample of traffic matching the flow. Flexible NetFlow Supports the NetFlow V9 Record Format including new fields for IPV6 and Multicast information. TCP Flags TCP Flags (SYN, FIN, RST, ACK, URGENT, PUSH) are now collected as part of a flow record. CPU Friendly Export Protects the CPU from being overrun by heavy NetFlow Data Export 74
Catalyst 6500 NetFlow PFC4 NetFlow Processing IFE NetFlow Sampling and Lookup IFE NetFlow Statistics Accounting of packets admitted by input processing Incoming Packet Ingress ACL Ingress NetFlow L3 Lookup Ingress QoS IFE Process OFE Process Outgoing Packet Egress QoS Egress NetFlow Egress ACL OFE NetFlow Statistics Accounting of forwarded packet OFE NetFlow Sampling and Lookup 75
Catalyst 6500 NetFlow TCAM Lookup on PFC4 DST IP SRC IP Proto DST Port SRC Port 10.1.2.11 10.1.1.10 0x6 80 33992 1 Flow Key Flow Key Hash Function 2 Compare all pages 4 Data Data Data Data Key Key Key Key Data Data Data Data Key Key Key Key Data Data Data Data Key Key Key Key Data Data Data Data Key Key Key Key Data Data Data Data Key Key Key Key Data Data Data Data Key Key Key Key HIT! Data Data Data Data Key Key Key Key Data Data Data Data Key Key Key Key Data Data Data Data Key Key Key Key Data Data Data Data Key Key Key Key Data Data Data Data Key Key Key Key Index Index Index Index Index Index Index Index Index Index Index Index Index Index Index Index Index Index Index Index Index Index Index Index Index Index Index Index Index Index Index Index Index Index Index Index Index Index Index Index Index Index Index Index Compare Flow Data 6 5 Index to NF Data Table Flow Data Flow Data Flow Data Flow Data Flow Data Flow Data Flow HIT! Data Flow Data Flow Data Flow Data Flow Data 7 Update Stats Statistics Statistics Statistics Statistics Statistics Statistics Statistics Statistics Statistics Statistics Statistics Lookup Key 3 Data Key Indexes row in Lookup Table 512K entries NetFlow Lookup Table NetFlow Data Table NetFlow Statistics Table 76
Catalyst 6500 NetFlow Flexible NetFlow Configuration Key Field Key Field Flow Record Non-Key Field Non-Key Field Multiple Exporters can be associated with a single FNF monitor Flow Export Export Profile Export Profile Key Fields trigger the creation of a new Flow entry every time their value change Non-Key Fields are data that is indexed by the Key Fields. Flow Monitor Flow Monitor Key Fields are defined using the match statement Non-Key-Fields are defined using the collect statement Interfaces Ingress or/and Egress. Ingress or/and Egress Same Flow Monitor can be associated with multiple Interfaces. 77
Catalyst 6500 NetFlow CPU Friendly Export NDE increases export rate until threshold reached CPU Utilization When threshold reached, NDE quickly backs off export rate 70% Yielding NDE threshold 30% Wait 5 seconds and then step up export rate again CPU before NDE begins 80
Catalyst 6500 NetFlow Integration with EEM Example I: Malformed Packets Detection & Reporting Attacker sending malformed pkts with TTL=0 NetFlow cache srcif SrcIPadd DstIf DstIPadd TTL Fa1/0 173.1.1.2 Fa0/0 10.0.277.1 0 Fa1/0 173.1.1.2 Fa0/0 10.0.277.1 10 TTL = 0 triggers an EEM event *MAR 29 2010 12:29:02.604 UTC: %HA_EM-6-LOG: my-ttl-applet: flow record with zero TTL Fa1/0 173.1.1.2 Fa0/0 10.0.277.1 200 syslog message generated based on preconfigured policies Example II : Anomaly Flow Detection and Mitigation Compromised user sending traffic with high rate NetFlow cache srcif SrcIPadd DstIf DstIPadd bytes Fa1/0 173.1.1.2 Fa0/0 10.0.277.1 34346 Fa1/0 173.1.1.2 Fa0/0 10.0.277.1 300 Fa1/0 173.1.1.2 Fa0/0 10.0.277.1 1000 NetFlow ED triggers policies to monitor flow rate. Typically, voice conversations are 64kbps *Feb 18 01:24:30.455: %LINK-5- CHANGED: Interface FastEthernet 1/0, changed state to administratively down interface Fa1/0 is shut down when the flow rate exceeds 1Mbps 81
Displaying NetFlow Utilization 6509E#show platform hardware capacity netflow Netflow resources: Netflow table size: 515032 entries total Netflow table usage: Module/Instance Input flows Output flows 3 10% 10% 7 25% 25% 82
Agenda Chassis and Power Supplies Supervisor Engine and Switch Fabric Architectures Module Architectures Layer 2 Forwarding IP Unicast Forwarding NetFlow Access Control Lists Packet Walks 83
Catalyst 6500 Access Control Lists Hardware Support 1 Create the ACL or traffic classification policy using CLI or Network Management System IP Access-List extended Internet permit ip any host 10.2.2.4 permit ip any host 10.5.2.33 permit ip any host 10.11.0.0 permit ip any host 10.4.0.0 DFC DFC PFC PFC DFC Hardware Support Policy Feature Card (PFC) Distributed Forwarding Card (DFC) Router ACLs Vlan ACLs Port Based ACLs Role Based ACLs 2 Hardware- Assist Features Netflow WCCP Reflexive ACLs Network Address Translation Cisco Trust Sec 3 84
Catalyst 6500 Access Control Lists Three Forms of Security ACLs The PFC3/PFC4 supports three forms of Security ACLs: the RACL, VACL and PACL Router ACL (RACL) VLAN ACL (VACL) Port ACL (PACL) Used to permit or deny the movement of traffic between Layer 3 Subnets Used to permit or deny the movement of traffic between Layer 3 Subnets/VLANs or within a VLAN Used to permit or deny the movement of traffic between Layer 3 Subnets/VLANs or within a VLAN Applied as an input or output policy to a Layer 3 interface Applied as a policy to a VLAN - is inherently applied to both inbound and outbound traffic Applied as a policy to a Layer 2 Switch port interface - is applied for inbound traffic only 85
Catalyst 6500 Access Control Lists ACL Order of Processing Should a RACL, VACL and PACL all be configured at the same time, there is a distinct order in which each form of ACL is processed Input RACL Output RACL VACL VACL Input PACL Note that no Output PACL exists Destination Source 86
Catalyst 6500 Access Control Lists PFC3 TCAM Population Protocol Dest IP Dest Port Source IP Source Port 00000000 FFFFFFFF 00 0000 0000 xxxxxxxx 10.1.2.100 xx xxxx xxxx xxxxxxxx 10.1.68.101 xx xxxx xxxx xxxxxxxx 10.33.2.25 xx xxxx xxxx 1 2 3 4 5 Permit Deny Deny 1= Compare 0= Mask 6 7 ip access-list extended example permit ip any host 10.1.2.100 deny ip any host 10.1.68.101 deny ip any host 10.33.2.25 permit tcp any any eq 22 deny tcp any any eq 23 deny udp any any eq 514 00000000 00000000 FF 0000 FFFF xxxxxxxx xxxxxxxx 06 xxxx 0016 xxxxxxxx xxxxxxxx 06 xxxx 0017 xxxxxxxx xxxxxxxx 11 xxxx 0202 xxxxxxxx xxxxxxxx 06 xxxx 0080 xxxxxxxx xxxxxxxx 11 xxxx 00A1 8 1 2 3 4 5 6 Permit Deny Deny Permit Permit permit tcp any any eq 80 permit udp any any eq 161 7 8 Masks Values 87
Catalyst 6500 Access Control Lists PFC3 TCAM Lookup Generate Lookup Key ip access-list extended example permit ip any host 10.1.2.100 deny ip any host 10.1.68.101 deny ip any host 10.33.2.25 permit tcp any any eq 22 deny tcp any any eq 23 deny udp any any eq 514 permit tcp any any eq 80 permit udp any any eq 161 1 Packet 10.1.1.10 xxxxxxxx 10.1.2.11 xxxxxxxx 06 xx xxxx 84C8 0050 xxxx 0050 Lookup Key 2 SIP=10.1.1.10 DIP=10.1.2.11 Protocol=TCP (6) SPORT=33992 DPORT=80 Entries matching only destination IP 00000000 FFFFFFFF 00 0000 0000 00000000 00000000 FF 0000 FFFF HIT! Entries matching only protocol and destination port Masks Compare xxxxxxxx 10.1.2.100 xx xxxx xxxx xxxxxxxx 10.1.68.101 xx xxxx xxxx xxxxxxxx 10.33.2.25 xx xxxx xxxx xxxxxxxx xxxxxxxx 06 xxxx 0016 xxxxxxxx xxxxxxxx 06 xxxx 0017 xxxxxxxx xxxxxxxx 11 xxxx 0202 xxxxxxxx xxxxxxxx 06 xxxx 0050 xxxxxxxx xxxxxxxx 11 xxxx 00A1 Values 3 1 2 3 4 5 6 7 8 1 2 3 4 4 5 6 7 8 Permit Result 88
Catalyst 6500 Access Control Lists PFC4 Mask Utilization PFC3 ACL TCAM MASK 0.0.0.255 MASK 0.0.255.255 10.1.1.0 permit 10.2.1.0 permit - - - - - - - - - - - - 10.3.0.0 permit - - - - - - - - - - - - - - Implements 8:1 Mask to Entry ratio Total 4K Masks, 32K Entries Mask resource is limited permit ip 10.1.1.0 0.0.0.255 any permit ip 10.2.1.0 0.0.0.255 any permit ip 10.3.0.0 0.0.255.255 any Mask 0.0.0.255 Mask 0.0.0.255 Mask 0.0.255.255 - - - - - - - - 10.1.1.0 permit 10.2.1.0 permit 10.3.0.0 permit - - - - - - - - - - - - - - - - Implements 1:1 Mask to Entry ratio Total 256K Masks, 256K Entries PFC4 ACL TCAM Mask resource is no longer a limited resource 90
Catalyst 6500 Access Control Lists PFC4 Lookup Example BANK 0 QoS TCAM A BANK 1 VACL TCAM B BANK 2 BANK 3 SGT RACL Forwarding Engine (PFC4 or DFC4) 3 TCAM Controller ACE Counters (L2 ) 2 Packet Header Information 1 2 X Lookup Keys ACL Labels ACL LOUs Classification Module 1 4 X Results 4 4 X Result Data 5 6 7 Classification Module 2 Final Result to Netflow 8 91
Catalyst 6500 Access Control Lists PFC4 ACL Dry Run Feature Make sure the ACL will fit in the TCAM before you apply the ACL - ACLs that do not fit can cause software forwarding and possible high CPU utilization Special configuration session SUP2T-E#show configuration session test status ==================================== Status of last config validation: Timestamp: 2010-02-20@17:27:06 ====================================== SLOT = [1] Result = Configuration will fit in TCAM - Create and edit ACls - Verifies if the changes will fit within the hardware resources The actual changes are not programmed into the hardware during the configuration session Configuration changes can be verified step by step 93
Catalyst 6500 Access Control Lists PFC4 ACL Hitless Update Allows updates to an ACL without interrupting traffic Multiple features updated at once IPv4 IPv6 MAC IPv4, IPv6, MAC RACL, VACL, PBR ACL Updates Global configuration option (default is on) Feature does consume double the number of TCAM entries 95
Catalyst 6500 Access Control Lists PFC4 ACL Hitless Update Each ACL feature is initially programmed into two different spaces into the TCAM Primary space (Label -1) Shadow space (label-2) While an ACL is being updated the PFC4 will use a temporary label that points to the shadow TCAM space Once the ACL changes have been completed the then PFC4 will then use the original label again BANK 0 QoS-1 QoS-2 TCAM A BANK 1 VACL-1 VACL-2 BANK 2 SGT-1 SGT-2 TCAM B 2 X Lookup Keys 4 X Results ACL Labels 1, 2 TCAM Controller ACL LOUs BANK 3 RACL-1 RACL-2 Classification Module 1 96
Agenda Chassis and Power Supplies Supervisor Engine and Switch Fabric Architectures Module Architectures Layer 2 Forwarding IP Unicast Forwarding NetFlow Access Control Lists Packet Walks 97
Centralized Forwarding: Classic to Classic 1 Port A P R P R P R P R P Port B Port A Port B 4 P Slot 1 Classic Slot 2 Classic Dbus Rbus Michael Engineering Switch Fabric Amanda Marketing R 3 H Fabric / Bus Interface & Replication R P 2 R Layer 2 Engine Layer 3 Engine PFC4 P H = Packet = Header Supervisor Engine 2T R = Result
Centralized Forwarding: Classic to CEF720 1 Port A Slot 1 Classic P R P Port B R P Port A Slot 2 CEF720 Port B CFC 5 P FIRE A FIRE B Dbus Michael Engineering Switch Fabric 4 Rbus Amanda Marketing Question : How will the packet get to the CEF720 Module? Bus or Switch Fabric? ADD QUESTION IN SAN DIEGO. Make it appear between Steps 3 & 4 R 3 Fabric / Bus Interface & Replication Supervisor Engine 2T H R P 2 R Layer 2 Engine Layer 3 Engine PFC4 P H R = Packet = Header = Result
Centralized Forwarding: CEF720 to Classic 9 Port A Slot 1 Classic R P Port B R P Port A Slot 2 CEF720 Port B 1 CFC R 2 FIRE A H FIRE B P 7 3 Dbus Michael Engineering Switch Fabric 6 Rbus Amanda Marketing 8 R 5 H H Fabric / Bus Interface & Replication R P 4 R 5 Layer 2 Engine Layer 3 Engine PFC4 P H = Packet = Header Supervisor Engine 2T R = Result
Centralized Forwarding: CEF720 to CEF720 1 Port A Slot 1 CEF720 Port B Port A Slot 2 CEF720 Port B 2 R CFC CFC 7 P FIRE A H FIRE B FIRE A FIRE B 6 3 Dbus Rbus Michael Engineering Switch Fabric Amanda Marketing R 5 H Fabric / Bus Interface & Replication H 5 4 R Layer 2 Engine Layer 3 Engine PFC4 P H = Packet = Header Supervisor Engine 2T R = Result
Distributed Forwarding: CEF720/DFC4 to CEF720/DFC4 1 Slot 1 CEF720/DFC4 Port A Port B 3 Slot 2 CEF720/DFC4 Port A Port B P 2 FIRE A DFC4 H R 4 L 2 L 3 FIRE B FIRE A DFC4 L 2 L 3 FIRE B 6 5 Dbus Rbus Michael Engineering Switch Fabric Amanda Marketing PFC4 Fabric / Bus Interface & Replication Layer 2 Engine Layer 3 Engine P H = Packet = Header Supervisor Engine 2T R = Result
Summary The Catalyst 6500 architecture provides a robust infrastructure upon which the system can provide hardware-based forwarding at high speeds L2 and L3 switching are done via the same hardware forwarding process, so there is no difference in performance between the two Enabling features such as Netflow, QoS and ACLs can be done without impact to forwarding performance as these features are processed in hardware in parallel to the L2 and L3 lookup processes The Catalyst 6500 architecture is designed so that unicast and multicast can coexist within the same infrastructure, providing a versatile platform for the networks of today and tomorrow 108
Conclusion You should now have a thorough understanding of the Catalyst 6500 switching architecture, packet flow, and key forwarding engine functions Any Questions? 109
Complete Your Online Session Evaluation Give us your feedback and you could win fabulous prizes. Winners announced daily. Receive 20 Passport points for each session evaluation you complete. Complete your session evaluation online now (open a browser through our wireless network to access our portal) or visit one of the Internet stations throughout the Convention Center. Don t forget to activate your Cisco Live Virtual account for access to all session material, communities, and on-demand and live activities throughout the year. Activate your account at the Cisco booth in the World of Solutions or visit www.ciscolive.com. 110
Final Thoughts Get hands-on experience with the Walk-in Labs located in World of Solutions, booth 1042 Come see demos of many key solutions and products in the main Cisco booth 2924 Visit www.ciscolive365.com after the event for updated PDFs, ondemand session videos, networking, and more! Follow Cisco Live! using social media: Facebook: https://www.facebook.com/ciscoliveus Twitter: https://twitter.com/#!/ciscolive LinkedIn Group: http://linkd.in/ciscoli 111
Presentation_ID