Catalyst 6500 Architecture

Transcription

1

2 Catalyst 6500 Architecture 2

3 Session Goal To provide you with a thorough understanding of the Catalyst 6500 switching architecture, packet flow, forwarding engine functions, and key feature operations. 3

4 Agenda Chassis and Power Supplies Supervisor Engine and Switch Fabric Architectures Module Architectures Layer 2 Forwarding IP Unicast Forwarding NetFlow Access Control Lists Packet Walks 4

5 Catalyst 6500 E-Chassis Family 6509-V-E 6513-E 6509-E 6506-E 6504-E 6503-E 7 Chassis Members From 3 Slot to 13 Slot 5

6 Catalyst 6500 E-Series Chassis Inside the Chassis FABRIC BUS Linecard Slots Supervisor 32/720/2T Slots Linecard Slots 6

7 Catalyst 6500 Switch Backplanes Classic (32Gb) BUS Backplane 720Gb / 2Tb Crossbar Backplane DBUS RBUS EOBC CROSSBAR Linecard Linecard Linecard Linecard Linecard Data Bus (DBUS) allows L/C to forward data to Supervisor for forwarding decision Results Bus (RBUS) returns forwarding result from Supervisor back to L/C Ethernet Out of Band Channel (EOBC) provide out of band management between Supervisor and LC Crossbar is a matrix of N channels to provide a data path between linecards Sup720 supports 18 channels at 8G/20G per channel (speed autodetected) Sup2T supports 26 channels at 20G/40G per channel (speed autodetected) 7

8 Catalyst 6500 Linecard Slot Support Slot 6503/ 6503-E 6504-E 6506/ 6506-E 6509/ 6509-E NEBS-A 6509-V-E E 1 Dual Dual Dual Dual Dual Dual Single Dual 2 Dual Dual Dual Dual Dual Dual Single Dual 3 Dual Dual Dual Dual Dual Dual Single Dual 4 - Dual Dual Dual Dual Dual Single Dual Dual Dual Dual Dual Single Dual In order to take advantage of the dual fabric channels in slots 1 8 of the 6513-E chassis, the Supervisor 2T is required Dual Dual Dual Dual Single Dual Dual Dual Dual Single Dual Dual Dual Dual Single Dual Dual Dual Dual Dual Dual Dual Dual With any version of the Supervisor 720, the 6513-E fabric channel distribution Is the same as the Dual Dual Dual Dual Dual Dual 8

9 Power Supply Redundancy The Catalyst 6500 can utilize two power supplies to work in either Combined or Redundant Mode Redundant Mode Catalyst 6500 Combined Mode Catalyst % 50% 83% 83% Use the Cisco Power Calculator on cisco.com to determine which supplies and which mode of operation is needed for your system. Power Supply 1 Power Supply 2 Each power supply operates at ~50% capacity Neither supply operates at >60% or <40% capacity If one fails, the second supply can power the system on its own This is the default and recommended configuration for the power supplies Power Supply 1 Power Supply 2 Each power supply provides up to 83% of its capacity The total system power available is 167% of the capacity of a single supply If one fails, the second supply may not be able to power the system on its own - this could result in devices or linecards being shut down This is not the recommended mode for production 11

10 Agenda Chassis and Power Supplies Supervisor Engine and Switch Fabric Architectures Module Architectures Layer 2 Forwarding IP Unicast Forwarding NetFlow Access Control Lists Packet Walks 12

11 Catalyst 6500 Supervisors Supervisor 720: Some Facts Supervisor 720 Quick Facts Integrated 720Gbps Switch Fabric Supervisor 720 3A / 3B / 3BXL Supervisor G 3C / 3CXL Integrated Policy Feature Card 3 (PFC3) supporting hardware acceleration for select features Integrated Multilayer Switch Feature Card 3 (MSFC3) supporting two CPUs for Layer 2 and Layer 3 functionality IPv6 unicast and multicast forwarding support in hardware Virtual Switching System (VSS) support with Sup720-10G models All uplinks can be active in systems with redundant Supervisors (more information Cisco in Public the notes) 13

12 Supervisor 720 3A / 3B / 3BXL Classic BUS EOBC Rbus Dbus Crossbar Fabric Channels L2 CAM Switch Fabric Layer 2 FWD Engine FIB TCAM FIB Table QOS ACL RP Flash SP Flash RP DRAM SP DRAM RP SP MET Fabric / Replication Port Port Layer 3 FWD Engine Security ACL Counters Netflow TCAM Netflow Table Netflow Stats Adjacency Adj Stats MSFC3 1G 1G 1G Policy Feature Card (PFC3) 14

13 Supervisor G 3C / 3CXL Classic BUS Crossbar Fabric Channels EOBC Rbus Dbus Fabric RP SP Switch Fabric MET 20Gbps Fabric / Replication L2 CAM Layer 2/3 FWD Engine FIB TCAM FIB Table QOS ACL Security ACL Counters Netflow TCAM Netflow Table Netflow Stats RP Flash RP DRAM SP Flash SP DRAM MSFC3 10G Port Quad Port PHY 1G 1G Port 1G 10G Adjacency Adj Stats Policy Feature Card (PFC3) 15

14 Catalyst 6500 Supervisors Supervisor 2T: Some Facts Supervisor 720 Quick Facts Integrated 2-Tbps Switch Fabric Integrated Policy Feature Card 4 (PFC4) supporting hardware acceleration for select features Supervisor 2T PFC4 / PFC4XL Integrated Multilayer Switch Feature Card 5 (MSFC5) supporting a single CPU for L2 and L3 functionality Connectivity Management Processor (CMP) for improved management capability One external compact flash slot (power controlled by IOS) All uplinks can be active in systems with redundant Supervisors (more information Cisco in Public the notes) 16

15 Supervisor 2T PFC4 / PFC4XL Classic BUS Crossbar Fabric Channels EOBC Rbus Dbus Fabric DRAM CPU Flash MSFC5 Switch Fabric MET 10G Port Fabric / Replication 1G 20Gbps Quad Port PHY 1G Port 1G 10G L2 NetFlow CAM LIF Table PFC4 FIB TCAM FIB Table CL2 TCAM QOS ACL Security ACL Layer 2/3 L3/4 Forwarding FWD Engine Engine LIF DB LIF Stats CL1 TCAM Counters Netflow TCAM Netflow Table Netflow Stats Adjacency Adj Stats L2 Forwarding Engine L2 CAM (128K) Policy Feature Card (PFC4) FIB ADJ RPF Table ACE Counter 17

16 Supervisor Chassis Requirements Chassis Supervisor 720s All E-Series All non-e Series E-Fans for E-Series Supervisor 2Ts Only E-Series E-Fans for E-Series E-Fan cannot be used in non-e Series Chassis Fan2 cannot be used in E-Series Fan Trays Power Supplies Fan2 for non-e Series 2500W AC / DC or greater With Supervisor 2T and 6513-E, only Supervisors are allowed in the Supervisor Slots 3-Slot : 1 and 2 Supervisor Slots 4-slot : 1 and 2 6-slot : 5 and 6 9-slot : 5 and 6 With Supervisor 720 and 6513-E, the fabric channel distribution is the same as with Supervisor 720 and slot : 7 and 8 18

17 Catalyst 6500 Supervisors Switch Fabric The Supervisor 720 and Supervisor 2T support a Switch Fabric which offers each connected linecard a set of discrete communication paths into the switch backplane Linecard Slot #9 Linecard Slot #8 Linecard Slot #1 Linecard Slot #2 Linecard Slot #7 Linecard Slot #6 Data Flows Linecard Slot #3 Linecard Slot #4 Supervisor Slot #5 19

18 Catalyst 6500 Supervisor 720 The 720Gbps Switch Fabric Switch Fabric - Integrated 720Gbps Switch Fabric - Provides backplane interconnects between linecards - Fabric Traces are distributed across each linecard slot - Each Fabric Trace can run at 8Gb/sec OR 20Gb/sec 20

19 Catalyst 6500 Supervisor 2T The 2Tbps Switch Fabric - Integrated 2Tbps Switch Fabric Switch Fabric - 26 Channels to support the 6513-E - Provides backplane interconnects between linecards - Fabric Traces are distributed across each linecard slot - Each Fabric Trace can run at 20Gb/sec OR 40Gb/sec 21

20 Catalyst Checking Fabric Utilization 6509E#show platform hardware capacity fabric Switch Fabric Resources Bus utilization: current: 25%, peak was 75% at 19:28:31 UTC Mon Feb Fabric utilization: Ingress Egress Module Chanl Speed rate peak rate peak G 10% 06Jan12 20% 06Jan G 20% 06Jan12 10% 06Jan G 0% 13Jan12 0% 06Jan G 0% 16Jan12 0% 06Jan G 20% 06Jan12 0% 06Jan G 0% 06Jan12 0% 08Jan G 0% 12Feb12 50% 06Jan12 22

21 Catalyst 6500 Multilayer Switch Feature Card MSFC Serves as Control Plane for 6500 Supervisors 720 and 32 have Two CPU s SP and RP SP serves as L2 control plane RP serves as L3 control plane MSFC3 Supervisor 2T has One CPU Single CPU performs L2 and L3 functions CMP on MSFC5 provides CPU, file system, and boot management MSFC5 Local Bootflash holds IOS images Only SP Bootflash holds Native IOS images for Supervisor 720 Config held in NVRAM 23

22 Catalyst 6500 Supervisor 2T MSFC5: Connectivity Management Processor (CMP) The Connectivity Management Processor (CMP) supports new capabilities that will aid Network Administrators in managing the system: CPU Image Recovery - TFTP boot of the system CPU File Transfer - Image on USB device or TFTP Remote CPU Reset - Hard or Soft reset CPU Console Logging - Record CPU console log for troubleshooting USB Support - USB serial console access 24

23 Catalyst 6500 Policy Feature Card PFC3 PFC Serves as Data Plane for 6500 Two primary s L2 and L3 TCAM s used for high speed lookup into Forwarding (FIB), ACL (Security and QoS) and Netflow Tables PFC3 48Mpps Maximum Forwarding PFC4 60Mpps Maximum Forwarding PFC4 Common features supported in hardware by PFC3 and PFC4 include: IPv4 - IPv6 - MPLS - Multicast - Policing - Classification - RACL - VACL - PACL - GRE - Tunneling - URPF - Control Plane Policing - and more Features introduced by the PFC4 include: Flexible NetFlow - ACL Dry Run - ACL Hitless Commit - Cisco TrustSec VPLS - Egress NetFlow - IPv6 urpf - Roles Based Access Control 512K Multicast Routes Improved EtherChannel Hash and more 26

24 Agenda Chassis and Power Supplies Supervisor Engine and Switch Fabric Architectures Module Architectures Layer 2 Forwarding IP Unicast Forwarding NetFlow Access Control Lists Packet Walks 29

25 Catalyst 6500 Classic Module Architecture Rbus Dbus EoBC Port Linecard Ingress and Egress packet queuing and scheduling is done in the Port All other functions (Lookups, Policing, Replication, etc) are performed on the Supervisor There is no connection to the Switch Fabric Packets destined to fabric-attached modules utilize the Supervisor s switch fabric connection 30

26 Catalyst 6500 CEF256 Module Architecture Rbus Dbus EoBC 8Gb Fabric Channel to Switch Fabric Fabric Replication Port Port Port Port Dbus Rbus Linecard CEF256 provides connection to Bus and Switch Fabric Ingress and Egress packet queuing and scheduling is done in the Port Can use either Bus or Fabric for data transmission Local replication for multicast and SPAN replication 31

27 Catalyst 6500 CEF720 Module Architecture 20Gbps Fabric Channel Dbus Rbus EoBC 20Gbps Fabric Channel Fabric and Replication Port Port Centralized Forwarding Card Fabric and Replication Port Port Linecard CEF720 has no local forwarding Uses CFC card to forward Packet header to Supervisor over BUS for forwarding lookup Ingress and Egress packet queuing and scheduling is done in the Port Data sent over fabric channel to destination linecard 33

28 Catalyst 6500 dcef720 Module Architecture 20Gbps Fabric Channel 20Gbps Fabric Channel EoBC Port Fabric and Replication.. Port Distributed Forwarding Card L2 FWD L3 FWD Port Fabric and Replication.. Port Linecard dcef720 uses DFC3 / DFC4 for local forwarding DFC3 / DFC4 contains same hardware and logic as PFC3 / PFC4 on Supervisor Module has no connection to Dbus or Rbus Ingress and Egress packet queuing and scheduling is done in the Port 34

29 Catalyst 6500 dcef2t Module Architecture 40Gbps Fabric Channel 40Gbps Fabric Channel EoBC FABRIC INTERFACE FIRE FIRE Distributed Forwarding Card L2 FWD L3 FWD FIRE FIRE Linecard PORT PORT PORT PORT PORT PORT PORT PORT CTS CTS CTS CTS CTS CTS CTS CTS dcef2t uses DFC4 for local forwarding and other operations (ACL, NetFlow, QoS, MPLS, etc) Ingress and Egress packet queuing and scheduling is done in the Port Linecard has no connection to Rbus or Dbus CTS s provide wire-rate encryption / decryption 35

30 Catalyst 6500 Module Architecture Centralized Forwarding Cards (CFC) The Centralized Forwarding Card (CFC) provides BUS connectivity for the CEF720 linecards The CFC is available only for certain CEF720 modules: WS-X GE WS-X6724-SFP WS-X6748-SFP WS-X6748-GE-TX The CFC provides the connection to the Dbus and Rbus The CFC is used to communicate with the Supervisor when centralized forwarding is used 36

31 Catalyst 6500 Module Architecture Distributed Forwarding Card 3 (DFC3) The DFC3 provides local forwarding lookups and feature enforcement (ACL, QoS, MPLS, NetFlow, etc) for the module to incrementally boost overall switch performance - if installed on a CEF720 linecard, it replaces the CFC The DFC3 supports forwarding rates up to 48Mpps The DFC3 stores a local copy of the forwarding table, as well as Security and QoS ACL s that are centrally defined The DFC3 IS field upgradeable and is supported only with Sup720 Three different versions of the DFC3 are supported DFC3A DFC3B/DFC3BXL DFC3C/DFC3CXL 37

32 Catalyst 6500 Module Architecture Distributed Forwarding Card 4 (DFC4) The DFC4 is an option for CEF720 linecards - it is used to provide local forwarding lookups and feature enforcement (ACL, QoS, MPLS, NetFlow, etc) for the module to incrementally boost overall switch performance - if installed on a CEF720 linecard, it takes the place of the CFC The DFC4 supports forwarding rates up to 60Mpps The DFC4 also stores a local copy of the forwarding tables, as well as Security and QoS ACL s that are centrally defined The DFC4 is located underneath a protective cover that protects the daughtercard from getting damaged when the linecard is inserted or removed from a chassis The DFC4 IS field upgradable Two different versions of the DFC4 are supported DFC4-A / AXL DFC4-E / EXL 38

33 Catalyst 6500 Module Architecture DFC3/4 Interoperability with PFC3/4 DFC3s work only with PFC3s, and DFC4s work only with PFC4s. When mixing DFCs and PFCs of different capabilities, the lower common denominator is in effect: Example 1 : A PFC3BXL on the Supervisor with a DFC3B on the module will result in the PFC3BXL running in PFC3B mode. Result : The larger FIB and NetFlow tables of the XL will not be used as they will need to be programmed to match the smaller tables sizes of the non-xl. Example 2: A PFC3C on the Supervisor with a DFC3B on the module will result in the PFC3C running in PFC3B mode. Result : The VSS capability of the PFC3C will be disabled when it runs in PFC3B mode since PFC3B mode does not support VSS. Mixing of different PFCs in the same chassis is not supported. When inserting a module with a lower level DFC than the PFC on the Supervisor, the system must be reloaded for the PFC to reprogram itself to the lower mode. 39

34 Catalyst 6500 Module Architecture Centralized Forwarding Modes of Operation When utilizing Centralized Forwarding, the backplane will operate in one of three modes these modes are determined by the combination of linecards installed in the chassis, from which module the traffic is sourced from and to which module the traffic is destined. Mode Description Illustration FLOW THROUGH Between non fabric modules and between a non fabric and a fabric enabled linecard Throughput 15 Mpps (@ 64 byte frames) Bandwidth 16 Gbps of bandwidth shared throughout Data Bus frame size is variable; min of 4 cycles (64B Data) on the DBus for every frame +1 wait cycle Data H Data H Bus TRUNCATED Between fabric linecards when a non fabric linecard is in the chassis. Throughput 15 Mpps (@ 64 byte frames); independent of frame size for CEF256 and CEF720 Bandwidth 16 G shared for classic; 8 G per CEF256; 20 G/channel CEF720 Data Bus frame size is variable; min of 4 cycles (64 Bytes Data) on the Data Bus for every frame. Data H D H Bus COMPACT When only ALL fabric enabled linecards in a chassis Throughput 30 Mpps (@ any frame size) Bandwidth 8 G CEF256; 20 G/channel CEF720 Data Bus frame size is constant (compact header); 2 cycles (32 B Data) on the DBus for every frame + no wait cycle D H D H D H D H Bus 41

35 Agenda Chassis and Power Supplies Supervisor Engine and Switch Fabric Architectures Module Architectures Layer 2 Forwarding IP Unicast Forwarding NetFlow Access Control Lists Packet Walks 42

36 Catalyst 6500 Internals L2 Forwarding Steps Frame received Layer 2 Table Source MAC Lookup Destination MAC Lookup Layer 2 Table Learn Yes New MAC? Router MAC? Yes L3 forwarding Layer 2 Table No No Update entry Layer 2 Table Known MAC? No Yes L2 forwarding L2 flooding

37 Catalyst 6500 Internals Layer 2 Table Structure The PFC has an integrated CAM Table that supports 4096 rows * X pages = MAC address space MAC Table Table MAC A B C D E F Port PFC 16, 24, or 32 pages 4096 rows MAC Table PFC3B/BXL = 16 pages (64K entries) PFC3C/CXL = 24 pages (96K entries) PFC4/XL = 32 pages (128K entries) 44

38 Catalyst 6500 Internals Layer 2 Forwarding Operation Frame 16, 24, or 32 Pages VLAN MAC PFC Hash MAC Table Row HIT!!! cccc dddd.a bbbb.ac1c Hash result identifies the starting Page and Row in MAC table 2. Lookup key (VLAN + MAC) compared to contents of indexed line on each page, sequentially 3. Destination lookup: Match returns destination interface(s), Miss results in Flood 4. Source lookup: Match updates age of matching entry, Miss installs new entry in table MAC Table 4096 Rows 45

39 Displaying the Layer 2 Table 6513E.SUP2T.SA.2#show mac address-table Legend: * - primary entry age - seconds since last seen; n/a - not available; S - secure entry; R - router's gateway mac address entry; D - Duplicate mac address entry Displaying entries from active supervisor: vlan mac address type learn age ports * d bc00 dynamic Yes 5 Gi7/3 R c4dc.d740 static No - Router R c4dc.d740 static No - Router * e dynamic Yes 65 Gi7/3 * 60 00d0.2bfc.23f5 dynamic Yes 30 Gi5/14 * e0.1e5d.e9ff dynamic Yes 30 Gi7/3 46

40 Catalyst 6500 Internals EtherChannel Combines multiple physical interfaces into ONE logical interface EtherChannel Load Sharing Deterministic PFC3 algorithm supports 8 results (3 bits) PFC4 algorithm supports 256 results (8 bits) Load Sharing is by flow and NOT per packet EtherChannel can be configured for L2 and L3 interfaces 47

41 EtherChannel Power-of-2 Ports PFC3 Flow Distribution Frame EtherChannel Hash 3 bit result E/Chan Bundle Link1 Link2 Link3 Link4 Link5 Link6 Link7 Link8 2 Links 50% 50% Links 37.5% 37.5% 25% Links 25% 25% 25% 25% Links 25% 25% 25% 12.5% 12.5% Links 25% 25% 12.5% 12.5% 12.5% 12.5% Links 25% 12.5% 12.5% 12.5% 12.5% 12.5% 12.5% -- 8 Links 12.5% 12.5% 12.5% 12.5% 12.5% 12.5% 12.5% 12.5% Even Distribution for Flows is for those cases highlighted in RED 48

42 EtherChannel Power-of-2 Ports PFC4 Flow Distribution Frame EtherChannel Hash 8 bit result E/Chan Bundle Link1 Link2 Link3 Link4 Link5 Link6 Link7 Link8 2 Links 50% 50% Links 33.6% 33.2% 33.2% Links 25% 25% 25% 25% Links 20.4% 19.9% 19.9% 19.9% 19.9% Links 16.8% 16.8% 16.8% 16.8% 16.4% 16.4% Links 14.5% 14.5% 14.5% 14.5% 14% 14% 14% -- 8 Links 12.5% 12.5% 12.5% 12.5% 12.5% 12.5% 12.5% 12.5% Even Distribution for Flows is for those cases highlighted in RED 49

43 Agenda Chassis and Power Supplies Supervisor Engine and Switch Fabric Architectures Module Architectures Layer 2 Forwarding IP Unicast Forwarding NetFlow Access Control Lists Packet Walks 52

44 Catalyst 6500 IP Unicast Forwarding Note This session covers IP Unicast forwarding. There is a dedicated Breakout Session at Cisco Live for IP Multicast Forwarding with the Catalyst 6500: BRKARC-3322 Catalyst 6500 IP Multicast Architecture 53

45 Catalyst 6500 Interface Management Supervisor 720 Supervisor 2T 4K VLAN POOL VLANs L3 Ports SVI Tunnels CoPP Etc 16K Bridge Domains VLAN 1 4K VLAN 1 4K VLAN 1 4K 128K Logical Interfaces SVI CoPP L3 Ports Tunnels Etc VLANs used for both L2 bridging and L3 routing L3 interfaces internally consume VLANs from the 4K VLAN pool Separate L2 bridging and L3 routing Break the 4K VLAN barrier Allows VLAN reuse on a per port basis Massive scale of L3 interfaces 54

46 Catalyst 6500 PFC3/DFC3 Lookup Process Netflow TCAM 5 L3 Engine 4 FIB TCAM & SSRAM Netflow Table 7 4 Security ACL TCAM Netflow Statistics Adjacency Statistics QoS ACL TCAM Adjacency Table 3 8 IP Packet Parse 1 L2 Engine 2 IP Packet Parse 8 L2 MAC Table 55

47 Catalyst 6500 PFC4/DFC Lookup Process Input Forwarding Engine Lookup Architecturally, the PFC/DFC4 is almost the same as the PFC/DFC3 What changes is the Dual-Cycle Input (IFE) and Output (OFE) Processing Here we perform the Input Forwarding Engine (IFE) pass... IFE process: Packet Header GV IF RP CL1 1.IF: Get Port and Ingress LIF QoS info 2.RP: Src FIB Lookup, Source QoS PO CL2 3.CL1: Ingress ACL TCAM Lookup 4.CL2: Select Ingress Class and Policy 5.NF: Ingress NetFlow lookup RI PL L3 NF 6.L3: Dst FIB Lookup, Dst QoS L2 Engine L3 Engine 7.PL: Apply Ingress Policing and Marking 56

48 Catalyst 6500 PFC4/DFC Lookup Process Output Forwarding Engine Lookup Architecturally, the PFC/DFC4 is almost the same as the PFC/DFC3 What changes is the Dual-Cycle Input (IFE) and Output (OFE) Processing Here we perform the Output Forwarding Engine (OFE) pass... OFE process: RBUS Result GV IF RP CL1 1.IF: Get Egress LIF QoS info 2.CL1: Egress ACL TCAM lookup 3.CL2: Select Egress Policy and Class PO CL2 4.NF: Select NF Egress Policy and Class 5.PL: Apply Egress Policing and Marking RI PL L3 NF 6.RI: Generate RBUS result L2 Engine L3 Engine 57

49 Catalyst 6500 IP Unicast Forwarding Layer 3 Forwarding on PFC Routing Protocols receive routing updates from the network... Routing Protocols OSPF, EIGRP, ISIS, BGP, etc Control Plane (RP) Holds routing tables in Routing information Base (RIB) from Static Routes and all running Routing Protocols FIB (on PFC/DFC) FIB & ADJ tables are used by EARL to perform L3 lookups & forwarding Hardware CEF Loads FIB into PFC & distributes to DFC s Hardware Based CEF Process Software CEF Takes RIB and builds a Forwarding Information Base (FIB) containing IP/mask prefixes 1. FIB lookup based on destination prefix (longest-match) 2. FIB Hit returns Adjacency pointer 3. Adjacency contains Rewrite (next-hop) information 4. ACL, QoS & NetFlow lookups occur in parallel, and effect final result 58

50 Catalyst 6500 IP Unicast Forwarding Layer 3 Forwarding on PFC Located on the PFC are the FIB and Adjacency Table The FIB contains: L3 entries are arranged logically from MOST to LEAST specific (based on /mask) Overall FIB hardware shared by: IPv4 Unicast IPv4 Multicast IPv6 Unicast IPv6 Multicast MPLS The Adjacency Table: L2 Re-Write information and / or pointers for replication Hardware adjacency table also shared among protocols MASK (/32) MASK (/24) MASK (/16) MASK (/0) FIB TCAM IF, MACs, MTU IF, MACs, MTU IF, MACs, MTU IF, MACs, MTU Adjacency Table 59

51 Catalyst 6500 Internals Layer 3 Forwarding on PFC Assuming a lookup was performed for a packet with a destination of /24, then the following would occur 1 Packet Key Gen 3 Lookup Key HIT! MASK (/32) MASK (/24) MASK (/16) 4 Load-Sharing Hash 5 6 IF, MACs, MTU IF, MACs, MTU IF, MACs, MTU IF, MACs, MTU 7 Adjacency Table MASK (/0) FIB TCAM 60

52 Supervisor FIB TCAM Resources IPv6 and IPv4 multicast require 2 entries MPLS and IPv4 only one XL PFCs = 1M entries Non-XL PFCs = 256K entries By default TCAM is allocated as seen in the table NON-XL PFCs XL PFCs IPv4, MPLS 192k 512k IPv6, Multicast 32k 256k SUP720-3BXL Example 6509E#sh mls cef maximum-routes FIB TCAM maximum routes : ======================= Current : IPv4 + MPLS - 512k (default) IPv6 + IP Multicast - 256k (default) Changing default (requires Reboot!) 6509E(config)#mls cef maximum-routes? ip number of ip routes ip-multicast number of multicast routes ipv6 number of ipv6 routes mpls number of MPLS labels 62

53 Displaying IPv4 Forwarding Summary 6509E#show platform hardware capacity forwarding <snip> L3 Forwarding Resources FIB TCAM usage: Total Used %Used 72 bits (IPv4, MPLS, EoM) % 144 bits (IP mcast, IPv6) % detail: Protocol Used %Used IPv4 28 1% MPLS 0 0% EoM 0 0% IPv6 1 1% IPv4 mcast 3 1% IPv6 mcast 3 1% <snip> Adjacency usage: Total Used %Used % 63

54 Displaying Hardware IPv4 Prefix Entries 6509E#show platform hardware cef Codes: decap - Decapsulation, + - Push Label Index Prefix Adjacency /32 receive /32 receive /32 receive /32 receive /32 Gi1/1, 0030.f272.31fe /24 receive /24 glean /24 Gi1/1, 0030.f272.31fe /24 Gi1/1, 0030.f272.31fe /24 Gi1/1, 0030.f272.31fe /24 Gi1/1, 0030.f272.31fe < > 64

55 Finding the Longest-Match Prefix Entry 6509E#show platform hardware cef Codes: decap - Decapsulation, + - Push Label Index Prefix Adjacency 6509E#show platform hardware cef lookup Codes: decap - Decapsulation, + - Push Label Index Prefix Adjacency /8 Vl192,00d bc E#show platform hardware cef ipv6 lookup FF00:: Codes: + - Push label Index Prefix Adjacency 512 FF00::/8 glean 65

56 IPv4 CEF Load Sharing Up to 16* hardware load-sharing paths per prefix Use maximum-paths command in routing protocols to control number of load-sharing paths IPv4 CEF load-sharing is per-ip flow /16 via Rtr-A via Rtr-B Per-packet load-balancing not supported Load-sharing based on Source and Destination IP addresses by default Unique ID in PFC3 and PFC4 prevents polarization A B Configuration option supports inclusion of L4 ports in the hash mls ip cef load-sharing full /16 Unique ID not included in hash in full mode 66

57 Load-Sharing Prefixes and Paths 6509E#show platform hardware cef lookup Codes: decap - Decapsulation, + - Push Label Index Prefix Adjacency /24 Gi1/1, 0030.f272.31fe Gi1/2, ca8.484c Gi2/1, 000e.382d.0b90 Gi2/2, 000d.6550.a8ea 6509E#show platform hardware cef exact-route Interface: Gi1/1, Next Hop: , Vlan: 1019, Destination Mac: 0030.f272.31fe 6509E#show platform hardware cef exact-route Interface: Gi2/2, Next Hop: , Vlan: 1018, Destination Mac: 000d.6550.a8ea 67

58 Agenda Chassis and Power Supplies Supervisor Engine and Switch Fabric Architectures Module Architectures Layer 2 Forwarding IP Unicast Forwarding NetFlow Access Control Lists Packet Walks 68

59 Catalyst 6500 NetFlow Netflow is a process designed to collect information about traffic flows that pass through the switch - Netflow collection of flow records is a hardware process while the exporting of flow records to an external collector is a control plane process Netflow Netflow Collection Server Data Flow Exported Netflow Record 69

60 Catalyst 6500 NetFlow PFC3 Flow Masks The Catalyst 6500 supports the following flow masks - these are used to identify which pieces of information in the header will be used as input into generating a key for flow lookups 70

61 Catalyst 6500 NetFlow TCAM Lookup on PFC3 2 Packet 1 Flow Key Hash Key 3 Hash Function 4 Compare Mask HIT! Mask Key Key Key Key Key Key Key Key Key Key Key 5 Compare NetFlow Table Index 128K/256K entries 6 Result Flow Data Flow Data Flow Data Flow Data Flow Data Flow Data Flow Data Flow Data Flow Data Flow Data Flow Data Flow Data HIT! 7 Statistics 128K/256K rows Netflow TCAM Netflow Table Key 128 entries Alias CAM 71

62 Catalyst 6500 NetFlow NetFlow Export Process Supervisor Netflow Export Netflow Data EOBC WS-X6748-GE-TX w\dfc4 Netflow Data WS-X G-2T\2TXL Netflow Data Netflow Collector Direct Export supported with Supervisor 720 and : WS-X GE-3C/3CXL WS-X x-3C/3CXL Direct Export supported with Supervisor 2T and : WS-X x upgraded with DFC4-E / DFC4-EXL WS-X x-2T/2TXL WS-X G-2T/2TXL WS-X G-2T/2TXL 73

63 Catalyst 6500 NetFlow PFC4 Key Enhancements The PFC4 can do everything the PFC3 can do and adds these new capabilities: Increased Support for NetFlow Entries Up to 1M NetFlow entries (512K for Ingress and 512K for Egress) can now be stored in PFC4XL. Improved NetFlow Hash The hash efficiency is improved to 99%, allowing a greater percentage of the NetFlow table to be utilized. Egress NetFlow Provides support for collecting flow statistics for packets after they have had ingress processing applied to them. Sampled NetFlow in Hardware Allows users to to have NetFlow records created based on a sample of traffic matching the flow. Flexible NetFlow Supports the NetFlow V9 Record Format including new fields for IPV6 and Multicast information. TCP Flags TCP Flags (SYN, FIN, RST, ACK, URGENT, PUSH) are now collected as part of a flow record. CPU Friendly Export Protects the CPU from being overrun by heavy NetFlow Data Export 74

64 Catalyst 6500 NetFlow PFC4 NetFlow Processing IFE NetFlow Sampling and Lookup IFE NetFlow Statistics Accounting of packets admitted by input processing Incoming Packet Ingress ACL Ingress NetFlow L3 Lookup Ingress QoS IFE Process OFE Process Outgoing Packet Egress QoS Egress NetFlow Egress ACL OFE NetFlow Statistics Accounting of forwarded packet OFE NetFlow Sampling and Lookup 75

65 Catalyst 6500 NetFlow TCAM Lookup on PFC4 DST IP SRC IP Proto DST Port SRC Port x Flow Key Flow Key Hash Function 2 Compare all pages 4 Data Data Data Data Key Key Key Key Data Data Data Data Key Key Key Key Data Data Data Data Key Key Key Key Data Data Data Data Key Key Key Key Data Data Data Data Key Key Key Key Data Data Data Data Key Key Key Key HIT! Data Data Data Data Key Key Key Key Data Data Data Data Key Key Key Key Data Data Data Data Key Key Key Key Data Data Data Data Key Key Key Key Data Data Data Data Key Key Key Key Index Index Index Index Index Index Index Index Index Index Index Index Index Index Index Index Index Index Index Index Index Index Index Index Index Index Index Index Index Index Index Index Index Index Index Index Index Index Index Index Index Index Index Index Compare Flow Data 6 5 Index to NF Data Table Flow Data Flow Data Flow Data Flow Data Flow Data Flow Data Flow HIT! Data Flow Data Flow Data Flow Data Flow Data 7 Update Stats Statistics Statistics Statistics Statistics Statistics Statistics Statistics Statistics Statistics Statistics Statistics Lookup Key 3 Data Key Indexes row in Lookup Table 512K entries NetFlow Lookup Table NetFlow Data Table NetFlow Statistics Table 76

66 Catalyst 6500 NetFlow Flexible NetFlow Configuration Key Field Key Field Flow Record Non-Key Field Non-Key Field Multiple Exporters can be associated with a single FNF monitor Flow Export Export Profile Export Profile Key Fields trigger the creation of a new Flow entry every time their value change Non-Key Fields are data that is indexed by the Key Fields. Flow Monitor Flow Monitor Key Fields are defined using the match statement Non-Key-Fields are defined using the collect statement Interfaces Ingress or/and Egress. Ingress or/and Egress Same Flow Monitor can be associated with multiple Interfaces. 77

67 Catalyst 6500 NetFlow CPU Friendly Export NDE increases export rate until threshold reached CPU Utilization When threshold reached, NDE quickly backs off export rate 70% Yielding NDE threshold 30% Wait 5 seconds and then step up export rate again CPU before NDE begins 80

68 Catalyst 6500 NetFlow Integration with EEM Example I: Malformed Packets Detection & Reporting Attacker sending malformed pkts with TTL=0 NetFlow cache srcif SrcIPadd DstIf DstIPadd TTL Fa1/ Fa0/ Fa1/ Fa0/ TTL = 0 triggers an EEM event *MAR :29: UTC: %HA_EM-6-LOG: my-ttl-applet: flow record with zero TTL Fa1/ Fa0/ syslog message generated based on preconfigured policies Example II : Anomaly Flow Detection and Mitigation Compromised user sending traffic with high rate NetFlow cache srcif SrcIPadd DstIf DstIPadd bytes Fa1/ Fa0/ Fa1/ Fa0/ Fa1/ Fa0/ NetFlow ED triggers policies to monitor flow rate. Typically, voice conversations are 64kbps *Feb 18 01:24:30.455: %LINK-5- CHANGED: Interface FastEthernet 1/0, changed state to administratively down interface Fa1/0 is shut down when the flow rate exceeds 1Mbps 81

69 Displaying NetFlow Utilization 6509E#show platform hardware capacity netflow Netflow resources: Netflow table size: entries total Netflow table usage: Module/Instance Input flows Output flows 3 10% 10% 7 25% 25% 82

70 Agenda Chassis and Power Supplies Supervisor Engine and Switch Fabric Architectures Module Architectures Layer 2 Forwarding IP Unicast Forwarding NetFlow Access Control Lists Packet Walks 83

71 Catalyst 6500 Access Control Lists Hardware Support 1 Create the ACL or traffic classification policy using CLI or Network Management System IP Access-List extended Internet permit ip any host permit ip any host permit ip any host permit ip any host DFC DFC PFC PFC DFC Hardware Support Policy Feature Card (PFC) Distributed Forwarding Card (DFC) Router ACLs Vlan ACLs Port Based ACLs Role Based ACLs 2 Hardware- Assist Features Netflow WCCP Reflexive ACLs Network Address Translation Cisco Trust Sec 3 84

72 Catalyst 6500 Access Control Lists Three Forms of Security ACLs The PFC3/PFC4 supports three forms of Security ACLs: the RACL, VACL and PACL Router ACL (RACL) VLAN ACL (VACL) Port ACL (PACL) Used to permit or deny the movement of traffic between Layer 3 Subnets Used to permit or deny the movement of traffic between Layer 3 Subnets/VLANs or within a VLAN Used to permit or deny the movement of traffic between Layer 3 Subnets/VLANs or within a VLAN Applied as an input or output policy to a Layer 3 interface Applied as a policy to a VLAN - is inherently applied to both inbound and outbound traffic Applied as a policy to a Layer 2 Switch port interface - is applied for inbound traffic only 85

73 Catalyst 6500 Access Control Lists ACL Order of Processing Should a RACL, VACL and PACL all be configured at the same time, there is a distinct order in which each form of ACL is processed Input RACL Output RACL VACL VACL Input PACL Note that no Output PACL exists Destination Source 86

74 Catalyst 6500 Access Control Lists PFC3 TCAM Population Protocol Dest IP Dest Port Source IP Source Port FFFFFFFF xxxxxxxx xx xxxx xxxx xxxxxxxx xx xxxx xxxx xxxxxxxx xx xxxx xxxx Permit Deny Deny 1= Compare 0= Mask 6 7 ip access-list extended example permit ip any host deny ip any host deny ip any host permit tcp any any eq 22 deny tcp any any eq 23 deny udp any any eq FF 0000 FFFF xxxxxxxx xxxxxxxx 06 xxxx 0016 xxxxxxxx xxxxxxxx 06 xxxx 0017 xxxxxxxx xxxxxxxx 11 xxxx 0202 xxxxxxxx xxxxxxxx 06 xxxx 0080 xxxxxxxx xxxxxxxx 11 xxxx 00A Permit Deny Deny Permit Permit permit tcp any any eq 80 permit udp any any eq Masks Values 87

75 Catalyst 6500 Access Control Lists PFC3 TCAM Lookup Generate Lookup Key ip access-list extended example permit ip any host deny ip any host deny ip any host permit tcp any any eq 22 deny tcp any any eq 23 deny udp any any eq 514 permit tcp any any eq 80 permit udp any any eq Packet xxxxxxxx xxxxxxxx 06 xx xxxx 84C xxxx 0050 Lookup Key 2 SIP= DIP= Protocol=TCP (6) SPORT=33992 DPORT=80 Entries matching only destination IP FFFFFFFF FF 0000 FFFF HIT! Entries matching only protocol and destination port Masks Compare xxxxxxxx xx xxxx xxxx xxxxxxxx xx xxxx xxxx xxxxxxxx xx xxxx xxxx xxxxxxxx xxxxxxxx 06 xxxx 0016 xxxxxxxx xxxxxxxx 06 xxxx 0017 xxxxxxxx xxxxxxxx 11 xxxx 0202 xxxxxxxx xxxxxxxx 06 xxxx 0050 xxxxxxxx xxxxxxxx 11 xxxx 00A1 Values Permit Result 88

76 Catalyst 6500 Access Control Lists PFC4 Mask Utilization PFC3 ACL TCAM MASK MASK permit permit permit Implements 8:1 Mask to Entry ratio Total 4K Masks, 32K Entries Mask resource is limited permit ip any permit ip any permit ip any Mask Mask Mask permit permit permit Implements 1:1 Mask to Entry ratio Total 256K Masks, 256K Entries PFC4 ACL TCAM Mask resource is no longer a limited resource 90

77 Catalyst 6500 Access Control Lists PFC4 Lookup Example BANK 0 QoS TCAM A BANK 1 VACL TCAM B BANK 2 BANK 3 SGT RACL Forwarding Engine (PFC4 or DFC4) 3 TCAM Controller ACE Counters (L2 ) 2 Packet Header Information 1 2 X Lookup Keys ACL Labels ACL LOUs Classification Module 1 4 X Results 4 4 X Result Data Classification Module 2 Final Result to Netflow 8 91

78 Catalyst 6500 Access Control Lists PFC4 ACL Dry Run Feature Make sure the ACL will fit in the TCAM before you apply the ACL - ACLs that do not fit can cause software forwarding and possible high CPU utilization Special configuration session SUP2T-E#show configuration session test status ==================================== Status of last config validation: Timestamp: @17:27:06 ====================================== SLOT = [1] Result = Configuration will fit in TCAM - Create and edit ACls - Verifies if the changes will fit within the hardware resources The actual changes are not programmed into the hardware during the configuration session Configuration changes can be verified step by step 93

79 Catalyst 6500 Access Control Lists PFC4 ACL Hitless Update Allows updates to an ACL without interrupting traffic Multiple features updated at once IPv4 IPv6 MAC IPv4, IPv6, MAC RACL, VACL, PBR ACL Updates Global configuration option (default is on) Feature does consume double the number of TCAM entries 95

80 Catalyst 6500 Access Control Lists PFC4 ACL Hitless Update Each ACL feature is initially programmed into two different spaces into the TCAM Primary space (Label -1) Shadow space (label-2) While an ACL is being updated the PFC4 will use a temporary label that points to the shadow TCAM space Once the ACL changes have been completed the then PFC4 will then use the original label again BANK 0 QoS-1 QoS-2 TCAM A BANK 1 VACL-1 VACL-2 BANK 2 SGT-1 SGT-2 TCAM B 2 X Lookup Keys 4 X Results ACL Labels 1, 2 TCAM Controller ACL LOUs BANK 3 RACL-1 RACL-2 Classification Module 1 96

81 Agenda Chassis and Power Supplies Supervisor Engine and Switch Fabric Architectures Module Architectures Layer 2 Forwarding IP Unicast Forwarding NetFlow Access Control Lists Packet Walks 97

82 Centralized Forwarding: Classic to Classic 1 Port A P R P R P R P R P Port B Port A Port B 4 P Slot 1 Classic Slot 2 Classic Dbus Rbus Michael Engineering Switch Fabric Amanda Marketing R 3 H Fabric / Bus Interface & Replication R P 2 R Layer 2 Engine Layer 3 Engine PFC4 P H = Packet = Header Supervisor Engine 2T R = Result

83 Centralized Forwarding: Classic to CEF720 1 Port A Slot 1 Classic P R P Port B R P Port A Slot 2 CEF720 Port B CFC 5 P FIRE A FIRE B Dbus Michael Engineering Switch Fabric 4 Rbus Amanda Marketing Question : How will the packet get to the CEF720 Module? Bus or Switch Fabric? ADD QUESTION IN SAN DIEGO. Make it appear between Steps 3 & 4 R 3 Fabric / Bus Interface & Replication Supervisor Engine 2T H R P 2 R Layer 2 Engine Layer 3 Engine PFC4 P H R = Packet = Header = Result

84 Centralized Forwarding: CEF720 to Classic 9 Port A Slot 1 Classic R P Port B R P Port A Slot 2 CEF720 Port B 1 CFC R 2 FIRE A H FIRE B P 7 3 Dbus Michael Engineering Switch Fabric 6 Rbus Amanda Marketing 8 R 5 H H Fabric / Bus Interface & Replication R P 4 R 5 Layer 2 Engine Layer 3 Engine PFC4 P H = Packet = Header Supervisor Engine 2T R = Result

85 Centralized Forwarding: CEF720 to CEF720 1 Port A Slot 1 CEF720 Port B Port A Slot 2 CEF720 Port B 2 R CFC CFC 7 P FIRE A H FIRE B FIRE A FIRE B 6 3 Dbus Rbus Michael Engineering Switch Fabric Amanda Marketing R 5 H Fabric / Bus Interface & Replication H 5 4 R Layer 2 Engine Layer 3 Engine PFC4 P H = Packet = Header Supervisor Engine 2T R = Result

86 Distributed Forwarding: CEF720/DFC4 to CEF720/DFC4 1 Slot 1 CEF720/DFC4 Port A Port B 3 Slot 2 CEF720/DFC4 Port A Port B P 2 FIRE A DFC4 H R 4 L 2 L 3 FIRE B FIRE A DFC4 L 2 L 3 FIRE B 6 5 Dbus Rbus Michael Engineering Switch Fabric Amanda Marketing PFC4 Fabric / Bus Interface & Replication Layer 2 Engine Layer 3 Engine P H = Packet = Header Supervisor Engine 2T R = Result

87 Summary The Catalyst 6500 architecture provides a robust infrastructure upon which the system can provide hardware-based forwarding at high speeds L2 and L3 switching are done via the same hardware forwarding process, so there is no difference in performance between the two Enabling features such as Netflow, QoS and ACLs can be done without impact to forwarding performance as these features are processed in hardware in parallel to the L2 and L3 lookup processes The Catalyst 6500 architecture is designed so that unicast and multicast can coexist within the same infrastructure, providing a versatile platform for the networks of today and tomorrow 108

88 Conclusion You should now have a thorough understanding of the Catalyst 6500 switching architecture, packet flow, and key forwarding engine functions Any Questions? 109

89 Complete Your Online Session Evaluation Give us your feedback and you could win fabulous prizes. Winners announced daily. Receive 20 Passport points for each session evaluation you complete. Complete your session evaluation online now (open a browser through our wireless network to access our portal) or visit one of the Internet stations throughout the Convention Center. Don t forget to activate your Cisco Live Virtual account for access to all session material, communities, and on-demand and live activities throughout the year. Activate your account at the Cisco booth in the World of Solutions or visit 110

90 Final Thoughts Get hands-on experience with the Walk-in Labs located in World of Solutions, booth 1042 Come see demos of many key solutions and products in the main Cisco booth 2924 Visit after the event for updated PDFs, ondemand session videos, networking, and more! Follow Cisco Live! using social media: Facebook: Twitter: LinkedIn Group: 111

91 Presentation_ID