Cisco Datacenter 3.0 Datacenter Trends David Gonzalez Consulting Systems Engineer Cisco 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 1
Agenda Data Center Ethernet (DCE) Fiber Channel over Ethernet (FCoE) Virtualization 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 2
What is Data Center Ethernet (DCE)? Data Center Ethernet is an architectural collection of Ethernet extensions designed to improve Ethernet networking and management in the Data Center. 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 3
What s the difference between DCE, CEE and DCB? Nothing! All 03 acronyms describe the same thing, meaning the architectural collection of Ethernet extensions (based on open standards) Cisco has co-authored many of the standards associated and is focused on providing a standards-based solution for a Unified Fabric in the data center The IEEE has decided to use the term DCB (Data Center Bridging) to describe these extensions to the industry. http://www.ieee802.org/1/pages/dcbridges.html 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 4
Data Center Ethernet Standards and Features Overview Feature Priority-based Flow Control (PFC) - 802.1Qbb Enhanced Transmission Selection - 802.1Qaz Congestion Notification (BCN/QCN) - 802.1Qau Data Center Bridging Capability Exchange Protocol - 802.1AB (LLDP) L2 Multi-path for Unicast & Multicast Lossless Service Benefit Provides class of service flow control. Ability to support storage traffic Grouping classes of traffic into Service Lanes IEEE 802.1Qaz, CoS based Enhanced Transmission End to End Congestion Management for L2 network Auto-negotiation for Enhanced Ethernet capabilities DCBX Eliminate Spanning Tree for L2 topologies Utilize full Bi-Sectional bandwidth with ECMP Provides ability to transport various traffic types (e.g. Storage, RDMA) 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 5
Data Center Ethernet Standards and Features Overview Feature Priority-based Flow Control (PFC) - 802.1Qbb Benefit Provides class of service flow control. Ability to support storage traffic 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 6
Link Level Flow Control Ethernet PAUSE: Transmit Frame STOP PAUSE Fibre Channel Buffer-to-buffer Credits: R_RDY Transmit Frame 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 7
Data Center Ethernet Features - PFC Priority-Based Flow Control (PFC) Enables lossless Fabrics for each class of service PAUSE sent per virtual lane when buffers limit exceeded Network resources are partitioned between VL s (E.g. input buffer and output queue) The switch behavior is negotiable per VL 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 8
Data Center Ethernet Standards and Features Overview Feature Priority-based Flow Control (PFC) - 802.1Qbb Enhanced Transmission Selection - 802.1Qaz Benefit Provides class of service flow control. Ability to support storage traffic Grouping classes of traffic into Service Lanes IEEE 802.1Qaz, CoS based Enhanced Transmission 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 9
Data Center Ethernet Features - ETS Enhanced Transmission Selection (ETS) Enables Intelligent sharing of bandwidth between traffic classes control of bandwidth Being Standardized in IEEE 802.1Qaz Also known as Priority Grouping 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 10
Data Center Ethernet Standards and Features Overview Feature Priority-based Flow Control (PFC) - 802.1Qbb Enhanced Transmission Selection - 802.1Qaz Congestion Notification (BCN/QCN) - 802.1Qau Benefit Provides class of service flow control. Ability to support storage traffic Grouping classes of traffic into Service Lanes IEEE 802.1Qaz, CoS based Enhanced Transmission End to End Congestion Management for L2 network 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 11
Data Center Ethernet Features Congestion Management Moves congestion out of the core to avoid congestion spreading Allows End-to-End congestion management Standards track in IEEE 802.1Qau 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 12
Data Center Ethernet Standards and Features Overview Feature Priority-based Flow Control (PFC) - 802.1Qbb Enhanced Transmission Selection - 802.1Qaz Congestion Notification (BCN/QCN) - 802.1Qau Data Center Bridging Capability Exchange Protocol - 802.1AB (LLDP) Benefit Provides class of service flow control. Ability to support storage traffic Grouping classes of traffic into Service Lanes IEEE 802.1Qaz, CoS based Enhanced Transmission End to End Congestion Management for L2 network Auto-negotiation for Enhanced Ethernet capabilities DCBX 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 13
Data Center Bridging Exchange Devices need to discover the edge of the enhanced Ethernet cloud: Each edge switch needs to learn that it is connected to a legacy switch. Servers need to learn whether or not they are connected to Enhanced Ethernet device. Within the Enhanced Ethernet cloud, devices need to discover the capabilities of its peers. DCBX utilizes the linklayer discovery protocol and handles local operational configuration for each feature 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 14
Data Center Bridging Exchange (Cont.) Link partners can choose supported features and willingness to accept configuration from peer. Feature TLVs: Priority Groups (Link Scheduling) Priority-based Flow Control Congestion Management (Backwards Congestion Notification) Application (frame priority usage) Logical Link Down Details on the DCBX can be found on Intel site http://www.intel.com/technology/eedc/index.htm 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 15
Data Center Ethernet Standards and Features Overview Feature Priority-based Flow Control (PFC) - 802.1Qbb Enhanced Transmission Selection - 802.1Qaz Congestion Notification (BCN/QCN) - 802.1Qau Data Center Bridging Capability Exchange Protocol - 802.1AB (LLDP) L2 Multi-path for Unicast & Multicast Benefit Provides class of service flow control. Ability to support storage traffic Grouping classes of traffic into Service Lanes IEEE 802.1Qaz, CoS based Enhanced Transmission End to End Congestion Management for L2 network Auto-negotiation for Enhanced Ethernet capabilities DCBX Eliminate Spanning Tree for L2 topologies Utilize full Bi-Sectional bandwidth with ECMP 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 16
Data Center Ethernet Features L2MP Layer 2 Multi-Pathing Phase 1 Phase 2 Phase 3 LAN LAN Virtual Switch LAN MAC A MAC B Active-Active vpc L2 ECMP MAC A MAC B L2 ECMP Eliminates STP on Uplink Bridge Ports Allows Multiple Active Uplinks Switch to Network Prevents Loops by Pinning a MAC Address to Only One Port Completely Transparent to Next Hop Switch We are here Virtual Switch retains physical switches independent control and data planes Virtual port channel mechanism is transparent to hosts or switches connected to the virtual switch STP as fail-safe mechanism to prevent loops even in the case of control plane failure Uses ISIS based topology Eliminates STP from L2 domain Preferred path selection TRILL is the work in progress standard by the IETF Working Group 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 17
Data Center Ethernet Standards and Features Overview Feature Priority-based Flow Control (PFC) - 802.1Qbb Enhanced Transmission Selection - 802.1Qaz Congestion Notification (BCN/QCN) - 802.1Qau Data Center Bridging Capability Exchange Protocol - 802.1AB (LLDP) L2 Multi-path for Unicast & Multicast Lossless Service Benefit Provides class of service flow control. Ability to support storage traffic Grouping classes of traffic into Service Lanes IEEE 802.1Qaz, CoS based Enhanced Transmission End to End Congestion Management for L2 network Auto-negotiation for Enhanced Ethernet capabilities DCBX Eliminate Spanning Tree for L2 topologies Utilize full Bi-Sectional bandwidth with ECMP Provides ability to transport various traffic types (e.g. Storage, RDMA) 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 18
Virtual Links An example Up to 8 VL s per physical link Ability to support QoS queues within the lanes DCE CNA VL2 - No Drop Service - Storage VL1 LAN Service LAN/IP LAN/IP Gateway DCE CNA DCE CNA VL1 VL2 VL3 VL3 Delayed Drop Service - IPC Campus Core/ Internet Storage Gateway Storage Area Network 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 19
Fiber Channel Over Ethernet 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 20
Fibre Channel over Ethernet How it works Direct mapping of Fibre Channel over Ethernet FC-4 FC-4 FC-3 FC-3 SOF FC Frame CRC EOF FC-2 FC-2 FC-1 FC-0 FCoE Mapping MAC PHY Ethernet Header Ethernet Payload Ethernet FCS (a) Protocol Layers Leverages standards-based extensions to Ethernet (DCE) to provide reliable I/O delivery Priority Flow Control (PFC) (b) Frame Encapsulation Data Center Bridging Capability exchange Protocol (DCBX) 10GE Lossless Ethernet Link (DCE) FCoE Traffic Other Networking Traffic 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 21
FCoE Enablers 10Gbps Ethernet Lossless Ethernet Matches the lossless behavior guaranteed in FC by B2B credits Ethernet jumbo frames Max FC frame payload = 2112 bytes Normal ethernet frame, ethertype = FCoE Same as a physical FC frame Ethernet Header FCoE Header FC Header FC Payload CRC EOF FCS Control information: version, ordered sets (SOF, EOF) 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 22
Encapsulation technologies Operating System / Applications SCSI Layer FCP iscsi FCP FCP FCP SRP FCIP ifcp TCP TCP TCP IP IP IP FCoE FC Ethernet IB 1, 2, 4, 8, 10 Gbps 1, 10... Gbps 10, 20, 40 Gbps 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 23
Encapsulation technologies OS / Applications SCSI Layer FCP FCoE E. Ethernet 1, 10... Gbps FCP layer is untouched Allows same management tools for Fibre Channel Allows same Fibre Channel drivers Allows same Multipathing software Simplifies certifications with OSMs Evolution rather than Revolution 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 24
Unified I/O (FCoE) Why? Fewer CNAs (Converged Network adapters) instead of NICs, HBAs and HCAs Limited number of interfaces for Blade Servers FC HBA FC Traffic FC HBA NIC NIC FC Traffic LAN Traffic LAN Traffic CNA CNA All traffic goes over 10GE NIC Mgmt Traffic NIC Backup Traffic HCA IPC Traffic 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 25
Unified I/O: What changes on the network? Today: Management LAN SAN A FC HBA FC HBA SAN B Core switches NIC NIC Access Top of the Rack switches Servers Ethernet FC 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 26
Unified I/O: just the access layer Management FCoE Switch LAN Unified TodayI/O SAN A SAN B Unified I/O Reduction of server adapters Fewer Cables Simplification of access layer & cabling Gateway free implementation - fits in installed base of existing LAN and SAN L2 Multipathing Access Distribution Lower TCO Investment Protection (LANs and SANs) Consistent Operational Model One set of ToR Switches FCoE Ethernet FC 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 27
Converged Network Adapters (CNA) 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 28
CNA View on Host 10 GE/FCoE Cisco ASIC 10 GE Fibre Channel 10 GE FC PCIe Bus 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 29
CNA View on ware ESX Fibre Channel Emulex Qlogic 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 30
CNA View on ware ESX 10 GE Both Emulex and Qlogic are using Intel Oplin 10 Gigabit Ethernet chip 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 31
Disk Management Storage is zoned to FC initiator of host. 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 32
Traffic Prioritization Applications PCI Functions (Each appears to OS as separate device or port.) Traffic Classes and Queues QOS Controls FC L2 NIC 0 1 2 3 4 5 6 7 OS assigns socket APIs to traffic classes via 802.1p QoS tagging (L2 NIC only). CNA selects type and quantity of each protocol and assignment to traffic classes. CNA configures QoS parameters for each traffic class. Physical Port 10 Gigabit Ethernet 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 33
FCoE SW Stack Supported on Intel Oplin 10GbE Adapters SW upgraded turns 10GbE adapter into FCoE adapter Software Implementation Initiator and Target mode FCP, FC class 3 Fully supports Ethernet pause frames (per priority pause) Supported OS Linux: Redhat & SLES Windows Free Access to the SAN Software Hardware FCoE Software Stack L2 Ethernet NIC Web site: www.open-fcoe.org 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 34
Common SAN/LAN Architecture Administrative Boundaries LAN SAN A SAN B Network Admin Login: Net_admin Password: abc1234 SAN Admin Login: SAN_admin Password: xyz6789 Ethernet FC 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 35
Common SAN/LAN Architecture Administrative Boundaries LAN SAN A SAN B Network Admin Login: Net_admin Password: abc1234 SAN Admin Login: SAN_admin Password: xyz6789 NX5000 CNA CNA CNA CNA DataCenter Ethernet with FCoE Ethernet FC 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 36
Virtualization 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 37
Today s Quad Core Processors Intel Core 2 Duo Q6600, 2.7 GHz, FSB-1200, 8 MB L2 Cache 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 38
Sample Intel CPU Comparisons Two to four Cores Source: Tom s Hardware 20000 18000 16000 14000 12000 10000 8000 6000 4000 2000 0 Core 2 Duo E6300 Core 2 Quad Q6700 Core i7 965 Extreme 3DMark Vantage CPU 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 39
2009 Cisco Systems, Inc. All rights reserved. Cisco Public 40
2009 Cisco Systems, Inc. All rights reserved. Cisco Public 41
2009 Cisco Systems, Inc. All rights reserved. Cisco Public 42
Power & Cooling, and Weight Directly affects server density per rack Greater density = more power, cooling & weight Rules of Thumb: 170 200KW/ft 2 (2,200watts/M 2 ) ~4.4KW cooling average per rack Average Watts Server Rack Optimized: ~250W Mid-range: ~700W Blades: ~5KW High-end: ~13KW Increasing power & cooling can significantly increase cost Decreasing density may require changing design Source: ESTIMATING TOTAL POWER CONSUMPTION BY SERVERS IN THE U.S. AND THE WORLD Jonathan Koomey PhD. Weight also needs to be considered, especially for raised floors 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 43
Virtual Appliances Pre-installed, pre-configured packages of application + OS > Easy to provision enterprise software > Deploy on any hardware > Less management: OS is small, more secure Over 300 available: OS, security, RDBS, infrastructure, collaboration, etc. 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 44
2009 Cisco Systems, Inc. All rights reserved. Cisco Public 45
Server Virtualization is #1 Priority for CIOs Efficient utilization, Reduce Cable & Power Costs Today Virtualization Step1 Virtualization Step2 Virtualization Step3 Hypervisor Hypervisor Hypervisor GE 10 GE 10 GE/FCOE Access Layer LAN SAN A SAN B Access Layer LAN SAN ASAN B Access Layer LAN SAN A Nexus 5000 SAN B 4 x 1GE 10GE Many under utilized servers Cable sprawl High power, cooling costs High CAPEX For every $1 spent on server capex ~$5 spent on opex Ethernet Server Fibre- Channel Cable sprawl power, cooling costs Less number of access layer Ethernet ports Ethernet Server Fibre- Channel GE to 10GE in access layer Less interfaces reduced Cable sprawl Savings from power and cooling Server Unified IO Unified I/O - LAN & SAN consolidation Reduce NICs, HBAs, Reduce cabling Ultimate Savings: from power and cooling, Lower capex Engaging Network, Server & Storage teams is key 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 46
VN-Link Brings Level Granularity VLAN 101 otion Problems: otion may move s across physical ports policy must follow Impossible to view or apply policy to locally switched traffic Cannot correlate traffic on physical links from multiple s VN-Link: Extends network to the Consistent services Coordinated, coherent management 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 47
Cisco Nexus 1000V Industry First 3 rd Party Distributed Virtual Switch #1 Server 1 #2 #3 W ESX #4 #5 Server 2 #6 #7 ware Nexus vswitch 1000VNexus 1000V DVS ware Nexus vswitch 1000V W ESX #8 Nexus 1000V provides enhanced switching for ware ESX Features Cisco VN-Link: Policy Based Connectivity Mobility of Network & Security Properties Non-Disruptive Operational Model Ensures proper visibility & connectivity during otion Enabling Acceleration of Server Virtualization Benefits 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 48
Data Center Architecture Evolution Nexus Optimized for the Data Center DC Virtual Access 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 49
Data Center 3.0 Evolution Path Location Freedom HW Freedom Provisioning Freedom Business Process Freedom Consolidation Virtualization Automation Utility Market Inter-Cloud Enterprise-Class Clouds Unified Computing Unified Fabric Data Center Networking 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 50
Cisco Unified Computing System 320 Blades, 640 Intel Nehalem CPUs, 2560 Cores Single Single Point Point of of Management Management Unified Unified Fabric Fabric Expanded Expanded Memory Memory Virtualized Virtualized Adapters Adapters 2009 Cisco Systems, Inc. All rights reserved. Service Service Profiles Profiles Cisco Public 51
Q and A 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 52
2009 Cisco Systems, Inc. All rights reserved. Cisco Public 53