Cisco Datacenter 3.0. Datacenter Trends. David Gonzalez Consulting Systems Engineer Cisco

Transcription

1 Cisco Datacenter 3.0 Datacenter Trends David Gonzalez Consulting Systems Engineer Cisco 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 1

2 Agenda Data Center Ethernet (DCE) Fiber Channel over Ethernet (FCoE) Virtualization 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 2

3 What is Data Center Ethernet (DCE)? Data Center Ethernet is an architectural collection of Ethernet extensions designed to improve Ethernet networking and management in the Data Center Cisco Systems, Inc. All rights reserved. Cisco Public 3

4 What s the difference between DCE, CEE and DCB? Nothing! All 03 acronyms describe the same thing, meaning the architectural collection of Ethernet extensions (based on open standards) Cisco has co-authored many of the standards associated and is focused on providing a standards-based solution for a Unified Fabric in the data center The IEEE has decided to use the term DCB (Data Center Bridging) to describe these extensions to the industry Cisco Systems, Inc. All rights reserved. Cisco Public 4

5 Data Center Ethernet Standards and Features Overview Feature Priority-based Flow Control (PFC) Qbb Enhanced Transmission Selection Qaz Congestion Notification (BCN/QCN) Qau Data Center Bridging Capability Exchange Protocol AB (LLDP) L2 Multi-path for Unicast & Multicast Lossless Service Benefit Provides class of service flow control. Ability to support storage traffic Grouping classes of traffic into Service Lanes IEEE 802.1Qaz, CoS based Enhanced Transmission End to End Congestion Management for L2 network Auto-negotiation for Enhanced Ethernet capabilities DCBX Eliminate Spanning Tree for L2 topologies Utilize full Bi-Sectional bandwidth with ECMP Provides ability to transport various traffic types (e.g. Storage, RDMA) 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 5

6 Data Center Ethernet Standards and Features Overview Feature Priority-based Flow Control (PFC) Qbb Benefit Provides class of service flow control. Ability to support storage traffic 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 6

7 Link Level Flow Control Ethernet PAUSE: Transmit Frame STOP PAUSE Fibre Channel Buffer-to-buffer Credits: R_RDY Transmit Frame 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 7

8 Data Center Ethernet Features - PFC Priority-Based Flow Control (PFC) Enables lossless Fabrics for each class of service PAUSE sent per virtual lane when buffers limit exceeded Network resources are partitioned between VL s (E.g. input buffer and output queue) The switch behavior is negotiable per VL 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 8

9 Data Center Ethernet Standards and Features Overview Feature Priority-based Flow Control (PFC) Qbb Enhanced Transmission Selection Qaz Benefit Provides class of service flow control. Ability to support storage traffic Grouping classes of traffic into Service Lanes IEEE 802.1Qaz, CoS based Enhanced Transmission 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 9

10 Data Center Ethernet Features - ETS Enhanced Transmission Selection (ETS) Enables Intelligent sharing of bandwidth between traffic classes control of bandwidth Being Standardized in IEEE 802.1Qaz Also known as Priority Grouping 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 10

11 Data Center Ethernet Standards and Features Overview Feature Priority-based Flow Control (PFC) Qbb Enhanced Transmission Selection Qaz Congestion Notification (BCN/QCN) Qau Benefit Provides class of service flow control. Ability to support storage traffic Grouping classes of traffic into Service Lanes IEEE 802.1Qaz, CoS based Enhanced Transmission End to End Congestion Management for L2 network 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 11

12 Data Center Ethernet Features Congestion Management Moves congestion out of the core to avoid congestion spreading Allows End-to-End congestion management Standards track in IEEE 802.1Qau 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 12

13 Data Center Ethernet Standards and Features Overview Feature Priority-based Flow Control (PFC) Qbb Enhanced Transmission Selection Qaz Congestion Notification (BCN/QCN) Qau Data Center Bridging Capability Exchange Protocol AB (LLDP) Benefit Provides class of service flow control. Ability to support storage traffic Grouping classes of traffic into Service Lanes IEEE 802.1Qaz, CoS based Enhanced Transmission End to End Congestion Management for L2 network Auto-negotiation for Enhanced Ethernet capabilities DCBX 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 13

14 Data Center Bridging Exchange Devices need to discover the edge of the enhanced Ethernet cloud: Each edge switch needs to learn that it is connected to a legacy switch. Servers need to learn whether or not they are connected to Enhanced Ethernet device. Within the Enhanced Ethernet cloud, devices need to discover the capabilities of its peers. DCBX utilizes the linklayer discovery protocol and handles local operational configuration for each feature 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 14

15 Data Center Bridging Exchange (Cont.) Link partners can choose supported features and willingness to accept configuration from peer. Feature TLVs: Priority Groups (Link Scheduling) Priority-based Flow Control Congestion Management (Backwards Congestion Notification) Application (frame priority usage) Logical Link Down Details on the DCBX can be found on Intel site Cisco Systems, Inc. All rights reserved. Cisco Public 15

16 Data Center Ethernet Standards and Features Overview Feature Priority-based Flow Control (PFC) Qbb Enhanced Transmission Selection Qaz Congestion Notification (BCN/QCN) Qau Data Center Bridging Capability Exchange Protocol AB (LLDP) L2 Multi-path for Unicast & Multicast Benefit Provides class of service flow control. Ability to support storage traffic Grouping classes of traffic into Service Lanes IEEE 802.1Qaz, CoS based Enhanced Transmission End to End Congestion Management for L2 network Auto-negotiation for Enhanced Ethernet capabilities DCBX Eliminate Spanning Tree for L2 topologies Utilize full Bi-Sectional bandwidth with ECMP 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 16

17 Data Center Ethernet Features L2MP Layer 2 Multi-Pathing Phase 1 Phase 2 Phase 3 LAN LAN Virtual Switch LAN MAC A MAC B Active-Active vpc L2 ECMP MAC A MAC B L2 ECMP Eliminates STP on Uplink Bridge Ports Allows Multiple Active Uplinks Switch to Network Prevents Loops by Pinning a MAC Address to Only One Port Completely Transparent to Next Hop Switch We are here Virtual Switch retains physical switches independent control and data planes Virtual port channel mechanism is transparent to hosts or switches connected to the virtual switch STP as fail-safe mechanism to prevent loops even in the case of control plane failure Uses ISIS based topology Eliminates STP from L2 domain Preferred path selection TRILL is the work in progress standard by the IETF Working Group 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 17

18 Data Center Ethernet Standards and Features Overview Feature Priority-based Flow Control (PFC) Qbb Enhanced Transmission Selection Qaz Congestion Notification (BCN/QCN) Qau Data Center Bridging Capability Exchange Protocol AB (LLDP) L2 Multi-path for Unicast & Multicast Lossless Service Benefit Provides class of service flow control. Ability to support storage traffic Grouping classes of traffic into Service Lanes IEEE 802.1Qaz, CoS based Enhanced Transmission End to End Congestion Management for L2 network Auto-negotiation for Enhanced Ethernet capabilities DCBX Eliminate Spanning Tree for L2 topologies Utilize full Bi-Sectional bandwidth with ECMP Provides ability to transport various traffic types (e.g. Storage, RDMA) 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 18

19 Virtual Links An example Up to 8 VL s per physical link Ability to support QoS queues within the lanes DCE CNA VL2 - No Drop Service - Storage VL1 LAN Service LAN/IP LAN/IP Gateway DCE CNA DCE CNA VL1 VL2 VL3 VL3 Delayed Drop Service - IPC Campus Core/ Internet Storage Gateway Storage Area Network 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 19

20 Fiber Channel Over Ethernet 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 20

21 Fibre Channel over Ethernet How it works Direct mapping of Fibre Channel over Ethernet FC-4 FC-4 FC-3 FC-3 SOF FC Frame CRC EOF FC-2 FC-2 FC-1 FC-0 FCoE Mapping MAC PHY Ethernet Header Ethernet Payload Ethernet FCS (a) Protocol Layers Leverages standards-based extensions to Ethernet (DCE) to provide reliable I/O delivery Priority Flow Control (PFC) (b) Frame Encapsulation Data Center Bridging Capability exchange Protocol (DCBX) 10GE Lossless Ethernet Link (DCE) FCoE Traffic Other Networking Traffic 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 21

22 FCoE Enablers 10Gbps Ethernet Lossless Ethernet Matches the lossless behavior guaranteed in FC by B2B credits Ethernet jumbo frames Max FC frame payload = 2112 bytes Normal ethernet frame, ethertype = FCoE Same as a physical FC frame Ethernet Header FCoE Header FC Header FC Payload CRC EOF FCS Control information: version, ordered sets (SOF, EOF) 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 22

23 Encapsulation technologies Operating System / Applications SCSI Layer FCP iscsi FCP FCP FCP SRP FCIP ifcp TCP TCP TCP IP IP IP FCoE FC Ethernet IB 1, 2, 4, 8, 10 Gbps 1, Gbps 10, 20, 40 Gbps 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 23

24 Encapsulation technologies OS / Applications SCSI Layer FCP FCoE E. Ethernet 1, Gbps FCP layer is untouched Allows same management tools for Fibre Channel Allows same Fibre Channel drivers Allows same Multipathing software Simplifies certifications with OSMs Evolution rather than Revolution 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 24

25 Unified I/O (FCoE) Why? Fewer CNAs (Converged Network adapters) instead of NICs, HBAs and HCAs Limited number of interfaces for Blade Servers FC HBA FC Traffic FC HBA NIC NIC FC Traffic LAN Traffic LAN Traffic CNA CNA All traffic goes over 10GE NIC Mgmt Traffic NIC Backup Traffic HCA IPC Traffic 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 25

26 Unified I/O: What changes on the network? Today: Management LAN SAN A FC HBA FC HBA SAN B Core switches NIC NIC Access Top of the Rack switches Servers Ethernet FC 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 26

27 Unified I/O: just the access layer Management FCoE Switch LAN Unified TodayI/O SAN A SAN B Unified I/O Reduction of server adapters Fewer Cables Simplification of access layer & cabling Gateway free implementation - fits in installed base of existing LAN and SAN L2 Multipathing Access Distribution Lower TCO Investment Protection (LANs and SANs) Consistent Operational Model One set of ToR Switches FCoE Ethernet FC 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 27

28 Converged Network Adapters (CNA) 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 28

29 CNA View on Host 10 GE/FCoE Cisco ASIC 10 GE Fibre Channel 10 GE FC PCIe Bus 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 29

30 CNA View on ware ESX Fibre Channel Emulex Qlogic 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 30

31 CNA View on ware ESX 10 GE Both Emulex and Qlogic are using Intel Oplin 10 Gigabit Ethernet chip 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 31

32 Disk Management Storage is zoned to FC initiator of host Cisco Systems, Inc. All rights reserved. Cisco Public 32

33 Traffic Prioritization Applications PCI Functions (Each appears to OS as separate device or port.) Traffic Classes and Queues QOS Controls FC L2 NIC OS assigns socket APIs to traffic classes via 802.1p QoS tagging (L2 NIC only). CNA selects type and quantity of each protocol and assignment to traffic classes. CNA configures QoS parameters for each traffic class. Physical Port 10 Gigabit Ethernet 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 33

34 FCoE SW Stack Supported on Intel Oplin 10GbE Adapters SW upgraded turns 10GbE adapter into FCoE adapter Software Implementation Initiator and Target mode FCP, FC class 3 Fully supports Ethernet pause frames (per priority pause) Supported OS Linux: Redhat & SLES Windows Free Access to the SAN Software Hardware FCoE Software Stack L2 Ethernet NIC Web site: Cisco Systems, Inc. All rights reserved. Cisco Public 34

35 Common SAN/LAN Architecture Administrative Boundaries LAN SAN A SAN B Network Admin Login: Net_admin Password: abc1234 SAN Admin Login: SAN_admin Password: xyz6789 Ethernet FC 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 35

36 Common SAN/LAN Architecture Administrative Boundaries LAN SAN A SAN B Network Admin Login: Net_admin Password: abc1234 SAN Admin Login: SAN_admin Password: xyz6789 NX5000 CNA CNA CNA CNA DataCenter Ethernet with FCoE Ethernet FC 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 36

37 Virtualization 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 37

38 Today s Quad Core Processors Intel Core 2 Duo Q6600, 2.7 GHz, FSB-1200, 8 MB L2 Cache 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 38

39 Sample Intel CPU Comparisons Two to four Cores Source: Tom s Hardware Core 2 Duo E6300 Core 2 Quad Q6700 Core i7 965 Extreme 3DMark Vantage CPU 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 39

40 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 40

41 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 41

42 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 42

43 Power & Cooling, and Weight Directly affects server density per rack Greater density = more power, cooling & weight Rules of Thumb: KW/ft 2 (2,200watts/M 2 ) ~4.4KW cooling average per rack Average Watts Server Rack Optimized: ~250W Mid-range: ~700W Blades: ~5KW High-end: ~13KW Increasing power & cooling can significantly increase cost Decreasing density may require changing design Source: ESTIMATING TOTAL POWER CONSUMPTION BY SERVERS IN THE U.S. AND THE WORLD Jonathan Koomey PhD. Weight also needs to be considered, especially for raised floors 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 43

44 Virtual Appliances Pre-installed, pre-configured packages of application + OS > Easy to provision enterprise software > Deploy on any hardware > Less management: OS is small, more secure Over 300 available: OS, security, RDBS, infrastructure, collaboration, etc Cisco Systems, Inc. All rights reserved. Cisco Public 44

45 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 45

46 Server Virtualization is #1 Priority for CIOs Efficient utilization, Reduce Cable & Power Costs Today Virtualization Step1 Virtualization Step2 Virtualization Step3 Hypervisor Hypervisor Hypervisor GE 10 GE 10 GE/FCOE Access Layer LAN SAN A SAN B Access Layer LAN SAN ASAN B Access Layer LAN SAN A Nexus 5000 SAN B 4 x 1GE 10GE Many under utilized servers Cable sprawl High power, cooling costs High CAPEX For every $1 spent on server capex ~$5 spent on opex Ethernet Server Fibre- Channel Cable sprawl power, cooling costs Less number of access layer Ethernet ports Ethernet Server Fibre- Channel GE to 10GE in access layer Less interfaces reduced Cable sprawl Savings from power and cooling Server Unified IO Unified I/O - LAN & SAN consolidation Reduce NICs, HBAs, Reduce cabling Ultimate Savings: from power and cooling, Lower capex Engaging Network, Server & Storage teams is key 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 46

47 VN-Link Brings Level Granularity VLAN 101 otion Problems: otion may move s across physical ports policy must follow Impossible to view or apply policy to locally switched traffic Cannot correlate traffic on physical links from multiple s VN-Link: Extends network to the Consistent services Coordinated, coherent management 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 47

48 Cisco Nexus 1000V Industry First 3 rd Party Distributed Virtual Switch #1 Server 1 #2 #3 W ESX #4 #5 Server 2 #6 #7 ware Nexus vswitch 1000VNexus 1000V DVS ware Nexus vswitch 1000V W ESX #8 Nexus 1000V provides enhanced switching for ware ESX Features Cisco VN-Link: Policy Based Connectivity Mobility of Network & Security Properties Non-Disruptive Operational Model Ensures proper visibility & connectivity during otion Enabling Acceleration of Server Virtualization Benefits 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 48

49 Data Center Architecture Evolution Nexus Optimized for the Data Center DC Virtual Access 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 49

50 Data Center 3.0 Evolution Path Location Freedom HW Freedom Provisioning Freedom Business Process Freedom Consolidation Virtualization Automation Utility Market Inter-Cloud Enterprise-Class Clouds Unified Computing Unified Fabric Data Center Networking 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 50

51 Cisco Unified Computing System 320 Blades, 640 Intel Nehalem CPUs, 2560 Cores Single Single Point Point of of Management Management Unified Unified Fabric Fabric Expanded Expanded Memory Memory Virtualized Virtualized Adapters Adapters 2009 Cisco Systems, Inc. All rights reserved. Service Service Profiles Profiles Cisco Public 51

52 Q and A 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 52

53 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 53