Data Center Design for the Midsize Enterprise. Silvo Lipovšek Systems Engineer

Transcription

1 Data Center Design for the Midsize Enterprise Silvo Lipovšek Systems Engineer

2 Designing Data Centers for Midsize Enterprises Client Access Midsize WAN / DCI Require Dedicated DC Switches Campus Mostly virtualized, some physical Transition from collapsed core Scalable Size for current needs Reuse components in larger designs Design Options Feature choice + priority = tradeoffs Driving efficiency: SDN, Programmability, Orchestration, Automation Cloud with Control L L2 FC FCoE iscsi / NAS

3 What are you ready for? Direction will depend on where you draw the line: Want to stay with existing toolsets for config & management? Interested in new toolsets to buy some efficiency? Capable of consuming a new set of tools? New or traditional operational model? Image Credit: In notes

4 Data Center Fabric Journey STP VPC FabricPath VXLAN MAN/WAN FabricPath /BGP VXLAN /EVPN MAN/WAN MAN/WAN

5 Common Building Blocks EXISTING 3-TIER DESIGNS PROGRAMMABLE SDN OVERLAY MODEL APPLICATION PROFILES & POLICIES APIC DC Core DC PODs Existing 2-Tier & 3-Tier Designs VPC FEX VXLAN Bridging & Routing Integrated Network Virtualization SDN Controllers Application Centric Infrastructure Policy Model Automation Nexus 3000, 5600, N7000 Nexus 9000

6 Single-Tier, Dual-Tier, Spine/Leaf Scalable Spine/Leaf DC Fabric VXLAN Dual Tier DC VXLAN Single Layer DC Small Spine/Leaf

7 Connectivity & Usage Needs Drive Design Choices Form Factor Unified Computing Fabric 3 rd Party Blade Servers Rack Servers (Non-UCS Managed) Storage Protocols Fibre Channel FCoE IP (iscsi, NAS) Virtualization Requirements vswitch/dvs/ovs/nexus 1000v Automation/Orchestration Abstraction APIs/Programmability/Orchestration Connectivity Model 10 or 1-GigE Server ports NIC/HBA Interfaces per-server NIC Teaming models FC FCoE iscsi NFS/ CIFS VM VM VM VM VM VM 7

8 Data Center Fabric Needs North-South : end-users and external entities. East-West : clustered applications, workload mobility. High throughput, low latency Increasing high availability requirements. Automation & Orchestration Public Cloud Mobile API Orchestration/ Monitoring Internet DATA CENTER FABRIC Server/Compute Enterprise Network EAST WEST TRAFFIC Services Offsite DC Site B FC FCoE iscsi / NAS Storage NORTH - SOUTH TRAFFIC 8

9 Why Spine-Leaf Design? Pay as You Grow Model Need even more more host host ports? ports? Add Add another a leaf leaf Per Spine Utilization Per Spine Utilization To speed up flow completion times, add more backplane, spread load across more spines FCT FCT 40G fabric ports FCT FCT FCT FCT * FCT = Flow Completion Times 10G host ports ports ports 3x48 2x48 10G 10G 192 (1440 (960 ports Gbps Gbps total) 4x48 10G (1920 Gbps total) total) Lower FCT = FASTER APPLICATIONS

10 Spine/Leaf DC Fabric Large Non-Blocking Switch

11 Spine/Leaf DC Fabric Large Modular Switch Line Card Line Card 4 Line Card Fabric Module Line Card 1 5 Line Card Fabric Module Line Card Line Card Fabric Module Line Card Line Card Line Card

12 Impact of Link Speed the Drive Past 10G Links 200G Aggregate Bandwidth 20 10Gbps Uplinks 5 40Gbps Uplinks 2 100Gbps Uplinks 200G Aggregate Bandwidth 20 10Gbps Downlinks 40 & 100Gbps fabric provide very similar performance for fabric links 40G provides performance, link redundancy, and low cost with BiDi 20 10Gbps Downlinks 20 10Gbps Downlinks

13 40G BiDi Optics Preserve Existing 10G Cabling SFP-10G-SR $995 MMF LC Patch cord OM4 Fiber Plant Used Fiber Pair MMF LC Patch cord SFP-10G-SR $995 QSFP-40G-SR4 $2995 MPO OM4 Fiber Plant Used Fiber Pair Used Fiber Pair Used Fiber Pair Used Fiber Pair MPO QSFP-40G-SR4 $2995 QSFP-40G-SR-BD $1095 MMF LC Patch cord OM4 Fiber Plant Used Fiber Pair MMF LC Patch cord Distance <= 125m with OM4 QSFP-40G-SR-BD $1095

14 Single Layer Data Center, Nexus 5500 Dedicated Nexus 5500-based switch pair Positive Negative Unified Port on all ports Max Flexibility Can work as FC/FCOE access transition switch Non-blocking, Line Rate 10Gpbs L2 ~2us Latency Supports Fabric Path, DFA* 160G Layer-3 with L3 daughter card or GEM Supports 24 FEX, A- FEX, VM-FEX L3 card: 160G max, not cumulative DFA L2 ONLY Leaf No VXLAN HW support No ACI support No native DCI support ISSU not supported w/l3 FEX count lower w/l3 Q: 5500 or 5600? Nexus Gig/100M Servers Client Access 10-GigE UCS C-Series Campus WAN / DCI L L2 FC 10 or 1-Gig attached UCS C-Series FCoE iscsi / NAS Nexus 2000

15 Single Layer Data Center, Nexus 5600 Dedicated Nexus 5600-based switch pair Positive Negative Low Price/Performance Unified Ports Good Flexibility (not all ports) Supports VXLAN, Fabric Path, DFA Non-blocking, Line Rate L2/L3 Native 40G/10G, breakout ~1us Latency Supports 24 FEX, A-FEX, VM-FEX VXLAN support No ACI support No native DCI support ISSU not supported w/l3 Q: 5500 or 5600? Nexus Gig/100M Servers Client Access 10-GigE UCS C-Series Campus WAN / DCI L L2 FC 10 or 1-Gig attached UCS C-Series FCoE iscsi / NAS Nexus 2000

16 Single Layer Data Center, Nexus 5696Q (6004) Positioned for rapid scalability and a 40-GigE Fabric Positive Negative Client Access WAN / DCI Unified Ports Good Flexibility with expansion No VXLAN support in HW in early models Campus Non-disruptive scale-up No ACI support 96*40G or 384*10G Supports VXLAN, Fabric Path, DFA Non-blocking, Line Rate L2/L3 No native DCI support FEX count Lower w/l3 ISSU not supported w/l3 Higher initial cost Nexus 6004 L L2 FC FCoE iscsi / NAS Native 100G/40G/10G, BiDi, breakout support ~1us Latency Supports 48 L2 FEX, 24 L3 FEX, A-FEX, VM-FEX 1Gig/100M Servers 10-GigE UCS C-Series Nexus or 1-Gig attached UCS C-Series

17 Single Layer Data Center, Nexus 9300 Dedicated Nexus 9300-based switch pair Positive Negative Low Price/Performance No Unified Ports Client Access Campus WAN / DCI VXLAN Support in HW FCoE will require SW ACI Leaf & Spine support Non-blocking, Line Rate L2/L3 Native 40G & 10G <1us Latency FEX Support - 16 FCoE Hardware Support* No FP, DFA support VXLAN Control plane is Mcast until EVPN No native DCI support Breakout on some 40G ports ACI Spine <> ACI Leaf Nexus Gig/100M Servers 10-GigE UCS C-Series L L2 10 or 1-Gig attached UCS C-Series iscsi / NAS Nexus 2000

18 Single Layer Data Center, Nexus 7000 Highly Available Virtualized Chassis Access/Aggregation Model Positive More feature rich platform Modular, easy scale up Flexible L2/L3 with ISSU LISP*, OTV, FEX, FCoE, FP, VXLAN* Native 100G, 40G & 10G, breakout DFA Spine/Leaf Supports 32 FEX Negative Higher initial capital cost No Unified Ports VXLAN support in Future No ACI Support Physical Footprint Nexus 7000 Client Access Campus WAN / DCI L L2 FCoE iscsi / NAS VDC, PBR, WCCP, MACSec 1Gig/100M Servers 10-GigE UCS C-Series 10 or 1-Gig attached UCS C-Series Nexus 2000

19 Single Layer Data Center, Nexus 9500 Highly Available Chassis Access/Aggregation Model Positive Negative Client Access Modular, easy scale up Higher initial capital cost WAN / DCI Flexible L2/L3 with ISSU* FEX*, FCoE*, VXLAN* Native 100G, 40G & 10G, breakout Supports 32 FEX* No Unified Ports FEX, VXLAN, FCoE support in future No DFA, FP Support ISSU coming in future Nexus 9500 Campus L L2 ACI Spine/Leaf Support* VDC in future No native DCI iscsi / NAS 10 or 1-Gig attached UCS C-Series

20 Scaling with Spine/Leaf: 24x40G 36x40G 48x40G 60x40G 72x40G fabric ports needed for non-oversubscribed 72x40G available 40G fabric ports 10G host ports*** Three Four Five Six Two Racks, 288x10G 240x10G 192x10G 144x10G 96x10G ports (2880GB) (960GB) (2400GB) (1920GB) (1440GB) *** This example is 100% non-blocking, non-oversubscribed. Could build an oversubscribed model with FEX or fewer fabric links. Server/Rack density dependent on load, power, cooling (geo-diverse)

21 When do you add/upgrade spines? 72x40G 96x40G fabric ports needed for non-oversubscribed 144x40G now 144x40G 72x40G available, available smaller failure impact 40G fabric ports 10G host ports*** Eight Six Racks, 288x10G 384x10G ports (2880GB) (3840GB) *** This example is 100% non-blocking, non-oversubscribed. Could build an oversubscribed model with FEX or fewer fabric links. Server/Rack density dependent on load, power, cooling (geo-diverse)

22 When do you add/upgrade spines? 96x40G fabric ports needed for non-oversubscribed 2x36 in each modular spine, 140x40G 280x40G, available LC Redundancy, Spine ISSU, etc. 40G fabric ports 10G host ports*** Eight Racks, 384x10G ports (3840GB)

23 Scaling a VPC-based DC design Access Layer VLANs L3 L2

24 Scaling a VPC-based DC design DC Core Layer Access Layer VLANs L3 L2 Access Layer VLANs

25 Integrating Spine/Leaf with an existing network Core Layer ACI Pod New DC Data Row Upgrade New Application Spine Layer Agg Layer L3 L2 ACI Fabric (VXLAN based) Access Layer VLANs Access Layer VLANS Access Layer ACI Border VLAN Leafs

26 Integrating Spine/Leaf with an existing network Core Layer ACI Pod New DC Data Row Upgrade New Application Spine Layer Agg Layer L3 L2 ACI Fabric (VXLAN based) Access Layer VLANs Access Layer VLANs ACI ACI Border Leafs Leafs and Border Leafs

27 Data Center Interconnect Options Options for L2 Interconnect ASR1000 Client Access WAN / DCI Client Access Campus WAN / DCI Campus L L2 ASR1000 N7K L L2 Virtual DC Services in Software VM VM VM VM VM VM VM VM VM VM VM VM Virtual DC Services in Software Virtualized Servers with Nexus 1000v, vpath, CSR 1000v CSR1000v Virtualized Servers with Nexus 1000v, vpath, CSR 1000v

28 Nexus Programmability Provisioning & Orchestration Nexus 7K Nexus 5K / 6K Nexus 9K Puppet/Chef Future Shipping Shipping PoAP Shipping Shipping Shipping OpenStack Shipping Shipping Shipping Protocols and Data Models Programmatic Interfaces XMPP Shipping Shipping Future LDAP Shipping Shipping Shipping NetConf/XML Shipping Shipping Shipping NXAPI (JSON/XML) Future Future Shipping YANG Future Future Future REST Future Future Shipping Native Python Shipping Shipping Shipping Integrated container Coming Future Shipping Guest Shell Future Future Shipping OnePK Future Shipping Roadmap OpenFlow Future Shipping Shipping OpFlex Future Future Future

29 Programming for Many Boxes Git Hub Repository

30 Programming Examples Here s an example that uses the NXAPI for the N9K. It can automate mundane configuration tasks: you launch it remotely (from your Mac/PC) and use it to get an inventory of the switch, configure new interfaces, etc: getting_started/nxapi_basics.py Here s another one that collects the output of several show commands and puts them together to create a super command whit nice NxOS-style formatting: showtrans.py There are a few others such as a CRC error check here:

31 Options for Spine-Leaf FabricPath FP hdr payload Fabric Encapsulation method Each Switch Managed Individually VXLAN hdr payload VXLAN Dynamic Fabric Automation (DFA) DCNM (CPoM) + Automation Managed as a SYSTEM Service Profiles Web for the network QoS QoS Outside (Tenant VRF) Filter SVC ACI Fabric Non-Blocking Penalty Free Overlay Application Centric Infrastructure (ACI) App QoS Filter Automation of Network & Services FP with CPoM for Simpler Admin APIC DB Policy-based Infrastructure Based on Application Modeling

32 UCS Manages Compute through Abstraction SAN SAN Connectivity Configuration LAN LAN Connectivity Configuration Service Profile Motherboard Firmware BIOS Configuration Adapter Firmware Boot Order RAID configuration Maintenance Policy 32

33 ACI Manages Communications through Abstraction Network Path Forwarding External Connectivity ACL QoS FW Configuration SLB Configuration Connectivity ACL QoS SLB Configuration Connectivity ACL QoS QoS QoS QoS FW Configuration Connectivity Application Network Profile 33

34 Cisco UCS Director for Compute & storage Consumption made easier Secure Cloud Container Converged Stack Control Panel Network Compute VMs Policy-Driven Provisioning Storage Single Pane of Glass for Virtual AND Physical Speed & Accuracy OS and Virtual Machines Compute VM VM Bare Metal Virtualized and Bare-Metal Compute and Hypervisor UCS Director More efficient use of People & Time Network A B C Network and Services Consistency, Less Error in Repetitive Tasks Storage Tenant A Tenant B Tenant C

35 Cisco InterCloud Architectural Details UCSD Administrator installs InterCloud Director SP Admin deploys ICPEP End Users IT Admins VM Manager InterCloud Director Installed and configured through InterCloud Director InterCloud Provider Enablement Platform Cisco Global InterCloud (or Partner White-Label) VM VM Cisco Global Intercloud Services VM VM InterCloud Secure Fabric Private InterCloud Extender InterCloud Switch InterCloud Services Public

36 InterCloud Components InterCloud Director UCSD-based, separate interface InterCloud Secure Fabric N1Kv-based, doesn t require a full N1Kv install vnic from intercloud connecter into the vswitch Optional services integration with CSR1000v InterCloud Provider Enablement Platform ICF-Provider Edition implemented by provider to provide

37 Key Takeaways Cisco has many options for building DC solutions All solutions can start small and grow Does not have to be a rip and replace Spine-Leaf does not have to be expensive Automated fabrics can provide new tools for simplified operations Cloud technologies can expose new operational models

38 Want more? Cisco Live sessions have been recorded and the slides are available on Here are some related sessions: BRKDCT Advanced - Building simplified, automated and scalable DataCenter network with Overlays (VXLAN/FabricPath) BRKDCT Intermediate - End-to-End Application-Centric Infrastructure (ACI) Automation with UCS Director BRKDCT Intermediate - Real World Data Center Deployments and Best Practice Session BRKDCT Intermediate - VPC Best Practices and Design on NX OS BRKDCT Intermediate - Evolution of Network Overlays in Data Center Clouds BRKDCT Intermediate - VXLAN deployment models - A practical perspective BRKDCT Intermediate - Overlay Transport Virtualization BRKAPP Intermediate - Data Center Mobility, VXLAN & ACI Fabric Architecture BRKACI Intermediate - Application Virtual Switch for Application Centric Infrastructure Overview BRKSEC Intermediate - Deploying Security in ACI BRKACI Intermediate - Integration and Interoperation of existing Nexus networks into an ACI architecture BRKACI Intermediate - Integration of Hypervisors and L4-7 Services into an ACI Fabric BRKVIR Intermediate - End-to-End Application-Centric Data Center 38

39

40