Technical Design Guide Data Center Networking: the Mesh & POD architecture

Transcription

1 Technical Design Guide Data Center Networking: the Mesh & POD architecture Release 1.1 Abstract This paper is to provide a general guidance on using Alcatel-Lucent s POD and MESH architecture to create very high speed, reliable and flexible Data Centre and large Enterprise Core Networks. With 40GBit and future 100GBit connections, Shortest Path Bridging (SPB), Virtual Chassis and loss-less Ethernet and thousands of wire-speed 10GBit server ports, the POD/MESH architecture will maximise throughput, minimise latency and provide optimal service for critical network operations.

2 TABLE OF CONTENTS Introduction... 4 About Alcatel-Lucent Enterprise... 4 Alcatel-Lucent Core Networking Equipment... 5 OmniSwitch 10K... 5 OmniSwitch OmniSwitch 6850E... 6 Application Fluent Network... 7 Shortest Path Bridging (SPB) MAC-in-MAC (SPB-M)... 9 Multi-Chassis Link Aggregation (MC-LAG)... 9 IS-IS Convergence...11 The POD/MESH Architecture An Overview...12 The POD st Gen POD OS6850E nd Gen POD OS The SUPERPOD...17 The MESH...18 Latency within the POD/MESH...19 POD Variations...20 POD Type POD Type POD Type POD Type POD Type POD Type POD Type POD Type The POD Physical vs Logical...31 SuperPod Configurations...32 End-to-End Latency...32 The SuperPod Physical vs Logical...33 MESH Configurations...34 References...37 Figures Figure 1: AFN Building Blocks... 7 Figure 2: MC-LAG Diagram...10 Figure 3: 8-Unit 1st Gen POD...13 Figure 4: 4-Unit 1st Gen POD...13 Figure 5: 1st Gen POD to Core...14 Figure 6: OS6900 POD...16 Figure 7: Superpod...17 Figure 8: MESH...18 Technical Design Guide Pod and Mesh Page 2

3 Figure 9: POD Figure 10: POD Figure 11: POD Figure 12: POD Figure 13: POD Figure 14: POD Figure 15: POD Figure 16: POD Figure 17: 2-Unit POD...31 Figure 18: 6-Unit POD...31 Figure 19: POD-MESH Connectivity...34 Figure 20: SuperPod-MESH Connectivity...35 Figure 21: Double SuperPod-MESH Configuration...35 Tables Table 1: Latency and Traffic Distribution...19 Table 2: POD Type Configurations...21 Table 3: POD Type Configurations...22 Table 4: POD Type Configurations...24 Table 5: POD Type Configurations...26 Table 6: Pod Type Configurations...27 Table 7: Pod Type Configurations...28 Table 8: Pod Type Configurations...29 Table 9: SuperPod Node Connections...33 Table 10: 10K MESH Port Numbers...34 Technical Design Guide Pod and Mesh Page 3

4 Introduction Data networking evolves continuously, with advanced features and technology being introduced at an astonishing rate. In addition the changing landscape of centralised Data Centres and the realisation of Enterprises, large and small, that they can take advantage of these technologies to increase efficiency and ultimately their bottom line, mean that the equipment, architecture and solutions be comprehensive, efficient and cost-effective. To that end, Alcatel-Lucent s vision of the Application Fluent Network (AFN) coupled with their POD and MESH core network design model, is working to deliver all of these requirements to enable their customers to achieve their goals. This document describes the POD and MESH architecture, its implementation today and the in-built future evolution that can be realised with the market leading, state-of-the-art OmniSwitch 10K and OmniSwitch 6900, ensuring that their networks perform optimally today, tomorrow and ongoing in the future. About Alcatel-Lucent Enterprise Alcatel-Lucent Enterprise is a world leader in communications and networking solutions for businesses of all sizes, serving more than 500,000 customers worldwide. A consumer-led revolution, inspired by the need for increased mobility and rich multimedia content along with the rapid adoption of smart phones and tablets, is driving organizations toward the necessity to change the conversation with customers and between employees by deploying communications services and network infrastructure that will transform the way people communicate. Alcatel-Lucent Enterprise solutions are designed and developed to help enterprise organizations make the shift from proprietary technologies to industry standards, such as IP and SIP, and shift to cloud-based architectures, while enhancing the user experience and lowering the total cost of ownership. Our vision, change the conversation, translates into the need for: - Collaborative Conversations: delivering a comprehensive evolution path from voice communications to multimedia conversations through a wide range of solutions spanning from hybrid telephony access to a full suite of unified communications and collaboration solutions, including solutions for visual collaboration across all devices; - Application Fluent Networks: maximizing the efficiency of the networks through effective monitoring and adjustment of resources, while simplifying the network architecture to deliver the imperatives of new multimedia applications. We bring comprehensive solutions in the area of converged campus networks, data center networks and enable the enterprise to speed the adoption of cloud services. Alcatel-Lucent Enterprise leverages Alcatel-Lucent s broad portfolio with its leading service provider offerings and selected third-party products to provide integrated, end-to-end solutions. A global team of business practice experts and service professionals, combined with a global partner ecosystem, meet the unique needs of our customers from small businesses to global companies, with tailored offers to accommodate the requirements of different market segments and industries. Technical Design Guide Pod and Mesh Page 4

5 Alcatel-Lucent Core Networking Equipment OmniSwitch 10K The Alcatel-Lucent OmniSwitch 10K Modular Ethernet LAN Chassis is the first of a new generation of modular, network adaptable LAN switching platforms. It is designed to deliver a high quality user experience through scalable, quality bandwidth that optimizes the performance of legacy, real-time, and multimedia applications in a secure networking environment. As the foundation of a simplified network architecture, it meets high availability demands, reduces layers, meets eco-sustainability goals, and eases deployment and administration of application converged networks. The OmniSwitch 10K provides high-density, non-blocking, lossless 5.12 terabits per second of switching performance that reduces network layers and provides the capacity and scalability for a variety of requirements, including enterprise networks and data centres. Hardware and software redundancy features, such as denial of service protection and Multi- Chassis Link Aggregation (MC-LAG), are built into the platform to provide transparent resiliency capabilities. These features allow a network to re-converge on a failure without impacting application performance, thereby ensuring network availability. In-service software upgrade capabilities further maximize uptime, while the platform s embedded policy engine improves application performance with automatic provisioning of quality of service (QoS) and security parameters. And Multi-protocol Label Switching (MPLS) and FCOE readiness ensure the OmniSwitch 10K will be a long-term component of any next generation network. With front-to-back cooling and a compact design, the OmniSwitch 10K form factor is optimized for the data centre. In addition, its front accessible and easily upgradeable components make it a suitable replacement for legacy LAN switches and installations constrained by limited space in an equipment rack. OmniSwitch 6900 The Alcatel-Lucent OmniSwitch 6900 Stackable LAN Switch series is the latest hardware platform in Alcatel-Lucent Enterprise s comprehensive network infrastructure portfolio, following the introduction of the OmniSwitch 10K and the OmniSwitch 6850E. The OmniSwitch 6900 represents the next step in realising the Alcatel-Lucent Enterprise s Application Fluent Network (AFN) vision for network infrastructure to the data centre and the LAN core. Alcatel-Lucent s AFN vision is based upon a resilient architecture with the network dynamically adapting to the applications, users and devices in order to provide a high quality user experience, as well as reducing operation costs and management complexity. The OmniSwitch 6900 series consists of two models: the OmniSwitch 6900-X40 and the OmniSwitch 6900-X20. They provide 10 Gigabit Ethernet connectivity with industryleading port density, switching capacity and low power consumption in a compact 1 rack unit (1RU) form factor. The OmniSwitch 6900-X40 can deliver up to 64 10GigE ports, while the OmniSwitch 6900-X20 can deliver up to GigE ports wire-rate non-blocking performance with unique high availability and resiliency features as demanded in data Technical Design Guide Pod and Mesh Page 5

6 centres and in the core of converged networks. The OmniSwitch 6900 series with up to two plug-in module bays is the most flexible 10GigE fixed configuration switch on the market. The OmniSwitch 6900 series provides a long term sustainable platform that will be able to support 40GigE connectivity as well as specialized features for the data center such as Data Center Bridging (DCB), Shortest Path Bridging (SPB) and Fibre Channel over Ethernet (FCoE) without any change out of hardware. OmniSwitch 6900 in the Data Center The OmniSwitch 6900 is an essential building block for the Alcatel-Lucent Mesh where it can be used to create the Alcatel-Lucent Pod and in small scale data centre deployments can also be used as a core switch interconnecting Pods to create the Mesh. The OmniSwitch 6900 enables applications to be managed as services providing the Virtual Network Profile (vnp). The vnp is an essential element in delivering virtual machine mobility, and it is agnostic to the application virtualization platform being used in the data centre. The OmniSwitch 6900 with MC-LAG today and with SPB as well as virtual chassis in the near future will be ideal to enable a hybrid cloud model converting the corporate data center into a multi-site private cloud. OmniSwitch 6900 for Converged Network LAN Core The OmniSwitch 6900 s industry-leading 10 Gigabit port density in a 1RU chassis can be deployed as either a small network s core switch or as a high capacity aggregation switch. The OmniSwitch 6900 with its combination of high port density and network virtualization capabilities with MC-LAG enables a resilient and simplified architecture for the core of a converged network. In addition, the support of Ethernet Ring Protection (ERP) makes it an ideal aggregation switch in campus rings. The OmniSwitch 6900 enables conversations to be managed in context providing a rich set of QoS controls, an essential element for the LAN core in delivering on the AFN vision for a high quality end user experience. OmniSwitch 6850E The new Alcatel-Lucent OmniSwitch 6850E Stackable LAN Switch is the latest layer-3 Ethernet LAN switch in the OmniSwitch portfolio. The OmniSwitch 6850E is specifically built to meet the requirements of the network edge for delivering the high quality user experience of an application fluent network. One of the key new features of the OmniSwitch 6850E is an architecture that can deliver up to 30W per port of Power over Ethernet (PoE+), providing sufficient power for devices required to deliver new voice, video and collaborative applications as well as the bandwidth, resiliency and security demanded by these real-time applications. The OmniSwitch 6850E is available in 24 and 48 port configurations, supports virtual routing and forwarding (VRF) and provides wire-rate L2 and L3 switching simultaneously for IPv4 and IPv6. The OmniSwitch 6850E provides fine grained controls for both quality of service (QoS) and security, including integrated network access control (NAC). Technical Design Guide Pod and Mesh Page 6

7 In addition, the OmniSwitch 6850E offers up to 4 10GigE SFP+ ports to be used as uplinks to a core switch and supports remote stacking, enabling resilient network ring architecture. The OmniSwitch 6850E stacks with the OmniSwitch 6850 allowing easy expansion of existing installations where PoE+ or SFP+ is needed. Application Fluent Network The demanding requirements of rich media applications and virtualization within the enterprise combined with the limitations of legacy networks are forcing enterprises to look for innovative ways to improve application performance while minimizing operating costs and capital investment. The application fluent network as defined by Alcatel-Lucent promises to transform the way enterprises deliver applications throughout an organization s global operations. It features a resilient architecture, streamlined operations, and automatic control, which together meet the challenges facing enterprise IT groups today. With tangible benefits such as a high-quality user experience, reduced demands on IT staff, and a faster time to ROI, the application fluent network represents market leading innovation in network design and operation one that Alcatel-Lucent is uniquely qualified to lead. Application fluency represents a unique approach to enterprise networking. In Alcatel- Lucent s view, the application fluent network possesses broad knowledge of both network devices and the applications to which they are connecting. Most importantly, the application fluent network understands the context of the conversation between device and application and makes decisions based on that understanding. Alcatel-Lucent s application fluent network is based upon a resilient architecture with streamlined operations and automatic control as shown in the following figure: Architecture Control Operations Figure 1: AFN Building Blocks Resilient Architecture: A simplified, single IP network with built-in security. Streamlined Operations: Reduced complexity through automated provisioning and integrated troubleshooting tools. Automatic Control: High-quality, real-time application delivery with unique dynamic tuning of network performance. Technical Design Guide Pod and Mesh Page 7

8 The application fluent network brings significant benefits to the enterprise including a highquality user experience, lower network administration costs, and a better return on investment (ROI). High-Quality User Experience The application fluent network allows network managers to offer a high-quality user experience with controls that automatically ensure service levels, avoid packet loss, and provide security such as: Dynamic tuning of network performance. The application fluent network automatically adapts to maintain service levels for individual conversations based upon context and business priorities. Automatic recovery from switch and link faults. The application fluent network recovers rapidly from single switch fabric and link failures, avoiding any perceptible service interruptions, even for voice and video. Embedded security. Security features in the application fluent network operate transparently to protect users and their conversations from threats. Lower Network Administration Costs The application fluent network eases the burden on IT staff with built-in features for automated provisioning and integrated management of the network including: Low-touch operations. The application fluent network manages provisioning of edge switches, wireless access points, and endpoints with minimal skilled human intervention. The use of network policies ensures consistent configurations across the entire network, reducing time spent tweaking the configuration of individual components. Converged management. In the application fluent network, a consolidated management tool provides a single pane of glass for monitoring and managing the network. It also allows administrators to manage the network infrastructure as a single fabric rather than individual components. IT staff spend less time switching between programs and screens, and more time on productive work. Better Return on Investment (ROI) The application fluent network both minimizes the initial capital investment and lowers operational costs, which allows the enterprise to recoup its investment faster through: Flatter network architecture. By eliminating an entire network layer, the application fluent network reduces the number of switches, links, and other costly network components. Network virtualization. The application fluent network virtualizes both switches and links, resulting in optimized utilization, lower capital requirements, and higher ROI. Green design. The new generation of energy-efficient high density switches used in the application fluent network reduces expenses for space, power, and cooling. Technical Design Guide Pod and Mesh Page 8

9 Shortest Path Bridging (SPB) MAC-in-MAC (SPB-M) The Spanning Tree Protocol (STP) has two main objectives: create a topology path to go from one node to the next, and in doing so, prevent loops on the network. It has been updated several times over the years and although was deemed to have its flaws it provided a (relatively) stable mechanism to achieve its goals. However the limitations of the STP protocol and its various reincarnations mean that it is not scalable for certain deployments. To overcome the well known limitations of STP, one solution is Shortest Path Bridging (802.1aq). Used in conjunction with IS-IS this allows the encapsulation (the MAC-in-MAC portion) of Ethernet frames to travel across a core network at L2. The customer MAC addresses are not learned by any of the node making up the core SPB network, and this additional header is removed at egress for delivery. The link state protocol IS-IS is used to discover and advertise the network topology and compute shortest path trees from all bridges in the SPB Region. The IEEE are standardising this protocol, which means that it is possible to use other existing protocol to enhance its usability aq builds on all existing Ethernet OA&M. Since 802.1aq ensures that its unicast and multicast packets for a given VLAN follow the same forward and reverse path and use completely standard 802 encapsulations, all of the methods of 802.1ag and Y.1731 operate unchanged on an 802.1aq network. Multi-Chassis Link Aggregation (MC-LAG) The Multi-chassis LAG (MC-LAG) feature is an extended version of LACP designed to provide resiliency at the edge by enabling dual homing of any standards-based LACP device to a pair of switches allowing Layer 2 multi-path. The goal is to support dual-homing without running Layer 2 redundancy protocols (e.g. Spanning Tree), while still ensuring preventing data loop conditions, failure detection and convergence. Benefits: 1. High-availability by providing node resiliency. 2. Provide dual-homed layer 2 multi-path connections for edge nodes in to the aggregation without running the spanning tree protocol. The edge device can be any LACP capable-device. Technical Design Guide Pod and Mesh Page 9

10 3. Fast fail-over detection and convergence time meeting sub-second performance (i.e. convergence time <= 1 second) for access uplink failures, virtual fabric link failures and node failures. 4. Deliver active/active forwarding mode whereby both sets of uplinks that are part of the dual homed aggregates are processing traffic to maximize the value of the customer investment. An important characteristic of this solution relates to the absence of logical loop between the edge and multi-chassis peer switches, even though a physical loop does exist. MC-LAG components: Figure 2: MC-LAG Diagram Multi-Chassis peer (mc-peer): Switches that terminate the aggregate links coming from edge switches. MCLAG: Multi-chassis link aggregate. An aggregate composed by multiple ports belonging to the mc-peers forming a single LAG (dual homed layer 2 multi-path). MCLAG port: Ports that are members of the dual-homed multi-chassis aggregate. Multi-Chassis Domain: virtual concept consisting on a set multi-chassis peer switches, the virtual fabric link and the MCLAG ports. Active Links: Links, members of the MCLAG, that are fully operational and may actively forward traffic. Virtual Fabric Link: Aggregate of high-speed ports, usually spanning multiple NI modules, used for inter-chassis traffic (feature designed as to minimize the flow of traffic) and control/state data transfer. Technical Design Guide Pod and Mesh Page 10

11 IS-IS Convergence Calculating the convergence time of IS-IS depends on a number of factors. The main variable is the number of participating nodes in the network the higher the number, the longer the convergence. In addition, the nature of the failure will impact on the reconvergence. It is not possible to give exact values to quote when describing a network solution. It will depend on the exact type and configuration of the proposed network. In the event of a failure, re-convergence is still very fast, fast enough that in a loss-less Ethernet network, the impact will not adversely affect the flow of traffic. This is obviously the goal of the SPB/lossless solution! In a scenario where multiple ECTs (Equal Cost Trees) are being used, traffic flowing through the network that does not use the faulted link/node, will continue to work while those paths that are affected will be being re-converged. Technical Design Guide Pod and Mesh Page 11

12 The POD/MESH Architecture An Overview The Alcatel-Lucent POD and Alcatel-Lucent MESH form part of the fundamental building blocks of the AFN. Ultimately, the Alcatel-Lucent POD will consist of a number of meshed units, together seen as one logical entity. The POD will be connected to other PODs or indeed to the Core OS10K. A fully meshed 6-unit OS6900 POD will deliver up to 240 server ports with <2µsec latency, using a 1-hop SBP link. The next step is interconnecting multiple PODS. 5 PODs can create a POD of PODs (or superpod ) enabling an end-to-end port latency of <4µsec. The last step is to consider connecting multiple SUPERPODS to an OS10K Virtual Chassis. This creates a full Alcatel-Lucent AFN MESH. As of today, there is the option to create PODs using OS6850E, OS6900 and OS10K. Features needed to create the ultimate vision of 5-unit Virtual Chassis will be available in future releases. The following sections cover what is available today, what will be ready in the next release of the software, and what will be the final goal of the AFN MESH vision. Technical Design Guide Pod and Mesh Page 12

13 The POD This term covers the logical entity consisting of several units connected together and considered a single management entity. The management of a POD depends on the type of equipment used to create it, and the networking architecture being used. There are two generic families of POD: OS6850E and OS6900. The 1 st generation consists of OS6850E, stacked together using traditional stacking technology (described later in this document). Up to 8 units can be joined together providing up to 384 Gig ports in a pod. The following figures show two types of 1 st Gen POD: Figure 3: 8-Unit 1st Gen POD Figure 4: 4-Unit 1st Gen POD The 2 nd, and subsequent, generations of the POD will consist of meshed OS6900, using SPB for inter-unit connectivity. The 1 st release of the OS6900 will allow a fully meshed 6-unit POD using SPB. Connectivity to the core will flow via aggregated 10Gb connections to the OS10K using MC-LAG. The second generation of POD introduces Virtual Chassis with the long term aim of creating a POD of any size within a VC configuraton. Technical Design Guide Pod and Mesh Page 13

14 1 st Gen POD OS6850E The concept of the POD can be constructed using the OS6850E. The ideas behind meshing units to reduce latency and to centralise switching by reducing the number of layers needed to create the network architecture, can be implemented today. Using the existing technology of a stack configuration, egressing the stack is by default via a shortest path algorithm. Particular attention should be taken when connecting the stack via MC-LAG to an OS10K, when these uplinks are themselves in a LAG grouping. The following diagram shows the OS6850E in a POD configuration: Figure 5: 1st Gen POD to Core On the left hand side is a POD of OS6850Es. As can be seen, this is a ring architecture with 8 units. The simplest implementation of this architecture is using the OS6850E s stacking feature. In this case, the name POD replaces stack. This gives some advantages: Management: The POD is essentially a Virtual Chassis meaning that the entire construct is managed as a single entity. This reduces management overhead, as the POD is managed via a single IP address and can be controlled via the Master/Secondary unit feature giving resiliency to the POD. Physical location versatility: Using the additional SFP+ ports on the optional expansion slot give the possibility to physically locate the switches at some distance from each other. The units no longer need to be sitting on top of each other. Of course, the distance between units is still restricted to the length of the available cables, and therefore machine room design need to take this into account. Technical Design Guide Pod and Mesh Page 14

15 Low Cost: The OS6850E is not only an upgrade of proven technology but allows an entry level option to take advantage of the AFN. Installed base: In there is a substantial install base of this product meaning that customers that have already invested in this model can take advantage of this type of architecture. Connection to other PODs or the Core would be done via redundant 10Gbit uplinks. Multiple links can be aggregated together giving more bandwidth. Dual Homed Links give the possibility to connect to multiple core switches. Using this type of approach leverages one of the goals of the AFN; to eliminate a layer within the architecture. The traditional 3-tier architecture no longer applies as the middle layer the Distribution Layer or Aggregation Layer - can be safely eliminated. The bandwidth capacity of the individual OS6850E coupled with the huge capacity of the OS10K means that it is no longer necessary to have an intermediate layer to redistribute links to the edge. The number of units within the POD/Stack is not limited or required to be 8. Stack numbers can range anywhere from 2 to 8 units. When creating the POD concept with the 6850E, attention needs to be taken when deciding on where the uplinks leaving the stack to the core, be placed. The 6850E stacking algorithm uses a shortest path manner to send packets to the uplinked unit. Therefore, minimising the number of units traversed will increase efficiency of the stack. Technical Design Guide Pod and Mesh Page 15

16 2 nd Gen POD OS6900 With the new OS6900, many additional features included in the next generation of switches helps lower latency and greatly increase bandwidth capacity within the switch and over the network. Using advanced protocols and techniques, it will also be possible to converge the LAN and SAN over the same equipment. The first implementation of the POD using the OS6900 will consist of 6 stand-alone units, interconnected with 40Gbit links using SPB (M). The only drawback of this configuration is that managing the units is slightly more expensive in terms of man power, but the advantages of using this particular switch with the protocols and bandwidth available, far outweigh any potential downside. Figure 6: OS6900 POD In the above diagram, the inter-switch connections are 40Gbit and run SPB. Meshing the units in this way ensures a 1-hop topology to connect to any other unit within the POD. The next iteration of the OS6900 POD will use the Virtual Chassis feature. This will give a POD consisting of 3 x 2-unit Virtual Chassis, reducing the administration overhead by a third. The long-term goal is to increase the number of members within the VC to accommodate a VC-POD, or Virtual Chassis POD. Using the OS-QNI-U3 (3 x 40GB ports) in both expansion slots allows the connection of up to 5 OS6900, leaving one free 40GB port for uplink. This gives the shortest path to any of the other unit within the POD using SPM(M). One POD can deliver 240 x 10GB wire-speed ports, between servers and switches. Technical Design Guide Pod and Mesh Page 16

17 The SUPERPOD The following diagram shows a 5-POD SUPERPOD of OS6900s in a fully-meshed configuration: Figure 7: Superpod The Super Pod regroups multiple PODs together before connecting to the core. Here it is possible to have traffic flow between different destinations without going through the core, keeping the intra-pod and intra-superpod traffic local to the Superpod entity. In this configuration the shortest path between any two units anywhere in the Superpod is not more than 3 hops. The majority of potential paths contain 2 hops. Using this configuration gives an end-to-end latency of < 4 µsec. The Superpod relies on two networking technologies: SPB and 40 Gigabit Ethernet. These will be released by Alcatel-Lucent during the middle of when it will be possible to realise the full potential of the Pod and Mesh architecture. Using the 40Gbit connection on 4 of the OS6900s within each POD, it is possible to interconnect the different PODs together in such a way as to keep the number of units on the path between each switch down to a minimum. 1 Subject to change Technical Design Guide Pod and Mesh Page 17

18 As is shown in the above diagram, 2 OS6900s are not used in the super pod architecture, giving each POD a connection out of the Superpod towards the Core OS10K. The MESH The following diagram shows a Superpod/Core MESH of OS6900s and OS10K: Figure 8: MESH Following on from the previous Superpod example, the remaining 40Gbit connections on the OS6900s are connected to the OS10K, one connection from each SUPPERPOD going to one OS10K and a second going to the other OS10K. This configuration maximises the remaining 40Gbit links, ensuring that the shortest path between any two ports is kept to a minimum to increase speed and reduce latency on the network. The above example does not use MC-LAG, and instead will use SPB between all of the switches used in the network architecture (OS OS10K). Technical Design Guide Pod and Mesh Page 18

19 Latency within the POD/MESH At the heart of the POD/MESH design is the reduction of latency. Within the Data Centre ecosystem there is a very low tolerance for delay indeed the very fact of having any sort of reduction in communication times can render the entire network unusable. The POD/MESH design attempts to reduce the latency by optimising the various interconnections delivering packets as quickly as possible to as many of the nodes as possible. In some of the POD variations not all units are fully meshed. Therefore by placing the interconnections in a certain way will deliver a low latency to as many as the units as possible. Overall latency numbers used in the POD variations consider an aggregate of traffic within a POD, within a Superpod and between Superpods. The following table shows the different traffic distribution used in these calculations: Traffic Distribution Required BW Oversubscription Latency POD E-W Traffic 70% <2usecs Inter POD E-W Traffic 20% <4usecs Inter POD of PODs E-W Traffic 10% <24usecs Table 1: Latency and Traffic Distribution Technical Design Guide Pod and Mesh Page 19

20 POD Variations The logical concept of a POD is to introduce a topology that sees each member of a particular layer as a single entity. Features used to achieve this are not yet released 2 (Virtual Chassis), but the simple idea of having a topology that delivers a shortest path network can be realised today. The POD has been conceived to deliver a resilient and fast network. Interconnecting every member to every other member is the best optimised configuration. However, it is possible to connect many more OS6900s together, and while not delivering a 100% 1- hop topology, can still provide a solution that increases the total number of interconnected nodes (be they server or network nodes) while maintaining low latency, multi-path and resilient networking topologies. PODs containing less than 6 will still be fully meshed giving the 1- hop topology delivering the lowest latency (assuming that all available 40Gbit links are used). The following POD variations use the following nomenclature: POD Type x.y.z Where: X: number of PODs (0 designates a POD itself). Y: Number of OS6900 in a single POD Z: The maximum latency within the POD Examples: A single POD, with 6 units with a maximum latency of 2 microseconds A SuperPod of 5 PODs, each POD has 2 units with a latency of 2 microseconds within a POD There are no hard and fast rules about POD naming and it is intended to differentiate the different configurations within the document. Different POD variations are detailed on the following pages. 2 As of Q Technical Design Guide Pod and Mesh Page 20

21 POD Type The following is a 2-unit POD with 2 microsecond latency: Figure 9: POD Using the features to be released in the near future, this POD will also take advantage of being a Virtual Chassis. An added advantage here is that the units can be connected using the hybrid expansion card with 2 x 40Gbit ports and 4 x 10Gbit ports. This reduces cost and still provides 2 additional 40Gbit ports per switch for uplink to the core. In addition, 8 x 10Gbit ports increases the number of server ports to 48 per switch. Three types of POD using the two OS6900 models in conjunction with the available expansion cards can be created depending on port density requirements. The following table shows the equipment, port types and numbers that can be achieved: Low Density Configuration Switch Type Expansion No. 40Gbit Ports per switch per switch per POD OS x OS-QNI- 3 (1 per switch for uplink) X20 U3 Medium Density Configuration Switch Type Expansion No. 40Gbit Ports per switch per switch per POD OS x OS-QNI- 3 (1 per switch for uplink) X40 U3 High Density Configuration Switch Type Expansion No. 40Gbit Ports per switch per switch per POD OS x OS-QNI- 4 (2 per switch for uplink) X40 U6 Table 2: POD Type Configurations Technical Design Guide Pod and Mesh Page 21

22 POD Type The following is a 4-unit POD with 2 microsecond latency: Figure 10: POD As with the previous example, the Virtual Chassis concept can reduce administration overhead with 2 x 2-unit VCs connected in a meshed configuration. As with the POD the hybrid expansion cards can be used for inter-switch connections. This will leave 1 x 40Gbit port per switch for eventual uplink or other usage per switch. Each switch will also have 48 x 10Gbit ports. The following table shows the equipment, port types and numbers that can be achieved: Medium Server-Port-Density Configuration Switch Type Expansion No. 40Gbit Ports per switch per switch per POD OS x OS-QNI- 6 (3 per switch for uplink) X40 U3 High Server-Port-Density Configuration(1) Switch Type Expansion No. 40Gbit Ports per switch per switch per POD OS x OS-QNI- 4 (1 per switch for uplink) X40 U6 High Server-Port-Density Configuration(2) Switch Type Expansion No. 40Gbit Ports per switch per switch per POD OS x OS-QNI- 5 (2 per switch for uplink) X40 U3 1 x OS-QNI- U6 Table 3: POD Type Configurations Technical Design Guide Pod and Mesh Page 22

23 Technical Design Guide Pod and Mesh Page 23

24 POD Type A variation on the standard meshed POD is a 6-unit POD which is not fully meshed. This will provide 2 microsecond connectivity for half the units when these units are the source of the packet, and 3 microsecond connectivity towards the remaining 2. Figure 11: POD The following table shows the equipment, port types and numbers that can be achieved: Medium Server-Port-Density Configuration Switch Type Expansion No. 40Gbit Ports per switch per switch per POD OS x OS-QNI- 6 (3 per switch for uplink) X40 U3 High Server-Port-Density Configuration(1) Switch Type Expansion No. 40Gbit Ports per switch per switch per POD OS x OS-QNI- 4 (1 per switch for uplink) X40 U6 High Server-Port-Density Configuration(2) Switch Type Expansion No. 40Gbit Ports per switch per switch per POD OS x OS-QNI- 5 (2 per switch for uplink) X40 U3 1 x OS-QNI- U6 Table 4: POD Type Configurations Technical Design Guide Pod and Mesh Page 24

25 POD Type The highest density POD which can be fully meshed is the POD Type detailed earlier in this document. In order to allow for sufficient uplinks to connect to other PODs or the Core, each switch requires a spare 40Gbit port to be available. However, it is possible to optimise the configuration to reflect an individual customer s needs. In theory each switch within this type of POD could have an uplink to the core, but this is not necessarily mandatory. In the event that this topology can be deployed without any adverse impact to the network, it is possible to increase the number of server ports within the POD. Figure 12: POD The following table shows some possible configurations: Fully Meshed-Uplink Configuration Switch Type Expansion No. 40Gbit Ports per switch per switch per POD OS6900-X40 2 x OS-QNI- 6 (1 per switch for uplink) U3 2-Switch-Uplink Meshed Configuration Switch Type Expansion No. 40Gbit Ports per switch per switch per POD 2 x OS x OS-QNI- 6 (1 per switch for uplink) 40 X40 U3 4 x OS x OS-QNI- 5 (no uplink) 44 X40 U x OS-QNI- U6 Technical Design Guide Pod and Mesh Page 25

26 Table 5: POD Type Configurations Technical Design Guide Pod and Mesh Page 26

27 POD Type Increasing the number of units in a POD to more than 6 automatically eliminates the option of the POD being fully meshed. However, there are still options for interconnecting them in such a way as to reduce the impact of the loss of 1 hop connectivity while increasing the number of server ports served by the POD. The following example shows an 8-unit POD with 2 microsecond connectivity from 3 of the units and 3 microseconds from the remaining 4 units (the 8 th unit being the source of the packet). Figure 13: POD Medium Server-Port-Density Configuration Switch Type Expansion No. 40Gbit Ports per switch per switch per POD OS x OS-QNI- 6 (3 per switch for uplink) X40 U3 High Server-Port-Density Configuration(1) Switch Type Expansion No. 40Gbit Ports per switch per switch per POD OS x OS-QNI- 4 (1 per switch for uplink) X40 U6 High Server-Port-Density Configuration(2) Switch Type Expansion No. 40Gbit Ports per switch per switch per POD OS x OS-QNI- 5 (2 per switch for uplink) X40 U3 1 x OS-QNI- U6 Table 6: Pod Type Configurations Technical Design Guide Pod and Mesh Page 27

28 POD Type An alternative to the above example, if the requirement to keep latency down to an absolute minimum while keeping the greater number of server ports available within the POD, is to interconnect every second switch with an additional link. This will deliver 2 microsecond connectivity to 5 of the units and 3 from 2 of the units (the 8 th unit being the source). Figure 14: POD Medium Server-Port-Density Configuration Switch Type Expansion No. 40Gbit Ports per switch per switch per POD OS6900-X40 2 x OS-QNI- 6 (1 or 3 per switch for U3 uplink) High Server-Port-Density Configuration Switch Type Expansion No. 40Gbit Ports per switch per switch per POD 4 x OS x OS-QNI- 6 (1 per switch for uplink) 40 X40 U3 4 x OS x OS-QNI- 4 (1 per switch for uplink) X40 U6 Table 7: Pod Type Configurations Technical Design Guide Pod and Mesh Page 28

29 POD Type The POD can be increased in size to accommodate many more server ports. example, 12 OS6900s are used to create a POD serving up to Gbit ports: In this Figure 15: POD Using this configuration, packets arriving on 6 of the nodes (ones directly connected to 5 other nodes) have 2 microsecond latency to these nodes, while having 3 microseconds to the remaining 6 nodes. The other 6 nodes will have 2 microsecond latency to 3 nodes, 3 microseconds to 6 nodes and 4 microseconds to the remaining 2 nodes. Medium Server-Port-Density Configuration Switch Type Expansion No. 40Gbit Ports per switch per switch per POD OS6900-X40 2 x OS-QNI- 6 (1 or 3 per switch for U3 uplink) High Server-Port-Density Configuration Switch Type Expansion No. 40Gbit Ports per switch per switch per POD 6 x OS x OS-QNI- 6 (1 per switch for uplink) 40 X40 U3 6 x OS x OS-QNI- 4 (1 per switch for uplink) X40 U6 Table 8: Pod Type Configurations Technical Design Guide Pod and Mesh Page 29

30 POD Type The 12-unit POD can be constructed to further reduce the overall maximum latency by adding a second ring of links between the switches: Figure 16: POD Packets soured from 5 of the units have a 1 hop path, while they have a 2 hop path towards the remaining 6 units. Medium Server-Port-Density Configuration Switch Type Expansion No. 40Gbit Ports per switch per switch per POD OS6900-X40 2 x OS-QNI- U3 6 (1 per switch for uplink) Technical Design Guide Pod and Mesh Page 30

31 The POD Physical vs Logical The description of a POD has so far been about its logical description. It can sometimes help to show how a POD fits in with a customer s network or how it can be represented physically. The following example shows the 2-unit POD: Figure 17: 2-Unit POD In the above example, 2 x OS6900s are connecting server racks. Connections to the rest of the network would be from both switches up to the Core. Figure 18: 6-Unit POD In this example, the highest-density fully-meshed POD is represented. Using the SPB protocol will ensure shorted path communication between the switches for end-to-end connectivity, while allowing up to 6 different paths from the POD to either the Core or another POD. Technical Design Guide Pod and Mesh Page 31

32 SuperPod Configurations A SuperPod is a collection of PODs that have been connected together forming an entity allowing traffic to flow East-West without going through the Core. As with the definition of a Pod, the SuperPod is not a unique configuration and should be considered to be an interconnected entity containing multiple Pods. With the OS6900, a maximum of 5 PODs can be connected in such a way as to create a meshed entity allowing as low latency as possible. SuperPods containing few PODs will also follow this same pattern. In order to connect more than 5 PODs in this way, the interconnection must be such that the idea of minimising the number of nodes traversed be as few as possible for the greatest number of nodes possible. The example give on page 17 of a 5 POD configuration is reproduced below: The choice of where to place the interconnections is such that each POD communicates with every other POD, keeping the hops down to a minimum. The maximum number of nodes comprising an end-to-end path containing different PODs will be 4 (see below). In addition to the placements of the inter-pod connections, the actual traffic load and type should be known and taken into account before deciding on the actual SuperPod configuration to ensure that traffic is using an optimal configuration. It is imperative that no artificial bottlenecks are created. End-to-End Latency The diagram to the left shows the maximum number of nodes to be traversed in a meshed SuperPod. It shows 1 hop per POD and 1 inter-pod hop giving a latency of 4 microseconds. Servers attached to nodes that make up the inter-pod connection have an end-to-end latency of that of intra-pod communications (2 nodes, 2 microseconds latency). This can be exploited when deciding on where to place servers that need to reside in different PODs but still require lower latency for their communication. Technical Design Guide Pod and Mesh Page 32

33 The SuperPod Physical vs Logical Here is a simple network diagram illustrating a SuperPod. Each POD has been numbered 1-5, and each node numbered 1-6. This gives the opportunity to create a nomenclature to identify each connection using the POD and Node number. While this example may seem overly complicated it represents the largest meshed configuration possible with this architecture, giving over a thousand 10Gig server ports with a maximum of 4microsecond latency: To keep the example simple, only the connections from POD 1 have been shown. This example is based directly on the logical diagram shown on page 31. The following table contains all of the connections that would make up the meshed SuperPod. POD 1 POD 2 POD 3 POD 4 POD 5 From To From To From To From To From To Node 1 CORE Node 1 CORE Node 1 CORE Node 1 CORE Node 1 CORE Node 2 CORE Node 2 CORE Node 2 CORE Node 2 CORE Node 2 CORE Node 3 P2N6 Node 3 P3N6 Node 3 P4N6 Node 3 P5N6 Node 3 P1N6 Node 4 P3N5 Node 4 P4N5 Node 4 P5N5 Node 4 P1N5 Node 4 P2N5 Node 5 P4N4 Node 5 P5N4 Node 5 P1N4 Node 5 P2N4 Node 5 P3N4 Node 6 P5N3 Node 6 P1N3 Node 6 P2N3 Node 6 P3N3 Node 6 P4N3 Notes: P2N6 = POD 2, NODE 6 Table 9: SuperPod Node Connections Technical Design Guide Pod and Mesh Page 33

34 MESH Configurations The evolution of the POD/MESH architecture arrives at the core with the MESH topology. The OmniSwitch 10K can support 256 x 10Gbit port and 64 x 40Gbit ports. In Core network configurations it is advisable to introduce a certain level of redundancy. Using 2 x 10K chassis in an MC-LAG (2 unit) or VC (Virtual Chassis) configuration gives additional throughput, redundancy and options for configuring the network. While the maximum number of ports that can be used in a 10K chassis is very high, it must be taken into account that inter-chassis VFL links will require that one slot in each chassis be dedicated for inter-chassis communication. With this in mind, the number of network ports available for the MESH are given in the following table: Port Speed Number per chassis available 10Gbit Gbit 60+ (inter-chassis link to be included) Table 10: 10K MESH Port Numbers Just as the POD can have different configurations, the MESH can also have different number of nodes, and is as flexible. Allowing for inter-chassis communication and other requirements for port types/speed, the following diagram describes POD-Core connectivity: Figure 19: POD-MESH Connectivity In this configuration, a theoretical maximum of 30 6-unit PODs can be connected to a 2-unit 10K Core (3 x 40Gbit per POD to each 10K). Technical Design Guide Pod and Mesh Page 34

35 Connecting the SuperPod to the MESH, while slightly increasing oversubscription, allows for a higher density configuration: Figure 20: SuperPod-MESH Connectivity Each SuperPod will have 5 x 40Gbit connections to each 10K, allowing for 12 SuperPods to be connected to a 2-Unit 10K Core. Following on from this, and applying the logic that has been used through-out this document, we can increase the number of Core OS10Ks to accommodate a larger configuration: Figure 21: Double SuperPod-MESH Configuration The above diagram is used to illustrate the potential of the OS10K and 6900 when meshed together, leveraging the high throughput to interconnect many hundreds of nodes. Technical Design Guide Pod and Mesh Page 35

36 The type and number of nodes within a SuperPod can change, depending on the nature of the deployment and the associated needs. The PODs themselves can decrease or increase in size, and the interconnectivity can be adjusted to suit the needs of the customer. Technical Design Guide Pod and Mesh Page 36

37 References The following links can be accessed on the ALU Intranet: OmniSwitch 10K Pre-Sales resources: OmniSwitch 6900 Pre-Sales resources: AOS Boilerplate Wiki: More information, including various discussions on Data Centre and its associated subtopics can be found on the following Engage sites: Network Infrastructure: OmniSwitch and AOS: Alcatel, Lucent, Alcatel-Lucent and the Alcatel-Lucent logo are trademarks of Alcatel-Lucent. All other trademarks are the property of their respective owners. The information presented is subject to change without notice. Alcatel-Lucent assumes no responsibility for inaccuracies contained herein. Copyright 2012 Alcatel-Lucent. All rights reserved (February)R1.1 Technical Design Guide Pod and Mesh Page 37