1 STATE OF THE ART OF DATA CENTRE NETWORK TECHNOLOGIES CASE: COMPARISON BETWEEN ETHERNET FABRIC SOLUTIONS Supervisor: Prof. Jukka Manner Instructor: Lic.Sc. (Tech) Markus Peuhkuri Francesco Maestrelli 17 November 2011
2 Agenda Data Centre to Finland (DC2F) Master s thesis goals A three-tier network Typical technologies Issues in modern data centres Modern two-tier networks Ethernet fabrics Conclusions Future work
3 Data Centre to Finland (DC2F) In the recent years Finland has emerged as a strategic location where to build new data centres. Energy efficiency is the common driver of all these new installations. DC2F is a research project at Aalto University involving three different departments: Department of Communications and Networking Department of Electrical Engineering Department of Civil and Structural Engineering Comnet s goal is to study the impact of the network and the networking equipment on the energy consumption of a data centre.
4 Master s thesis goals Explore the available solutions in the data centre field to set the basis for the realization of a test data centre which is currently being realized. Understand what is the state of the art of data centre networks Standard solutions Typical problems faced by network designers Trends and future technologies Compare different Ethernet fabric solutions to better understand whether they differ and how.
5 A three-tier network Typically data centres present a subdivision in three tiers Core Aggregation Access Services are typically provided at the Aggregation layer using service modules or dedicated servers. Looped topologies are used to allow redundancy of the connections. Bandwidth oversubscription between layers.
6 Typical technologies Link Aggregation Group: allows bundling together up to 16 links of the same capacity into a logical one. M-LAG: Allows the terminating a LAG on two separate physical devices. Virtual Chassis: Allows linking together two or more switches so that they can act as a single one. Virtual Chassis by Juniper Cisco Fabric extenders Nexus 2000 series Virtualization of data centre resources. Servers are typically virtualized so to maximize the utilization of the physical servers rearranging as needed the VMs. Virtual Ethernet Port Aggregator (VEPA) Port Profile Migration
7 Issues in modern data centres STP is inadequate Halves the network capacity, Too high re-convergence time upon failure, MSTP is too cumbersome to maintain, Non-optimal forwarding. Virtual Machine mobility imposes L2 adjacency IP and MAC addresses should remain the same to ease VM mobility. Need to increase east-west bandwidth. Flattening the data centre networks With the advent of virtualization, 10GbE and faster connections, unified fabrics based on lossless Ethernet, and other technological innovations, data centre networking vendors started proposing two-tier topologies. Ethernet fabrics
8 Modern two-tier networks Access layer and Aggregation layer are merged together. Virtual chassis provides flexibility as needed. Spanning Tree Protocol is no longer needed. Troubleshooting the network might be more challenging
9 Ethernet fabrics Motivations Increase east-west bandwidth Reduce latency by eliminating layers Necessity of utilizing multiple paths simultaneously Abandoning STP providing resiliency and redundancy Scale to large L2 domains
10 Ethernet fabrics Characteristics Utilize high-capacity and non-blocking switching nodes Adapt L3 routing protocols to be utilized at L2 and forward frames on the shortest path between nodes Provide the network administrators better tools to troubleshoot L2 networks Maintain L2 plug-and-play capabilities VMs can move preserving the IP and MAC address Generally based on a so called Clos or fat-tree topology avoid oversubscription guaranteeing a non-blocking switch fabric Topology independent in practice (few exceptions) Eliminate the need to utilize STP maintaining the interoperability with legacy networks
11 Ethernet fabrics Increased network flexibility Simplified VMs mobility thanks to the ability to extend VLANs anywhere is required Resiliency is guaranteed by the many available equal cost paths TRILL and SPB Concurrent standards by IETF and IEEE. Standardization boards do not agree on how to implement the fabric. TRILL has a proprietary header, while SPB uses Provider Backbone Bridge (PBB). SPB-VID and SPB-MAC are the two available alternatives for SPB TRILL has currently no OA&M features, while SPB is compatible with 802.1ag Connectivity Fault Management, 802.3ah and Y.1731 Ethernet OA&M. TRILL and SBP differs in how multicast traffic is handled. TRILL uses only a limited set of distribution trees.
12 Ethernet fabrics Cisco is proposing a TRILL-like technology called FabricPath It is not compatible with TRILL. A set of extensions such as vpc+, multiple topologies, active/active HSRP are available only in FP mode. It uses Conversational MAC learning This reduces the amount of MAC addresses that the access switches have to learn. MAC addresses are not learnt from flooded frames Performances evaluated by NetworkWorkd FP proved to use the available ECMPs in a fair way exhibiting a variation of the load distributed between the equal cost paths of only 0.07% or less. Convergence time while adding new switches to the network is zero. FabricPath converges on average, in case of a link failure, in 162 milliseconds.
13 Ethernet fabrics Brocade is proposing a solution really close to TRILL called VCS It adopts TRILL data plane, so the frames are structured as in TRILL It uses a different control plane since the routing protocol used in this case is Fabric Shortest Path First (FSPF) typically utilized in Fibre Channel fabrics. FSPF only defines and implements point-to-point links. No designated router concept. Brocade is committed to be TRILL compliant once the standard will be approved Only a software update is needed to implement a new routing protocol
14 Ethernet fabrics Juniper is proposing a totally different solution named QFabric It s a proprietary technology that is part of Juniper s Data Centre Network Architecture strategy. QFabric creates the illusion of a singles virtual switch supervised by a management platform that gives one view of the fabric. Evolution of the chassis switch architecture Line cards -> QF/Nodes Backplane -> QF/Interconnects Route engine/ Supervisor card -> QF/Director 40 GbE connections Latency < 5µs
15 Ethernet fabrics From the research field the most interesting idea that emerged in the past years is the realization of a Distributed Hash Table (DHT) to store associations between IP and MAC addresses in switches and eliminate the broadcast traffic generated by ARP requests. Originally introduced in SEATTLE where also the location of the hosts is stored in switches using a DHT Consistent hashing is used to identify where to store tuples of values. Special hashing mechanism where when the number of bins changes only K/n keys need to be remapped. (n = # of bins, K= tuples of values) on average. Can be adapted to be used also with other technology Nuutti Varis and Jukka Manner from the Department of Communications and Networking at Aalto University proposed a way to introduce a DHT in TRILL
16 Conclusions The future of data center network is flat. Two-tier networks, Ethernet fabrics Resource virtualization and consolidation Currently serious risk of vendor lock-in Currently impossible to determine which solution is better. It all depends on how the different vendors will be able to market their solutions. Impossible to test all the solutions proposed by the different vendors, hence the comparison was done interviewing experts form different companies. At the moment TRILL seems to be the most widely adopted solution. Most likely small data centre such as the ones available in Finland will not need, at least in the near future, to adopt Ethernet fabric solutions that seem to be cost effective only for extremely big data centres (tens of thousands of servers/virtual machines)
17 Future Work DC2F is currently entering in its implementation phase. A test data center is being built in Otakaari 5 Test different network configurations such as M-LAG and virtual chassis Test, if possible, FabricPath. At the moment it is available only for Nexus 7000 Series switches, but it should be released also fo the Nexus 5000 Series switches that we have. Investigate if power consumption of the switches is related to the network technology adopted. Currently it seems that newer switches are more energy efficient but because of the hardware they utse and not because of the protocols they support. Much more...
The Road to Cloud Computing How to Evolve Your Data Center LAN to Support Virtualization and Cloud Introduction Cloud computing is one of the most important topics in IT. The reason for that importance
I n t e r n a t i o n a l T e l e c o m m u n i c a t i o n U n i o n ITU-T TELECOMMUNICATION STANDARDIZATION SECTOR OF ITU FG Cloud TR Version 1.0 (02/2012) Focus Group on Cloud Computing Technical Report
CHAPTER 1 LAN Design Objectives Upon completion of this chapter, you will be able to answer the following questions: How does a hierarchical network support the voice, video, and data needs of a small-
White Paper How to Build a Better Cloud: Leveraging Unified, Virtualized Storage and Data Center Fabrics By Bob Laliberte, Senior Analyst, and Kerry Dolan, Research Analyst September 2012 This ESG White
Industrial Ethernet: A Control Engineer s Guide Abstract As part of a continuing effort to make their organizations more efficient and flexible, manufacturers are rapidly migrating to Industrial Ethernet
Institut für Technische Informatik und Kommunikationsnetze Kirila Adamova Anomaly Detection with Virtual Service Migration in Cloud Infrastructures Master Thesis 263-8-L October 22 to March 23 Tutor: Dr.
Software-Defined Networking: The New Norm for Networks ONF White Paper April 13, 2012 Table of Contents 2 Executive Summary 3 The Need for a New Network Architecture 4 Limitations of Current Networking
White PAPER 10 Gigabit Ethernet Virtual Data Center Architectures Introduction Consolidation of data center resources offers an opportunity for architectural transformation based on the use of scalable,
Fibre Channel over Ethernet (FCoE) Data Center Bridging (DCB) Concepts and Protocols Version 15 Fibre Channel over Ethernet (FCoE) and Ethernet Basics Storage in an FCoE Environment EMC RecoverPoint and
Network Monitoring and Analysis Techniques Using Taps and SPAN Switches Networks have evolved into complex structures supporting critical business processes and communications. As this complexity has increased,
Network Infrastructure TSL has a great deal of experience in designing and implementing secure, high availability broadcast network systems, and working with clients to integrate such with new or existing
Contents Introduction...1 Overview of x86 Virtualization...2 CPU Virtualization...3 The Challenges of x86 Hardware Virtualization...3 Technique 1 - Full Virtualization using Binary Translation...4 Technique
Virtual Machine Mobility with VMware VMotion and Cisco Data Center Interconnect Technologies What You Will Learn VMware has been the industry leader in virtualization technologies for the past decade and
ARISTA WHITE PAPER Software Driven Cloud Networking Arista Networks, the leader in high-speed, highly programmable data center switching, has outlined a number of guiding principles for integration with
Duke Energy: Developing the communications platform to enable a more intelligent electric grid David Masters David.Masters@duke-energy.com Manager, Technology Development Duke Energy February 1, 2011 Certain
SAN Design and Best Practices Version 2.3 A high-level guide focusing on Fibre Channel Storage Area Network (SAN) design and best practices, covering planning, topologies, device sharing in routed topologies,
ProActive Routing in Scalable Data Centers with PARIS Dushyant Arora Arista Networks email@example.com Theophilus Benson Duke University firstname.lastname@example.org Jennifer Rexford Princeton University email@example.com
Network Monitoring with Software Defined Networking Towards OpenFlow network monitoring Vassil Nikolaev Gourov Master of Science Thesis Network Architectures and Services Faculty of Electrical Engineering,
Onix: A Distributed Control Platform for Large-scale Production Networks Teemu Koponen, Martin Casado, Natasha Gude, Jeremy Stribling, Leon Poutievski, Min Zhu, Rajiv Ramanathan, Yuichiro Iwata, Hiroaki
IP TELEPHONY POCKET GUIDE BY BARRY CASTLE 2nd Edition September 2004 ShoreTel, Inc. 960 Stewart Drive Sunnyvale, CA 94085 408.331.3300 1.800.425.9385 www.shoretel.com firstname.lastname@example.org TABLE OF CONTENTS
Firewall Strategies June 2003 (Updated May 2009) 1 Table of Content Executive Summary...4 Brief survey of firewall concepts...4 What is the problem?...4 What is a firewall?...4 What skills are necessary