The Hitchhiker s Guide to the New Data Center Network by Zeus Kerravala November 2010 I. The Virtual Data Center Requires a New Network The enterprise data center has undergone several major transformations since the introduction of computing as a corporate resource. The computing platform has evolved from mainframe computing, to client server, to Internet-based computing, and now we sit on the precipice of the next major data center transition the evolution to a fully virtualized data center (see Exhibit 1). Each transition saw the cost of computing driven down and the importance of the network elevated. Each phase allowed organizations to increase the efficiency of their data center operations and improve asset utilization, ultimately leading to a better experience for end-users. Exhibit 1: The Data Center Through the Years Source: Yankee Group, 2010 NETWORK VALUE Mainframe Era Client/Server Internet Computing Virtual Computing 1960-1980 1980-1995 1995-2010 2010+ COST OF COMPUTING The transition to a virtual data center will be the most significant IT transformation since the Internet was born, as it aims to bring together storage, computing and data networking. A virtual data center has a profound impact on the network. The vision of a fully virtualized data center is for all data center resources to exist as pools of virtual resources that can be accessed by whatever service needs them, whether across the data center, across the city or, in theory, across the globe. In this case, the network acts as the backplane for the virtualized data center (see Exhibit 2 on the next page). This custom publication has been sponsored by Avaya.
The Hitchhiker s Guide to the New Data Center Network Exhibit 2: The Network Is the Backplane of the Virtual Data Center Source: Yankee Group, 2010 Mobility Services UC Services Virtualized Access Layer Data Center Network Storage Pool Application Resources Computer Resources The key trends in the virtual data center that will put additional emphasis on the network are as follows: Compute virtualization and virtual machine (VM) mobility. Despite its relatively short life as a mainstream IT technology, virtualization is already well adopted. Yankee Group s Anywhere Enterprise Large: 2009 U.S. Transforming Infrastructure and Transforming Applications Survey, Wave 1-12, reveals that 71 percent of responding organizations have already deployed server virtualization. However, thus far, virtualization has been primarily used for consolidating servers. The next wave will be mobilization of VMs, which allows an application to move from one physical server to another in real time. Virtualization of other data center assets. To date, virtualization has primarily been focused on servers. As the technology continues to mature, other compute resources, such as storage, I/O and even the network, will be virtualized. A green data center. A tough economy combined with a greater focus on corporate social responsibilities has put an emphasis on energy-efficient IT. This means all physical infrastructure should be optimized for power and cooling, and the architectural design of virtual data centers must take energy efficiency into consideration. Simplification of IT operations and automation. The evolution of the data center has certainly driven costs down and allowed businesses to use technology much more efficiently, driving productivity to new heights. However, the IT environment has become much more complex than in prior years. Yankee Group s ongoing research indicates that approximately 80 percent of an IT budget is now used to maintain the status quo, driving organizations to find ways of simplifying the current environment. Virtualization will allow IT managers to automate many of the operational tasks done today through the movement of virtual resources. If the movement of VMs is not well coordinated with network operations, the network could be blind to the traffic flows. Cloud computing. The long-term vision of virtualization is to ultimately move many of the virtual resources to the cloud. Cloud computing s success is highly dependent on the network to provide the foundation for cloud-based resources to traverse. All these data center trends create new demands on the network. This means the traditional evaluation criteria of simply buying from the incumbent vendor because the solution is good enough is no longer sufficient. Evaluators of network infrastructure must understand the limitations of traditional networking in the context of the virtual data center s new requirements. II. Limitations of the Current Network The current network architecture used to support data centers is outdated and cannot support the virtual data canter. The historical value chain in the traditional data center stack assigned applications the highest value and considered the network to be plumbing, having the least value relative to the rest of the stack. The network was a necessary part of the overall data center, but since most applications were best effort, it ultimately did not play a critical role in the overall performance of applications. More specifically, the network of the traditional data center has the following limitations in its ability to support a virtual data center: Its three-tier architecture has too much latency to support real-time and large workloads. In a typical three-tier design, all traffic between servers is sent through each tier, to the core and then back through each tier to the other server. All these hops between network devices add a tremendous amount of latency that will impact the real-time needs of the virtual data center. The traditional three-tier design will no longer be sufficient. 2
November 2010 Its use of Spanning Tree Protocol (STP) leads to inefficient network infrastructure usage. STP prevents routing loops and broadcast radiation by disabling ports that are not part of the spanning tree. These disabled ports are only made active in the event of a failure of one of the active ports. This inefficient use of network ports requires companies to purchase many more ports than necessary. Traditional technology is not designed for virtual or cloud environments. Traditional data center technology is designed for best-effort traffic such as e-mail and Internet. Cloud computing and the real-time movement of VMs requires much lower latency and guaranteed delivery. Best effort is no longer good enough. Its poor management tools and high up-front cost drive up TCO. Many of the market share leaders portfolios are made up of a combination of in-house developed technology plus acquired technology. These different systems often have different OSs and management tools. This is often the case in mature markets where acquisitions are used to increase share as well as fill in technology holes. Trying to manage the endto-end performance of the disparate components of a data center network is almost impossible with today s network infrastructure and management platforms. The transformation of the data center is ushering in a new era in networking. A solution optimized for the virtual data center incorporating high performance, low latency and guaranteed delivery is required. Historically, companies made network infrastructure decisions based on brand and incumbency versus technical superiority. But as the era of the virtual data center gains momentum, the network will continue to increase in value and network managers will find infrastructure that was good enough for Internet computing will not be good enough for virtual computing. III. Virtual Data Network Requirements The network requirements of a virtual data center have created the need for a change in infrastructure and architecture. There can be no compromise between performance, reliability, scalability, features and cost. A solid network foundation with infrastructure that has been designed with virtualization in mind is necessary for the long-term evolution and success of the data center. Key attributes for this foundation include: Network fabric capable of supporting east-west and north-south traffic flows. Traditional data center networks are designed to have all traffic flow north-south that is, from the edge of the network, through each tier, through the core and then back on a similar path. This creates significant congestion on the inter-switch links and adds latency to the traffic flow. A network fabric can be thought of as a single entity where traffic can move east-west and is never more than a single hop away from its destination. This is critical for the movement of virtual workloads where any latency will cause a disruption to the business. The network fabric must support both east-west traffic movement and north-south traditional traffic flows. Data center bridging to create a virtual backbone between data centers. Bridging data centers has become a primary focus for many companies today. The ability to move storage traffic between data centers over Ethernet has many benefits. Storage protocols such as Fibre Channel or Fibre Channel over Ethernet (FCoE) operate at Layer 2 of the OSI stack. This means that interconnecting storage networks between geographically dispersed data centers is a significant challenge. Root bridge STP-based VLANs are not robust enough to transport storage. Additionally, these networks do not support shortest path switching, which is necessary to minimize the latency of the storage traffic. Advanced network standards such as Shortest Path Bridging (SPB) and the Transparent Interconnection of Lots of Links (TRILL) are being positioned to replace the outdated STP and allow Layer 2 domains to be extended between data centers, creating a single, seamless backbone for storage traffic. It s critical that the bridging technology be based on a standard for greater interoperability in the future. However, the Layer 2 issue is only one area that must be addressed. The ability to support dual-homing of servers and Layer 3 routing must also be part of the design criteria for the next-generation data center. Solving only one piece of the equation is insufficient and, realistically, not deployable in the enterprise. 3
The Hitchhiker s Guide to the New Data Center Network Exhibit 3: Two- and Three-Tier Architectures Source: Yankee Group, 2010 Server Rack Server Rack Access Access TRAFFIC FLOW Core Aggregation TRAFFIC FLOW TRAFFIC FLOW Core Three Tier Two Tier Architectural choice in next-generation data center design. The most obvious and commonly talked about migration strategy from the current three-tier data center architecture is to move to a two-tier architecture (see Exhibit 3), in which all the devices in a single rack connect to a top-of-rack switch. These top-of-rack switches are horizontally stacked together, and instead of running to an end-of-rack switch, they are connected directly to the core switch, thereby eliminating the tier of switches at the end of each row. This leads to TCO savings that are both capital (removal of hardware) and operational (simpler design means simplified management) in nature. There is no one right solution here. Some companies will opt to stay with a three-tier architecture, while some will choose a two-tier environment. The network infrastructure must be capable of supporting both options. Active/active switching. STP is widely used in data center networks today to prevent network traffic loops that is, traffic that takes a circular path and does not reach its destination. This can happen when multiple links are used between network switches for redundancy purposes. STP operates by disabling one of the two active connections and putting the second link in a passive standby mode until the primary link fails. While STP does achieve the goal of avoiding routing loops, it is a highly inefficient protocol in that up to half the ports could be disabled. Ideally, the network would be active/active, meaning all ports are active and able to pass traffic simultaneously. This leads to a lower cost, simpler network that performs better. The active/active fabric needs to be a topology-independent solution, meaning it can be a fully meshed fabric or partially meshed network. 4
November 2010 High-density 10 Gbps Ethernet (GbE) with seamless upgradability to 40 and 100 GbE. Speed matters in a virtual data center. The bigger the pipe, the more information can be passed across it. However, companies often have to choose between line cards with high density and lower speeds or high-speed cards with fewer ports. Ultimately, the network switch should support high-density 10 GbE and then provide an upgrade path to 40 and 100 GbE when those standards are ratified and products are commoditized. Ideally, the network would be upgradable from 10 to 40 and then 100 GbE without a forklift upgrade, allowing for minimal disruption and the best TCO scenario. Increased network availability. The virtual data center requires a 24/7 network. As virtualization drives the automation of the compute layer, the continued availability of the network will become even more important. The replacement of STP with SPB or TRILL can simplify the network and add to its overall uptime. For example, the core SPB infrastructure needs to be configured only once. Virtual networks can be provisioned at the edge and will seamlessly traverse the core without any need for manual configuration changes, thus helping to eliminate the human-error factor. Multi-tenant capabilities. In many organizations, the network requires the total separation of traffic. This could be the case where the IT organization is supporting many subsidiary companies or the business units require it. This is very common for verticals such as state and local government, education and financial services. The network must be a single physical network that looks and acts like several logical networks, each with its own unique needs and requirements. For this to operate efficiently, the network must automatically separate traffic into distinct virtual networks but have access to a common set of data center services (see Exhibit 4). IV. What To Look for in a Solutions Provider The demands of the virtual data center are driving network evolution faster than ever before. The network will play a key role in determining the ultimate success or failure of fulfilling the vision of a virtual data center. This is a new role for the network and, consequently, network decision-makers must shed old-school buying strategies based primarily on brand or vendor incumbency and instead evaluate network infrastructure based on its ability to support the needs of the evolving data center. Exhibit 4: Multi-Tenant Network Source: Yankee Group, 2010 Organization A Organization B Organization C Virtual Network A Virtual Network B Virtual Network C Data Center Network Fabric Common Resources Virtual Data Center 5
The Hitchhiker s Guide to the New Data Center Network However, a full understanding of what to look for in a network solution provider may not be obvious, especially with the ongoing computing transition under way. Enterprises evaluating solution providers must consider the following. Standards-Based Solutions There are many ways for solution providers to meet the challenges of network evolution. Many vendors choose to use proprietary protocols and solutions to develop new products. This can sometimes short-cut development time and allow vendors to get products to market early. However, in the long term, it causes vendor lock-in and impairs a customer s ability to choose bestof-breed products in the future. A standards-based solution guarantees interoperability with other best-of-breed products and ensures a wide variety of choice. For example, there are many possible replacement protocols for STP, but only SPB is based on a standard (IEEE 802.1aq specifications). TRILL and other protocols are not currently standardized and may lead to interoperability issues in the future. A Migration Path Away From Current Network Topology The data center is the most important IT asset for the majority of organizations today. It contains all of a company s critical data, and any disruption could be crippling to an organization. Because of this, any architectural change must be based on a simple, reliable migration path and not a rip and replace strategy. The solution provider must be able to support the existing topology while providing immediate benefits where the new technology is deployed. For example, in a three-tier data center, a migration path might be to first replace only the core switching infrastructure. This would provide the core with the benefits of a network fabric where required without disrupting the compute access layer. Eventually, the rest of the network can be migrated at the organization s own pace. This allows IT departments to get comfortable with the new technology without disrupting business operations. Unified Management Network hardware is obviously a very important piece of the overall solution, but it s not the only one. A unified management platform is just as important for the ongoing reliability and efficiency of the network. A unified management tool needs to provide the following: End-to-end network management: This includes end-toend provisioning, monitoring and troubleshooting of virtualized and physical elements. Performance and fault management: Key functions here are topology discovery, fault correlation and network discovery. Ideally, the management tool provides these functions in a multivendor environment. IP flow management: This management tool needs to be able to monitor network flows for traffic analysis, capacity planning and usage analysis. Wizard-based configuration management: This provides network managers with a step-by-step method of streamlining the configuration and provisioning of the network, significantly reducing the number of outages due to human error currently the largest cause of downtime in networks today. Orchestration management: A key requirement of this management tool is to orchestrate the configuration of virtual services across the network fabric. This simplifies many of the complex tasks needed to create the fabric, such as configuration of the SPB infrastructure. Network Infrastructure Optimized for the Virtual Data Center Many network products in data centers today were designed in the early part of the decade or even the late 90s. A network that could support best-effort traffic, i.e., a network that was good enough, was all that was really required in that era. The network infrastructure for the virtual computing era must be designed and optimized for the unique demands of the evolving data center. Key criteria for the network infrastructure are: Fully redundant hardware devices with no single point of failure for continuous operations. This should not be an option, but a standard part of the solution. An OS designed with the data center in mind. A few network vendors have been building solutions that use the same OS across all network devices. However, the needs of the data center are significantly different than those of wiring closets and branches, and the OSs should be optimized accordingly. 6
November 2010 Future-readiness. The solution must allow for seamless upgrades to higher speeds and greater throughput when the needs of the organization dictate. Converged Solutions Virtualization is driving the network and compute layers together. The current focus of data center virtualization has been on compute servers, but storage transport is equally important. The network fabric needs to support all flavors of current and converged storage, including FCoE, iscsi, network-attached storage (NAS) and ATA over Ethernet (AoE). The network must also be able to segregate the storage traffic into its own unique virtual network. V. Conclusions and Recommendations The drive to virtualization is transforming the data center faster than at any other time in the history of IT. Virtualization has already had a significant impact on the software and server industry and will have a similar impact on networking, as the network plays a critical role in mobilizing virtual workloads. A fully virtualized data center will allow companies to allocate virtual IT resources to the applications and services that require them on an on-demand basis, creating a truly agile IT infrastructure. Such a data center puts the network in a position to become the key point of competitive differentiation for organizations as they look to capitalize on the flexibility and cost-efficiencies of virtualization. This introduces new requirements for network design and choice of solution provider. To realize its full potential, the network needs to undergo a major transformation. Customers must move decision criteria away from things like market share and vendor incumbency and instead focus on things like how resilient the network is, how the network simplifies architecture and how it enables virtualization to be used more broadly. With this in mind, Yankee Group makes the following recommendations for companies looking to use the network and virtualization as a key point of competitive differentiation: Simplify the network architecture as much as possible. Companies should look to migrate away from a three-tier architecture to a two-tier or even single-tier option when feasible. The less complex the network is, the easier it is to manage and troubleshoot. Stick with a standards-based, open solution. A virtual data center comprises a large ecosystem of solution providers. A network built on closed, proprietary technologies may not be able to interoperate with all the necessary ecosystem vendors. A standards-based, open solution guarantees the broadest range of choice for customers. Make power efficiency a key part of the decision criteria. Power and cooling requirements vary widely between solution providers. Choose a solution provider that includes power and cooling efficiency as part of its overall solution design and can quantify the benefits to you. Deploy a no-compromise network, even if that means moving away from your incumbent vendor. Choosing the incumbent vendor often seems like the easy choice because of product familiarity it s the safe option. However, this often forces IT organizations to make compromises it may regret in the future. The needs of the virtual data center are significantly different than those of current data centers, so going down the path of least resistance might be the wrong choice. Organizations should no longer have to compromise between purchase price, resiliency, power efficiency and feature set. Choose the vendor that best supports your virtualization initiatives, and do not compromise on any features needed to fulfill on the vision of a fully virtualized data center. 7
Yankee Group the global connectivity experts The people of Yankee Group are the global connectivity experts the leading source of insight and counsel trusted by builders, operators and drivers of connectivity solutions for 40 years. We are uniquely focused on the evolution of Anywhere, and chart the pace of technology change and its effect on networks, consumers and enterprises. For more information, visit http://www.yankeegroup.com Yankee Group s products and services provide clients the insight, analysis and tools to navigate the global connectivity revolution. Research Data Interaction Consulting Events Leverage qualitative research to make informed business decisions today and plan for the future. Gain quantitative insight into current markets and new opportunities via monitors, surveys and forecasts. Connect with analysts to gain deeper insight into research and trends. Get in-depth analysis and actionable recommendations tailored to your needs. Access world-class events live and online with industry leaders and Yankee Group experts. Zeus Kerravala, Senior Vice President and Distinguished Research Fellow Zeus Kerravala, senior VP and distinguished research fellow, leads the Research Council and is chartered with the responsibility of providing thought leadership to the research organization. Comprising senior research leaders, the Research Council provides outreach to clients and the broader Yankee Group community, as well as ensures that the company s research agenda addresses the needs of business leaders. Zeus drives the strategic thinking of the research organization and helps shape the research direction. Much of his expertise involves working with customers to solve their business issues through the deployment of infrastructure technology. Headquar ters Copyright 2010. Yankee Group Research, Inc. Yankee Group published this content for the sole use of Yankee Group subscribers. It may not be duplicated, reproduced or retransmitted in whole or in part without the express permission of Yankee Group, One Liberty Square, 7 th Floor, Boston, MA 02109. All rights reserved. All opinions and estimates herein constitute our judgment as of this date and are subject to change without notice. Corporate One Liberty Square 7th Floor BOSTON, MASSACHUSETTS 617-598-7200 phone 617-598-7400 fax European 30 Artillery Lane LONDON E17LS UNITED KINGDOM 44-20-7426-1050 phone 44-20-7426-1051 fax