The New Brocade Data Center and Its Advantages

Transcription

1 WHITE PAPER DATA CENTER World-Class Data Center Realizes Goals of Investment Protection, Energy Efficiency, and Scalability In March of 2008, Brocade had a vision to build a worldclass campus, including a new data center to demonstrate the Brocade One concepts of unmatched simplicity, nonstop networking, investment protection, and optimized applications. In September of 2010, the vision was realized and we officially dedicated the new campus and data center.

2 The new Brocade campus in San Jose, California, was built to provide the business with flexibility needed to satisfy growth for our employees and for R&D and solution labs. Design principles of both the campus and the data centers were flexibility, scalability, energy efficiency, and sustainability. And the results were nothing short of extraordinary. Three buildings provide 562,000 square feet to house Brocade s 2,500 employees, labs, data center, café, fitness center, and so on. An overarching goal was thoughtful investment and that guided our decisions with regards to our employees, partners, customers, and vendors as well. Prior to the move, we had three data centers and R&D labs in five locations with a total of about 2000 racks. On the new campus, labs were consolidated in one building. The highly efficient electrical and mechanical design of Brocade data center and labs will save over 10 million Kilowatt-hours of energy consumption annually over the previous year. And over 4500 tons of carbon dioxide emissions annually will be offset over the previous year. This paper describes how we designed and built the data center and labs to achieve these spectacular results. 2

3 Introduction Looking back at the planning phase, a number of goals were mandated by the executive staff: Energy efficiency and sustainability Consolidation of footprint, while maximizing real estate investment Investment protection, that is, use as much of the existing equipment as possible Create a baseline for future growth and enhancements The following sections describe how we accomplished these goals followed by some best practices and lessons learned about planning and building or refreshing a corporate data center. Energy Efficiency and Sustainability Both industry best practices and innovative design played a part in delivering on the green promise in the new data center. Hot Aisle/Cold Aisle As is common in most organization, the IT equipment load is the number one consumer of electricity at a building level. One of the best ways to handle the heat generated by the many devices in the data center is by using a hot aisle/cold aisle equipment floor plan. Servers and network equipment are typically configured in standard 19 (wide) racks and rack enclosures, in turn, are arranged for accessibility for cabling and servicing. Increasingly, however, the floor plan for data center equipment distribution must also accommodate air flow for equipment cooling. This requires that individual units be mounted in a rack for consistent air flow direction (all exhaust to the rear or all exhaust to the front) and that the rows of racks be arranged to exhaust into a common space, called a hot aisle/cold aisle plan, as shown in Figure 1. COLD aisle Equipment row HOT aisle Figure 1. Hot aisle/cold aisle plan for consistent air flow. Equipment row Air flow COLD aisle Equipment row HOT aisle A hot aisle/cold aisle floor plan provides greater cooling efficiency by directing cold to hot air flow for each equipment row into a common aisle. Each cold aisle feeds cool air for two equipment rows while each hot aisle allows exhaust for two equipment rows, thus enabling maximum benefit for the hot/cold circulation infrastructure. Even greater efficiency is achieved by deploying equipment with variable-speed fans. Hot row containment provides a better working environment for IT personnel and temperatures are more comfortable. In Brocade data center and labs, we can control temperature to the level of a pod if we need to (two rows of equipment back to back make up a pod). The Solutions Center lab also uses both high end-doors and more economical curtain system heat containment methods. 3

4 Variable Speed Fans Variable speed fans increase or decrease their spin rate in response to changes in equipment temperature. As shown in Figure 2, cold air flow into equipment racks with constant speed fans favors the hardware mounted in the lower equipment slots and thus nearer to the cold air feed. Equipment mounted in the upper slots is heated by their own power draw as well as the heat exhaust from the lower tiers. Use of variable speed fans in equipment racks enables each unit to selectively apply cooling as needed, with more even utilization of cooling throughout the equipment rack. Figure 2. Variable speed fans in equipment racks enable more efficient distribution of cooling. More even cooling Equipment at bottom is cooler Server rack with constant speed fans Server rack with variable speed fans Power Distribution Units High-voltage power is fed into the data center and labs to Power Distribution Units (PDUs), which physically sit inside these labs, minimizing the energy and line loss to the end-point. The PDUs have shadow metering capabilities to monitor and measure energy consumption with utility-grade precision. Green-labeled PDUs are backed up by UPS and emergency power, while red-labeled PDUs are backed up by emergency power only. Inside the data center, equipment is fed from both A and B power sources for redundancy, with A power having UPS and emergency generator backup. Figure 3. PDUs minimize energy and line loss to the end-point. In addition to known best practices in this area, Brocade worked with a trusted vendor to design and build a custom In-Row Cooling (IRC) system. To quote the vendor, This may well be the best system on the market today that s not on the market but will be. This IRC unit provides the ultimate solution in flexibility due to its unique custom design. It can fit into any rack space in the data center because of its similar physical footprint to the rack. Brocade s custom IRC is 80 percent more energy efficient than the conventional Clean Room Air Conditioning Unit (CRAC) and 70 percent more efficient than current alternative IRC units on the market. 4

5 The IRC unit provides added reliability by utilizing three high-efficiency fans directly driven by three EMC motors. The motors themselves are the most efficient on the market, consuming about 70 percent less energy than the nearest competitor. Each motor operates at 33 percent, and if a fan or motor fails the unit automatically ramps up the remaining fans to 66 Hz to provide the full cooling capacity of the unit. Automated Controls with Energy Monitoring Because vendor wattage and BTU specifications may assume maximum load conditions, using data sheet specifications or equipment label declarations does not provide an accurate basis for calculating equipment power draw or heat dissipation. An objective multi-point monitoring system for measuring heat throughout the data center is really the only means to observe and proactively respond to changes in the environment. A number of monitoring options are available today. For example, some vendors are incorporating temperature probes into their equipment design to provide continuous reporting of heat levels via management software. Some solutions provide rack-mountable systems that include both temperature and humidity probes and monitoring through a Web interface. In addition, new monitoring software products can render a three-dimensional view of temperature distribution across the entire data center, analogous to an infrared photo of a heat source. Although monitoring systems add cost to data center design, they are invaluable diagnostic tools for fine-tuning airflow and equipment placement to maximize cooling and keeping power and cooling costs to a minimum. An example of a three-dimensional view of temperature in the Brocade data center is shown in Figure 4. Starline Busway System It s interesting to note that Brocade was an early adopter of the Starline Busway System, an electrical power distribution system for mission-critical facilities. It is simple, versatile, fast and economical solution for supplying power to electrical loads and is unique because the busway can be tapped instantly at any location, without losing uptime giving you the ability to change or add to floor plans quickly and easily. Although the Starline Busway System has been on the market for a number of years, it s still winning awards, most recently the Tech 50 award from the 2009 Pittsburgh Technology Council in the Advanced Manufacturing category. Figure 4. A 6 Sigma application generates heat topologies at different heights from the floor. Cooling System The number two consumer of electricity at the building level are the chiller and then the pumps to circulate water for cooling to the HVAC units in the lab. Brocade selected the highest-efficiency chillers at partial loads and cooling towers with variable frequency drives. As a general rule, chillers never reach 100 percent load, the point at which their highefficiency features are triggered. However, the high-efficiency features of our chillers are triggered at partial loads below 50 percent load, which helps conserve energy at Day One loads Our chilled water system utilizes a water side heat exchanger, which is passive and has no electrical pumps and is capable of providing 2000 tons of free cooling without the use of a compressor. This water-side heat exchanger provides free cooling when the outside temperature permits. Despite what you may have heard, in Silicon Valley the sun doesn t shine all the time. 5

6 Finally, we have a primary system only and no secondary pumping system. Before the 1980s, primary-only systems were the norm. With the growth of high-tech labs and critical labs, a primary/secondary system has been typically prevalent until very recently. Organizations were reluctant to do away with primary/secondary systems, because they felt that it made them vulnerable in the event of a fluctuation differential pressure that might exist in a primary only system. But in Brocade s case where through the design and selection of the correct equipment, automation and controls, and its constant load, there is no concern or need to utilize a primary/secondary system to regulate pressure in load. We simply don t need it and have avoided initial capital investment cost and inefficiencies. Figure 5. Main cooling system substations. Consolidation Through the consolidation of three data centers into one we were able to consolidate our space requirements by decreasing the physical footprint by 30 percent and decreasing data center energy consumption by 37 percent. Racks One of the decisions that we made was to use taller racks 8 tall instead of 6 with standard 19 width. We were able to consolidate 2011 racks to the 1860 rack capacity in for all labs. In 2011, it is projected that we will be occupying 1587 of these racks, with 300 targeted for growth. With these taller racks and greater rack count, a question that comes to mind is how to fit them into the data center and labs. We used a flat floor design, which allows for more vertical rack space, rather than a raised floor, which requires ramps and stairs for access. A flat floor design was selected because it allowed 10 percent more rack counts in comparison to conventional raised floor systems which require ramps and stairs. Cabling One of the most critical areas for consolidation is in fact the proliferation of cables in highdensity racks, with high-port-count devices. In both the data center and other labs, two separate conveyance systems were deployed: structured and horizontal cabling systems (Systimax) and ad hoc cabling in the data center (Systimax, Tyco) and ad hoc cabling in the solutions and proof-of-concept labs (Corning, Splice, Tyco, and Systimax). This lab also features a flexible two-tier cable conveyance design that provides both a structured cable tray conveyance system for the backbone network and a high capacity open cable conveyance system for ad-hoc cabling. In the Proof-of-Concept (PoC) lab, Brocade products are physically connected to an optical switching matrix that lets us rapidly build out and store network topologies for each customer. This allows us to share equipment and minimize set up time. 6

7 Investment Protection The new data center is initially a Tier 1 (no N+1 power or cooling), but the infrastructure is in place for a Tier 2 infrastructure (generator pads, UPS, additional chillers) on which N+1 systems can be placed when the business requirement exist The data center has installed the latest in fire suppression technology. This includes a double-action, pre-action dry pipe system that prevents water leaks and false alarms from interrupting business-critical applications. Fire and smoke detection is monitored by very early smoke detection, which senses the smallest amount of combustion or smoke particles and activates a dry green agent fire suppression system that stops and prevents fire from spreading by removing the oxygen in the space. Campus Sustainability To summarize, by meeting sustainability challenges head on: This campus uses 40 percent less water than a standard office or campus building. This is achieved by the collection of rain and run-off water recycled, filtered, and reused for irrigation. The selection of sustainable trees and landscaping features, and the selection of no-water and low-flow water fixtures throughout the campus help us accomplish this. The campus utilizes a 550 KW photovoltaic (PV) system attached to our garage structure that harnesses the suns energy to offset power consumption and the carbon footprint. The power from the PV system is fed into Building 2. This system is capable of offsetting the total amount of power consumed by our data center. By strategically implementing the best practices at a building and lab level one of the lowest calculated Power Utilization Factors in Data Center design of 1.3 has been achieved. The design has also allowed us to exceed California energy code requirements by 16%. Due to the selection of a sustainable site, water efficiency, energy efficiency, selection and reuse of renewable materials, indoor air quality achievements, and innovation of design, the campus is scheduled to receive accreditation from the U.S. Green Building Council for Leadership and Energy and Environmental Design (LEED) certification of silver for core and shell and gold for corporate Interiors. Figure 6. Power Usage Effectiveness (PUE): a measure of data center energy efficiency calculated by dividing the total data center energy consumption by the energy consumption of the IT computing equipment. About Brocade Founded in 1995, Brocade is an industry leader in providing reliable, high-performance network solutions that help the world s leading organizations transition smoothly to a virtualized world where applications and information reside anywhere. Today, Brocade is extending its proven data center expertise across the entire network with future-proofed solutions built for consolidation, network convergence, virtualization, and cloud computing. Headquartered in San Jose, California, the company has approximately 4000 employees worldwide and serves a wide range of industries and customers in more than 160 countries. With a complete family of Ethernet, storage, and converged networking solutions, Brocade helps organizations achieve their most critical business objectives through: Unmatched simplicity to overcome today s complexity Non-stop networking to maximize business uptime Optimized applications to increase business agility and gain a competitive advantage Investment protection to provide a smooth transition to new technologies while leveraging existing infrastructure Brocade combines a proven track record of expertise, innovation, and new technology development with open standards leadership and strategic partnerships with the world s leading IT companies. This extensive partner ecosystem enables truly best-in-class business solutions. To find out more about Brocade, visit 7

8 WHITE PAPER Corporate Headquarters San Jose, CA USA T: European Headquarters Geneva, Switzerland T: Asia Pacific Headquarters Singapore T: Brocade Communications Systems, Inc. All Rights Reserved. 10/10 GA-WP Brocade, the B-wing symbol, BigIron, DCFM, DCX, Fabric OS, FastIron, IronView, NetIron, SAN Health, ServerIron, TurboIron, and Wingspan are registered trademarks, and Brocade Assurance, Brocade NET Health, Brocade One, Extraordinary Networks, MyBrocade, and VCS are trademarks of Brocade Communications Systems, Inc., in the United States and/or in other countries. Other brands, products, or service names mentioned are or may be trademarks or service marks of their respective owners. Notice: This document is for informational purposes only and does not set forth any warranty, expressed or implied, concerning any equipment, equipment feature, or service offered or to be offered by Brocade. Brocade reserves the right to make changes to this document at any time, without notice, and assumes no responsibility for its use. This informational document describes features that may not be currently available. Contact a Brocade sales office for information on feature and product availability. Export of technical data contained in this document may require an export license from the United States government.