Data Center Facility Basics Ofer Lior, Spring 2015 Challenges in Modern Data Centers Management, Spring 2015 1
Information provided in these slides is for educational purposes only Challenges in Modern Data Centers Management, Spring 2015 2
Outline Introduction Four main elements of a Data Center facility Total Cost of Ownership (TCO) Power Usage Effectiveness (PUE) Challenges related to improving PUE Optimizing cooling Optimizing power feeding Optimizing refresh rate Open Compete Project (OCP) Challenges in Modern Data Centers Management, Spring 2015 3
What is a Data Center? A Data Center is a facility used to house computer systems and associated components, such as telecommunications and storage systems. It generally includes redundant or backup power supplies, redundant data communications connections, environmental controls (e.g., air conditioning, fire suppression) and security devices. Large data centers are industrial scale operations using as much electricity as a small town. Challenges in Modern Data Centers Management, Spring 2015 4
How did Data Centers become the production floor of most companies? Data Center history and development: Started as a facility to house old complex computing systems Big size systems, required special environment in which to operate Many cables were needed A single mainframe consumed a lot of power and was hard to cool Security requirements for computing facilities started in the military computing environment Challenges in Modern Data Centers Management, Spring 2015 5
How did Data Centers become the production floor of most companies? Data center history and development: Data Center golden age started during the DotCom bubble Internet connection needed for business connectivity and stock exchange 24/7 companies representation on the net Data started to be centrally stored on computing devices and available on the web Cloud computing and Information explosion caused the Data Center industry to grow dramatically Data Centers are the engine behind all known application and web use-cases. 100% availability and connectivity data is available from everywhere, all the time Data is no longer kept on the personal device Challenges in Modern Data Centers Management, Spring 2015 6
How are Data Centers used by big enterprises? Build vs. rent vs. use hosting facilities Room/rack/servers/service hosting What are the Data Centers in Intel/Disney/Amazon/ Google/Facebook used for? Storage farms Computing farms Hosting farms SMB server room Visualized: ring around the world of data center power usage * * Different articles may show different numbers due to data confidentiality inside large organizations Courtesy of Business2community.com for educational purposes only Challenges in Modern Data Centers Management, Spring 2015 7
Four main elements of Data Center infrastructure 1. Power 1 MW = 10,000 x 100 Watt light bulbs / medium neighborhood Main element and cost of the Data Center operation 2. Cooling up to 50% of total Data Center power: Heat is created by the electronic elements of all computing devices. Multiply the heat of your laptop by many thousands to understand the challenge. 3. Space each rack space can cost up to $20,000/Y Floor space should be populated with as many devices as possible. Construction costs are high. 4. Network the veins and arteries of the data center Typical Data center power consumption for a non efficient facility Courtesy of blog.opower.com for educational purposes only Challenges in Modern Data Centers Management, Spring 2015 8
Data Center Power What consumes more power? OR Courtesy of wgt.co.il for educational purposes only Courtesy of brimag.co.il - for educational purposes only Behind every smartphone there are several hundred watts of server power!! The cost of a Google search is estimated at 0.0003 kwh of energy. Challenges in Modern Data Centers Management, Spring 2015 9
Data Center Power cont. Electricity used in global data centers in 2010 likely accounted for between 1.1% and 1.5% of total electricity use. For the US that number was between 1.7 and 2.2%. Growth in data center electricity use 2005 to 2010 by Jonathan Koomey Why do we need redundancy? Main elements in Data Center power designs Why is the Data Center power always growing? What causes inefficiency? Challenges in Modern Data Centers Management, Spring 2015 10
Data Center Power cont. Courtesy of buildings.com for educational purposes only AC/DC current conventions in Data Center operation Courtesy of embedded.com for educational purposes only Challenges in Modern Data Centers Management, Spring 2015 11
Data Center Cooling What creates heat? ~70% of all computing heat is created by the CPU The heat density of a CPU core is similar to the heat density of the sun s surface (energy per SQF) Charging and discharging capacitors even with minimal electrical resistance creates heat High 20Amp Current flowing in a 8mm footprint Every CPU core can consume 20 watts and reach 100 degrees Current leaks at the transistor level create heat Challenges in Modern Data Centers Management, Spring 2015 12
Data Center Cooling cont. How can heat impact Data Center uptime? How is heat evacuated and neutralized? Cooling methods Basic Data Center cooling diagram Cold water systems vs. DX units High-density computing challenges Courtesy of datacenterjournal.com for educational purposes only Challenges in Modern Data Centers Management, Spring 2015 13
Data Center Cooling cont. Cooling inefficiencies: Hot & cold air mix Over-cooling High cold air volume Heat exchanging cycles Electronics to air Air to water/gas Water to air Courtesy of electronics-cooling.com for educational purposes only Challenges in Modern Data Centers Management, Spring 2015 14
Data Center space and physical layout Data Center facility is extremely expensive to build. A single rack location construction cost can sum up to $80K* A single rack location maintenance, rent and power can sum up to $20K/y* Layout is arranged in hot/cold aisle design for better cooling efficiency * Estimation for best of breed data center facility Challenges in Modern Data Centers Management, Spring 2015 15
Data Center Network To be discussed in a coming lecture Challenges in Modern Data Centers Management, Spring 2015 16
Total Cost of Ownership (TCO) TCO the sum of initial capital expenditures (CapEx) added to ongoing and long-term operational expenditures (OpEx) Basic formula: Building the facility Buying the computing Power cost Cooling cost Space cost Network cost Staff Courtesy of http://datacenter10.blogspot.co.il/ for educational purposes only Challenges in Modern Data Centers Management, Spring 2015 17
Total Cost of Ownership TCO can be improved by the following: Balance between the requirements, flexibility and expected growth Set the right amount of redundancy DC efficiency management Smart CaPex investment Why does 1 percent in energy efficiency matter? Courtesy of www.datacenterknowledge.com for educational purposes only Challenges in Modern Data Centers Management, Spring 2015 18
Power Usage Effectiveness PUE a measure of how efficiently a data center uses energy; specifically, how much energy is used by the computing equipment (in contrast to cooling and other overhead) Pros A way to measure power efficiency, benchmark, and start improving Easy to measure and calculate at basic level Commonly used in IT industry by all main players Cons Hard to benchmark between different facilities creates deviations Does not take into account computing utilization, usage, and business value Complicated formula in mixed facilities Challenges in Modern Data Centers Management, Spring 2015 19
PUE not so simple Challenges in Modern Data Centers Management, Spring 2015 20
PUE benchmarking Challenges in Modern Data Centers Management, Spring 2015 21
Optimizing PUE it s all about power!!! PUE is a power efficiency performance indicator Three main ways to improve PUE 1. Optimizing cooling 2. Optimizing power feeding 3. Optimizing refresh rate PUE impacts the carbon footprint of the facility Total set of greenhouse gas emissions caused by an organization (tones per year) Challenges in Modern Data Centers Management, Spring 2015 22
www.epa.gov/cleanenergy/energy-resources/calculator.html Challenges in Modern Data Centers Management, Spring 2015 23
Challenge #1: Optimizing cooling Why is cooling an issue? Efficiency of cooling mechanicals efficient pumps, chillers, piping, and airflow Available solutions: Hot aisle can improve PUE from 3 up to 2.5 Hot/cold air containment can improve PUE from 2.5 up to 1.8 Free cooling can improve PUE from 1.8 up to 1.05 Data center physical location Building the data center where free cooling is available (ocean, cold areas) http://www.youtube.com/watch?v=4l0kpek7ibo http://www.youtube.com/watch?v=srn_xw7vtwc Challenges in Modern Data Centers Management, Spring 2015 24
Cold and hot air separation enclosing the hot/cold air Data center layout arranged in Hot/cold aisle method Hot air enclosure by chimney racks the room area is cold Cold air enclosure the room area is hot Challenges in Modern Data Centers Management, Spring 2015 25
Cold and hot air separation enclosing the hot/cold air Courtesy of psmtyech.com.sg for educational purposes only Challenges in Modern Data Centers Management, Spring 2015 26
Challenge #2: Optimizing power feeding Redundancy conflict uptime vs. efficiency and cost A 600Kw UPS consumes 10Kw (fans, logic ) in addition to 2%-10% of load for losses Dividing the load to 2 units multiplies the numbers above UPS connection mode can impact the losses Offline vs online mode Eco mode allows offline mode with clean power Challenges in Modern Data Centers Management, Spring 2015 27
Challenge #2: Optimizing power feeding AC vs. DC Data center using AC power typically converts power five times while a DC system usually requires only two conversions Redundancy & low density create inefficiency An under-loaded UPS is less efficient Pumps and AC units electrical efficiency Efficient motors Installation of VSD (variable-speed drive) Courtesy of digikey.com for educational purposes only Challenges in Modern Data Centers Management, Spring 2015 28
Advantages of DC power to a data center Lower line resistances Fewer power losses due to conversions The frequency of a DC system is zero Hz, eliminating synchronization procedures when connecting to the power grid. High-voltage DC can transmit large amounts of power over long distances with potentially lower capital costs and fewer losses than AC. DC power allows for simpler integration of renewable energy, such as photovoltaic. Simplified connection of energy storage devices such as UPS batteries Research on DC delivery system showed a minimum of 5%-7% efficiency gains comparing to an AC delivery system to a data center. Challenges in Modern Data Centers Management, Spring 2015 29
Challenge #3: Optimizing refresh rate Part of the data-center s CapEx Optimal lifespan of a server in a data center is about four years What creates positive ROI (Return-of-Investment) for 4-year refresh plan? Warranty costs Malfunctions increase in old servers Technology improvements enable refresh rates that are higher that 1:1 (e.g., 2:1) Purchasing less servers to produce the same or even higher compute power Higher rack density creates better power/cooling/space efficiency Courtesy of triplepundit.com for educational purposes only Challenges in Modern Data Centers Management, Spring 2015 30
Optimizing cooling Optimizing power feeding Optimizing refresh rate Better layout alignment could reduce PUE from 3 to 2.5 cold/hot air containment could reduce PUE from 2.5 to 1.8 Free cooling could reduce PUE from 1.8 to 1.05 *PUE numbers are industry estimations and are impacted by many variables Have the right amount of redundancy Manage the data center power consumption Efficient UPS units and transformers A DC Data Center is expensive but very efficient A four-year lifespan has a positive ROI server refresh is a good investment Part of the data-center CapEX Improvements in technology allows to purchase less servers and get same or high compute power The total efficiency of a data center should be calculated by comparing the business value it brings to the company divided by the TCO including run rate costs. Challenges in Modern Data Centers Management, Spring 2015 31
Data center energy productivity Courtesy of triplepundit.com for educational purposes only Challenges in Modern Data Centers Management, Spring 2015 32
Data center energy productivity cont. Equipment utilization has the highest impact on energy productivity Includes servers, storage, networks, etc. In the coming lectures you will learn (a lot) about utilization Challenges related to network management (design) in the data center Challenges related to resource management in the data center (job scheduling) Challenges related to data access in the data center Challenges in Modern Data Centers Management, Spring 2015 33
Open Compute Project (OCP) http://www.opencompute.org Challenges in Modern Data Centers Management, Spring 2015 34
What is OCP? Facebook-sponsored program aiming to change the way data-centers are built in order to scale computing infrastructure in the most efficient and economical way possible Include all DC aspects: Storage Networking Server Design Racks Physical infrastructures Hardware management All specs and documents are available online and open to all: http://www.opencompute.org/projects/ Project is the open source of DC industry with proven results of 38% less energy use with a 24% cost reduction in DC facilities Challenges in Modern Data Centers Management, Spring 2015 35
OCP: Design Designed in tandem with Open Compute servers The data center maximizes mechanical performance and thermal and electrical efficiency Open racks and modified 19 racks are used http://www.opencompute.org/assets/download/open-compute-project- Deploying-OCP-Hardware-in-a-Colo.pdf Challenges in Modern Data Centers Management, Spring 2015 36
OCP: Electrical OCP data center design is a highly-efficient electrical system The facility utilizes an electrical system with a 48VDC UPS system integrated with a 277VAC server power supply 2 choices of power supplies to the servers: OCP 700W-SH AC/DC power converter, a single voltage 12.5Vdc, closed frame, self-cooled power supply used in high efficiency IT applications OCP 450W power supply Challenges in Modern Data Centers Management, Spring 2015 37
OCP: Mechanical OCP data centers use a highly-efficient cooling system Mechanical system uses 100% airside economization with an evaporative cooling system All PUE data is open to the public and available online https://www.facebook.com/prinevilledatacenter/app_399244020173259 Challenges in Modern Data Centers Management, Spring 2015 38
OCP: Energy efficiency and green aspects OCP server's vanity-free design eliminates nearly 6 pounds of material per server, reducing the amount of materials that need to be produced, transported, assembled, and eventually disassembled OCP servers can operate in a higher-temperature environment, reducing the overall cooling load required in a data center (all non relevant parts are removed from server EX front panel PUE of 2 largest DCs is 1.1 and 1.08 WUE (Water usage efficiency is measured in real-time Data centers consumes 52% less energy than a data center built to code requirements, consumes 72% less water for occupant use, and meets 100% of irrigation needs through rainwater capture Challenges in Modern Data Centers Management, Spring 2015 39
Visit to Intel s largest data center outside the US Courtesy of itsupport24.com for educational purposes only Challenges in Modern Data Centers Management, Spring 2015 40