VOLUME 1. Physical infrastructure for a scalable, adaptable, efficient, reliable, predictable data center with 5 to 100 racks

Transcription

1 VOLUME 1 Optimized for Small and Medium Data Centers Physical infrastructure for a scalable, adaptable, efficient, reliable, predictable data center with 5 to 100 racks 2012 Schneider Electric. All Rights Reserved. Schneider Electric and APC are trademarks owned by Schneider Electric Industries SAS or its affiliated companies. CSI 2004 MasterFormat

2 Scope of this book This book is a specification for a small to medium data center of racks. For smaller installations, please refer to the APC Press companion book, System Specification and Project Manual, Volume 2: Optimized for Telecom / Networking / Server Rooms. For larger installations, this book cannot replace the services of a qualified consulting engineer, although it can be used as a reference and educational resource for the user as the system is being designed. Format of this book This book is directed toward the data center IT community it is therefore organized specifically toward the IT viewpoint of the data center design/ build process, not toward the general construction of a building. Facilities and trades personnel will notice that the format of this specification deviates from the way construction specifications are usually published, and does not strictly follow CSI division ordering, although all items are numbered according to CSI MasterFormat This book is divided into three parts: STANDARD PROCESS - Overview of the standardized data center design/ build project process (Section 2) To be read and reviewed by data center personnel This part of the book illustrates how project phases, steps, and tasks relate to each other in sequence, to accomplish the work needed to create the system. USER REQUIREMENTS Forms for gathering user-specific data for this project (Section 3) To be filled out by data center personnel with data center consultant The data gathered in this part of the book will be used to generate an appropriate design for the system. It also includes checklists for documenting who will assume the various roles and responsibilities of the project s process. STANDARD SYSTEM SPECIFICATION - The common rules for building any small/medium (Sections 4-11) data center To be reviewed by IT and Facilities This part of the book provides detailed specifications for building a standardized data center including power, cooling, fire suppression, security, management, and racks. This standard specification uses the 2004 edition MasterFormat CSI numbering system. The authors would like to acknowledge the work and dedication of the APC Data Center Science Center team that created this System Specification and Project Manual. Thanks to Neil Rasmussen, Chief Technology Officer and Suzanne Niles, Senior Research Analyst, for providing vision and a clear definition of processes. Thanks to Victor Avelar and Wendy Torell, both Senior Research Analysts, for the hours they dedicated to clarifying the user specifications section of this book. In addition, thanks to Dennis Bouley, Senior Research Analyst, for providing professional editorial expertise and for consistently advocating the customer s point of view. Finally, we would like to thank Paul Marcoux, former Director of the Data Center Science Center, for authoring the bulk of the standard specifications and for sharing his extensive knowledge of data center design drawn from his many years of experience in the data center design/build industry. The information contained in this publication is provided as is and has been prepared solely for the purpose of evaluating data center design and construction. American Power Conversion Corporation makes no representation and no warranty is given, either expressed or implied, as to the completeness or accuracy of the information contained herein. The information in this publication should not be used for financing purposes of any kind. In no event shall American Power Conversion Corporation be liable for any direct, indirect, consequential, punitive, special, or incidental damages (including, without limitation, damages for loss of business, contract, revenue, data, information, or business interruption) resulting from, arising out of, or in connection with the use of, or inability to use, this publication or its content, even if American Power Conversion Corporation has been expressly advised of the possibility of such damages. American Power Conversion Corporation reserves the right to make changes or updates with respect to or in the content of this publication or the format thereof at any time without notice. Copyright, intellectual, and all other proprietary rights in the content (including but not limited to software, audio, video, text, and photographs) rests with American Power Conversion Corporation or its licensors. All rights in the content not expressly granted herein are reserved. No rights of any kind are licensed or assigned or shall otherwise pass to persons accessing this information. This publication shall not be for resale in whole or in part. Rights granted American Power Conversion Corporation grants the user of this publication the right to copy any line or lines, diagrams, photographs, or designs in sections 4 through 10, provided the use of the copied material is for use within the user s specific evaluation, design, and construction project. Copied material shall not be used in whole or in part either electronically or physically for use outside a specific evaluation, design, and construction project or for individual or company material gain or volume reproduction. APC Press ISBN ii APC Part Number:

3 1 SYSTEM and PROCESS Preferences/Constraints General and System Level Room System Specification and Project Manual Physical infrastructure for a scalable, adaptable, efficient, and predictable data center Racks Power Cooling Management Fire Protection Physical Security 11 CSI 2004 MasterFormat iii Acronyms Glossary Appendices Index 12

4 CSI Division Locator This book is organized around the major elements of data center physical infrastructure Although a data center requires physical construction as with any facility, the taxonomy of its elements is slightly different from the order and naming of the CSI (Construction Specification Institute) MasterFormat system, which was designed to categorize elements of conventional brick-andmortar construction. Here is where the CSI MasterFormat divisions are located in the tabbed sections of this book: CSI Division Division 00 Procurement and Contracting Section of This Book Tab 2 Project Process Division 01 Tab 2 Project Process General Requirements Division 07 Tab 5 Room Thermal and Moisture Protection Division 09 Tab 5 Room Finishes Division 13 Tab 5 Room Special Construction Division 21 Fire Suppression Tab 10 Fire Protection Division 22 Tab 5 Room Plumbing Division 23 Tab 8 Cooling HVAC Division 25 Tab 9 Management Integrated Automation Division 26 Tab 7 Power Electrical Division 27 Tab 5 Room Communications Division 28 Electronic Safety and Security Tab 10 Fire Protection Tab 11 Physical Security iv

5 v Tab #1 goes here

6 What Is This Book? 1 This specification describes a small-to-medium data center constructed in a room within an existing building using modular, scalable, row-based rack/power/cooling architecture. This data center is defined using a point-by-point specification of its physical system in combination with Small/Medium Data Center a standardized process for executing the steps of its deployment racks The population of IT equipment to be supported by this physical infrastructure Does not include wiring closets or telecom rooms hence the power and cooling capacity required will have been determined by earlier needs assessment analysis, which establishes specific design requirements based on current and projected business needs of the organization. Whole-system approach The purpose of this data center specification is not to provide detailed descriptions of specific products, but rather to describe the essential requirements of performance, interoperability, and best practice that will allow all physical infrastructure elements to work together as an integrated whole rather than as a custom-engineered collection of loosely related components. Such a whole-system design optimizes functionality while reducing cost, and can be managed as a system using integrated management tools. Management Power Fire Security PHYSICAL INFRASTRUCTURE Racks Cooling Physical NCPI infrastructure Elements as are combined into an integrated system Services Standardized core of common requirements Nonessential variations in physical infrastructure from data center to data Fire Cabling and Security Racks Power and Cooling Physical Structure center common in current-generation data centers create a breeding ground for defects, delays, surprises, and human error. These costly effects of one-time engineering have been mitigated in this specification by creating a standardized core of best-practice requirements, which forms the central part of this book (Sections 4-11). Maximizing the extent of standardization in the data center increases reliability, speeds deployment, drives out defects and human error, reduces total cost of ownership, and fosters predictability of performance. User-specific requirements for the data center the essential design mandates that ensure the data center meets a user s particular needs regarding density, redundancy, power/cooling architecture, and so on are distinct from the common specification and are logged in a separate section (Section 3). These userspecific requirements serve as the missing piece that completes the standard specification and makes it an actionable, complete specification for the user s data center. Maximizing the standardized part simplifies the system and drives out defects Standard (common) specification User-specific elements 1

7 Continued PROCESS as part of the project This specification includes requirements not only for the physical equipment and construction of the system that is the outcome of the project, but also for elements of the process that carries out the execution of the project, such as scheduling, site access, installation, The commissioning, and training. As with the physical PROJECT (system) specification, some of these process Prepare Design Acquire Implement elements will be standardized, and some will be configured for the user-specific aspects of the particular project at hand. Multiple architectures supported The standard specification in this book includes performance requirements and best practices that will apply to any well-designed and constructed data center, as well items that can be selectively included to support the user s design decisions regarding a variety of alternative data center architectures, such as: Central vs. distributed UPS Central vs. distributed cooling Dropped ceiling vs. no dropped ceiling The PROCESS creates The SYSTEM Raised floor vs. slab floor Where there are choices to be made in the standard specification, the selection is based on the user-specific requirements recorded in Section 3. Mixed performance in the same room Since this specification includes a user-specific set of requirements for this particular project including a room layout drawing with row/rack particulars as specific as the user desires it allows for rows of racks, cooling equipment, and power equipment to be configured for differing density, availability, and redundancy according to the needs of the IT equipment supported by each row. For example, critical applications can have power, cooling, and network infrastructure configured for 2N redundancy (see page 99 for definition) and reside in the same computer room as non-critical configurations requiring only N+1 or N redundancy. Availability, redundancy, and density goals, and the resulting power and cooling configurations they require, will have been determined in earlier assessment and planning steps that establish the user-specific details of the system specification. 2

8 Criticality tags Criticality is a number from 1 to 4 representing how important the data center s operation is to the business, in terms of toleration for downtime. A data center s criticality level represents a major decision in the planning process since it impacts many other decisions including location, building type, fire suppression, security system, and many others. The planning phase allows designers to balance the total cost of ownership of a data center with the preferences and constraints of a business s availability requirements. It is through this iterative planning exercise that a final availability level is specified. In the standard system specification (Sections 4-11 of this book), criticality tags are used to mark items that are for critically levels higher than criticality level 1. Criticality level 1 is the default, if no tag is shown. Table 1 shows the meaning of these tags. Table 1-1 Meaning of criticality tags on Standard Specification items Criticality Tag Item applies to criticality level(s)... Lowest criticality No tag 1, 2, 3, 4 (All data centers) Highest criticality C 2+ C 3+ C 4 2, 3, 4 3, 4 4 Specification items apply to all data centers (criticality levels 1-4) unless there is a tag indicating that the item applies only to higher criticality. In the example below, item #1 applies to all data centers (criticality levels 1 through 4) because it shows no criticality tag. Item #2 applies only to data centers of criticality level 3 or higher. Example of criticality tag 1 Generator(s) installed outdoors shall be sheltered by an enclosure. 2 Walk-in enclosures shall house all generator mechanical, electrical, and fuel systems. C 3+ Criticality is an expansion of the familiar concept availability tiers. The selected criticality will determine the major characteristics of the system architecture, such as redundancy of power and cooling systems, as well as the robustness of system monitoring and various room construction details that affect reliability. Table 2 provides a brief summary of criticality levels. 3

9 Table 1-2 Summary of criticality levels For a complete discussion of criticality levels, see APC White Paper #122, Guidelines for Specification of Data Center Criticality / Tier Levels Criticality Level Business Characteristics Effect on System Design 1 (Lowest) Typically small businesses; mostly cash-based; limited online presence; low dependence on IT; and perceive downtime as a tolerable inconvenience Numerous single points of failure in all aspects of design; no generator; extremely vulnerable to inclement weather conditions; generally unable to sustain more than a 10 minute power outage (Highest) Some amount of online revenue generation; multiple servers; phone system vital to business; dependent on , some tolerance to scheduled downtime World-wide presence; majority of revenue from online business; VoIP phone system; high dependence on IT; high cost of downtime; highly recognized brand Multi-million dollar business; majority of revenues from electronic transactions; business model entirely dependent on IT; extremely high cost of downtime Some redundancy in power and cooling systems; generator backup; able to sustain 24 hour power outage; minimal thought to site selection; vapor barrier; formal data room separate from other areas Two utility paths (active and passive); redundant power and cooling systems; redundant service providers; able to sustain 72-hour power outage; careful site selection planning; one-hour fire rating; allows for concurrent maintenance Two independent utility paths; 2N power and cooling systems; able to sustain 96 hour power outage; stringent site selection criteria; minimum two-hour fire rating; high level of physical security; 24/7 onsite maintenance staff CSI MasterFormat 2004 numbering and the NCPI tag MasterFormatTM is a numbering system created by the Construction Specifications Institute (CSI) for categorizing construction specifications, much like the Dewey Decimal System for books in North America and Europe. MasterFormat 2004, used in this specification, is the latest edition of this numbering system. While the physical infrastructure elements of a data center do not align perfectly with the divisions and hierarchy of MasterFormat, the specifications in this book have been aligned and labeled as closely as practicable with MasterFormat numbering. To avoid possible conflict with an organization s other, non-datacenter construction specifications (for example, in a construction NCPI stands for network-critical physical What is NCPI? infrastructure, a term sometimes used to project involving both a data center and the building that contains it) refer to the physical infrastructure of a data this book has assigned new, data-center-specific MasterFormat center (the subject of this book). NCPI includes power, cooling, racks, fire subsections in the appropriate categories, using the MasterFormat protection, physical security, management, and services. rules for creating new subsections. All data center specification topics are tagged with NCPI in their CSI title, to distinguish them from any similar specifications that apply to ordinary building construction. 4

10 For organizational clarity the sections of this book reflect a data center perspective in the ordering and categorization of the major infrastructure elements (power, cooling, management, etc.). While all specifications appear in their correct MasterFormat divisions, the divisions themselves do not appear sequentially in their MasterFormat numbered order. The correspondence between MasterFormat division numbers and the sections of this book are shown on in the CSI Division Locator list on the back of the table of contents page. How to mark items that do not apply, or conflict with other items When using the standard system specification (Sections 4-11) as a basis for contract work, there is an industry convention for marking items that do not apply to your project. Such items might represent higher criticality than is needed, or they may be in conflict with other items that represent an alternative implementation (as in the choice of fire suppression techniques), or may not be applicable to the project for some other reason. The proper way to eliminate an item is to cross it out and mark it NIC (not in contract) as shown below. This retains a record of what was eliminated from the contract. Figure 1-1 Correct method of eliminating items from contract specification NCPI Preaction Sprinkler Systems 1 Sprinkler system shall be preaction type which shall not allow water to enter the sprinkler piping during an accidental breakage of a sprinkler head. 2 Sprinkler system shall be double-acting preaction type which shall combine preaction and smoke detection. 3 Sprinkler actuation system shall be armed by the initiation of an alarm from a cross-zoned detection system. 4 Sprinkler system design density shall conform to the owner s / owner representative s insurance company, lease agreement fire protection requirements, and local and national codes. 5 All sprinkler system plans and calculations shall be submitted to the authority having jurisdiction (AHJ) for review and approval prior to installation. 6 The following sprinkler system controls shall be included in fire alarm control panel: NIC a b c water flow switch PIV (post indicator valve) tamper switch sprinkler piping supervisory air pressure switch 7 In data centers with no suspended (drop) ceiling, sprinklers shall be configured upright. NIC 5

11 Tab #2 goes here 6

12 Project Process 2 Specification of the execution of this project that supports and carries out the The success of the project depends not only on the acquisition and deployment of hardware, but equally upon the successful execution of the process that guides the project through its phases from concept to deployment. The process creates the system (Figure 2-1.) Figure 2-1 A project is comprised of the system plus the process that creates it The PROJECT Prepare Design Acquire Implement The PROCESS creates The SYSTEM Every step of the process must be clearly defined, assigned to an owner, and tracked. Responsibilty for the various process steps can be assumed by the user or outsourced to a hardware vendor or to a third-party service provider. Regardless of who does what, it is essential that every step of the process be accounted for and executed at the appropriate time, with each step providing input to subsequent steps that depend on it. 7

13 Process - General 1 Process shall be configured from the standard process map of Figure 2-2 as appropriate to the project: 2 All process steps in Figure 2-2 shall be evaluated for relevance to the project, and unneeded steps deleted. 3 Additional steps shall be added as necessary to handle extraordinary, non-standard activity. 4 Process flow and timing dependencies shall follow Figure 2-2, specifically: a b c Process activity shall flow through all phases in sequence, left to right. Each phase shall have all its steps (listed vertically below it) completed before proceeding to the next phase. Asynchronous process elements, which handle unplanned changes and problems that can occur anywhere in the process, shall be executed as needed. Figure 2-2 Standardized project process Milestones Each phase ends with a milestone Project change management Product defect correction Process defect correction Can occur at any point in the process For more about this standardized process, see APC White Paper #140, Data Center Projects: Standardized Process 8

14 5 Project information shall be maintained on a secure website accessible to all parties to the project. Website information shall include: a b c Ongoing project status For each process step, intermediate and final data and reports The ability to accept feedback, comments, requests, and problem statements 6 Each step shall have the following elements (Figure 2-3) a b c d An owner A task list, consisting of work instructions and a checklist of specific actions to be completed A list of INPUTS, which are the outputs of previous steps A list of OUTPUTS, which become the inputs to subsequent steps Figure 2-3 Step detail Task Work Instructions Task Checklist INPUTS Data from previous steps OUTPUTS Data needed by subsequent steps This step owned by For more about process steps, see APC White Paper #140, Data Center Projects: Standardized Process 9

15 Task Detail The bulleted items in Figure 2-4 below are the tasks that comprise each step of the standardized project process. Each box is a STEP Bulleted items are TASKS Figure 2-4 Task detail of process steps 10

16 Table 2 Sample responsibility list for project steps This table, or one like it, shall be filled out as part of the user-specific information in Section 3 Who Will Do It? Process Step User ( ) Primary Equipment Vendor ( ) 3 rd Party (Who?) Not needed (X) Assess needs Develop concept Initialize process Establish Requirements Configure Solution Finalize proposal Create P.O. Site coordination Shipment Assembly Installation by subcontractors Startup Network Integration Orientation & training Project changes Product defect correction Process defect correction 11

17 Steps General Requirements 1 Every step shall be assigned either to in-house resources or to an outside service provider, to ensure that all activities of the project are accounted for and executed (see Table 2-2). 2 All steps shall include, at a minimum, the items listed in the step descriptions that follow, as applicable to the project [Step descriptions to be included in a future revision of this book] Role of Standard System Specification Within the Project Process The Standard System Specification (Sections 4-11 of this book) plays a key role in the planning sequence, which is the portion of the project process that transforms the system from idea to detailed design. This role is illustrated in Figure

18 Figure 2-5 Role of Standard System Specification within the project process 13

19 Tab # 3 goes here 14

20 User-Specific Data 3 15

21 NCPI Project Title Page Fill in below, or replace this page with user s page containing the title and executive summary of this project. The title of the project should include the name of the company along with a unique site identifier for example ACME Corporation - 1 Main Street The executive summary should include the following: Description of the data center project for example High density 200 kw criticality 2 Brief purpose of the data center project for example, consolidation, expansion, updating, retrofitting Estimated timeframe for the completion of the project of this project of this project 16

22 Purpose The purpose of this section is to gather key user-specific information, as part of the system planning sequence (see Figure 2-5), for the development of a detailed data center construction design. The information gathered in this section is critically important to achieving the business objectives of the data center. Context This section addresses user requirements, which are preferences and constraints ranging from high level business considerations (i.e. what is the expected IT growth?) to specific design elements (i.e. raised floor or not?). User requirements (this section) are distinct from the standard specification (Sections 4-10). User requirements will vary from project to project. Standard specifications, for the most part, will remain the same from project to project. These user requirements combined with the standard specification serve as a complete specification that provides the rules by which this data center will be designed and built. Who should fill in these forms? Various individuals who play important roles in the data center design/build or upgrade project should be consulted in order to complete this user-specific data center project specification. The IT Parameters section should be completed by outside vendor/partner in conjunction with data center management including CIOs, VPs, general managers, and other individuals who understand the core business needs and objectives. The IT load profile requires knowledge of specific current and future kw values. The System Concept section should be completed by outside vendor/partner in conjunction with IT and facilities management including data center manager, IT manager, IT director, IT consultant, and other personnel that help execute the strategies determined by executive management. The User Requirements section should be completed by an outside vendor/partner in conjunction with IT and facilities operating personnel including network engineers, facilities engineers, and other personnel dealing with the day-to-day data center activities. 17

23 Instructions The forms in this section are designed to facilitate the gathering of user preferences and constraints. They are partitioned into four sequenced levels: Level of user input information Filled out during this step of the project process... For more about the project process, sec Section 2 of this book IT parameters DEVELOP CONCEPT Prepare Design Acquire Implement System concept DEVELOP CONCEPT User requirements ESTABLISH REQUIREMENTS Phase-in plan ESTABLISH REQUIREMENTS Each level consists of user-supplied information that is entered into one or more forms. The information provided at each level affects the levels that follow it, and so must be carefully considered. Inaccurate data in the IT parameters form, for example, can misdirect subsequent downstream activity (i.e. system concept) resulting in design strategies that may be unnecessary, expensive, or even impossible. In some cases (i.e., small organizations) the outside vendor/partner in conjunction with a single individual may be able to fill out the forms. In other cases, the outside vendor/partner may need to work with a group of individuals in order to properly complete the forms. 18

24 1 IT Parameters The IT Parameters form, Form 3-1 below, should be filled in jointly by management (or individuals that have knowledge of a general IT need or an overriding business need) and the outside vendor/partner. This form will involve an analysis of the IT criticality, IT capacity, and IT load profiles for the new data center or for the upgraded existing data center. Guidance for choosing the appropriate IT criticality level for the business can found in APC White Paper #122, Guidelines for Specifying Data Center Criticality / Tier Levels. The IT load profile represents the expected IT load over the data center lifetime, and is expressed by four numeric parameters. Figure 3-1 assists in creating the IT load profile. For further guidance on creating an appropriate IT load profile, see APC White Paper #143, Data Center Projects: Growth Model. Form 3-1 IT parameters IT Parameters Data / Input Value Guidance IT criticality level IT load profile Initial IT load (kw) A goal for the availability and reliability of the data center, consistent with the business mission - See APC White Paper #122 for guidance in choosing an appropriate IT criticality level See APC White Paper #143 for guidance in creating an appropriate IT load profile The IT load of initial installation Maximum final IT load (kw) The maximum anticipated IT load Minimum final IT load (kw) Average final IT load (kw) Ramp-up time to final IT load (yrs) Has data above been validated? Yes No #DIV/0! The minimum anticipated IT load The average of min and max anticipated IT load The time it takes to go from initial load to final load Cost drives many of the choices above. Therefore, there may be a few iterations of this data before a final decision is reached. 19

25 Figure 3-1 IT load profile NCPI capacity plan (determined later) kw MAXIMUM final load INITIAL load MINIMUM final load RAMP-UP time Time For more about this growth model, see APC White Paper #143, Data Center Projects: Growth Model 20

26 2 System Concept The system design concept form, Form 3-2, should be filled in jointly by IT and facilities management (or individuals that have knowledge of the physical location of the proposed data center) and the outside vendor / partner. The information gathered in this form, as well as the previous form, is used to select a reference data center design that establishes the basis for the new data center. Form 3-2 System concept System Concept Data / Input Value Guidance What is the name of the room where the data center will be installed? Is a cooling solution required for this project? Yes No What existing heat rejection mediums are available at the site? Chilled w ater Glycol Refrigerant Water cooled Is a UPS solution required for this project? Yes No What is the target power density? Less than 3 kw / rack 3-10 kw / rack kw / rack The location where the proposed data center solution will be installed. This should be an alpha-numeric label understood by the end user. Must the reference design include cooling? If NO, skip next question and ensure that the existing cooling system has enough bulk capacity and redundancy, and air distribution for the current project phase. Identifies the types of heat rejection systems that pre-exist in a facility. This will help in choosing the reference design with a compatible cooling system. Must the reference design include power? If no, ensure that the existing UPS system has enough bulk capacity and redundancy, and distribution for the current project This is a target density only for the purposes of helping to select a reference design. Reference design number Enter the reference number from the options selector i.e. high density, high criticality Reference design description reference design Could be design that was built already. Could be from owner's existing Source of ref design design, from InfraStruxure Designer tool, or reference selector. Free form text section (special issues) that allows for comments on the chosen ref design. i.e. I want to use tall vs. short racks this time for this ref design. 21