List of Criteria developed for Server Room and Data Center Audits.

Transcription

1 reliable data center List of Criteria developed for Server Room and Data Center Audits.

2 Preface The list of criteria was developed for server room and data center audits. It is intended to be used as a guideline for planning, implementation and acceptance testing of server rooms and data centers. Nowadays, the physical security and availability of IT is of vital importance to companies as many IT-supported business processes run as real-time applications. Interruptions would immediately result in inconvenient failures and possibly losses in production. Many banks and insurance companies include availability results in their risk assessments for loans and premiums. This new test method from TÜV Rheinland was developed due to growing market demands. It considers the previously known Basel III requirements for IT availability (effective from 2013). Pocket Overview of Classes The list of criteria is based on standards established by the German BSI (Federal Office for Information Security), Uptime Institute, TIA 942 (1) (2) and the publications of the German VdS institution (Property Insurers Association), in particular VdS This list of criteria allows comprehensive testing of IT infrastructure and operation. It is constantly updated. New rules and regulations are added at regular intervals. The list of criteria was developed in cooperation with Dr. Braun Consultants GmbH, Frankfurt am Main, Germany. Server room / data center n n n 2 (n+1) > 400 kva / to 320 kw / 200 m 2 CAT: Electrical supply Supply / medium-voltage supply X X X X X X 2 Transformers X X X X X 2 2 (n + 1) Low-voltage main distribution (LVMD) X X X X X 2 2 (n + 1) Stand-by unit (diesel) Provision X X X 2 2 (n + 1) Uninterruptible power supply unit (UPS) X X X X (n + 1) Data center power distributions X X n + 1 n (n + 1) Rack supply X X Air conditioning Coolers (chillers / cooling system) X X n + 1 n n 2 n 2 (n + 1) Computer room air conditioners X X n + 1 n n 2 n 2 (n + 1) Pump system X n 2 n 2 (n + 1) Piping X X X X Ring Ring Ring Building management system Operational threshold value display /indication X X X X X X Alarm messages by , SMS text message, alarm annunciator X X X X X Data recording X X X Evaluation option (ISO 50001) optional optional optional optional optional optional optional Maintenance Redundancy X X X X X Redundant supply paths X X X Maintenance during operation X X X Maintenance window X X X X

3 Site Selection Architecture / site selection / risks Proximity to flood hazard areas mapped on an official or insurance-related map (distance to coasts and waterways) no requirements not in a flood hazard zone not within 100-year flood hazard area or less than 100 m from 50-year flood hazard area not less than 100 m from 100-year flood hazard area Proximity to major traffic arteries or traffic routes with transport of dangerous goods no requirements no requirements not less than 100 m not less than 800 m Proximity to airports (entry lanes, missed approach procedure) no requirements no requirements not less than 1.6 km not less than 8 km Proximity to large cities (cable routes) no requirements no requirements not greater than 50 km not greater than 15 km Dangerous goods locations (gas pipeline, refineries, flammable liquids storage room) no requirements no requirements not less than 1.6 km not less than 8 km Military facilities and embassies, stadiums, places of assembly and locations/routes of rallies (external risk assessments) no requirements no requirements not less than 1.6 km not less than 8 km EMC active sources (transformer, transformer substation, track, transmitting stations) check check measurements recommended, constructional measures measurements recommended, constructional measures Contaminating emission of noxious substances (ship unloading stations, agricultural production, steel and galvanizing works) must be observed must be observed must be observed must be observed Seismic areas, earthquake zone according to DIN, constant ground vibration, collapse areas of buildings must be observed must be observed must be observed must be observed Third-party users in the building or on the premises (hosting companies) no restrictions possible if companies are non-hazardous possible if all tenants are IT or telecommunications companies possible if all tenants are IT or telecommunications companies

4 Building Construction Architecture / construction / DC & engineering Type of construction according to VdS / DIN (in accordance with applicable standards and guidelines) according to VdS / DIN (in accordance with applicable standards and guidelines) according to VdS / DIN (in accordance with applicable standards and guidelines) according to VdS / DIN (in accordance with applicable standards and guidelines) Walls encompassing the room F 30 - RC 1 F 90 - RC 1 F 90 - TC 2 F 90 - RC 2 Ducts F 30 at least: F 30; recommended: F 90 F 90 F 90 Floors and ceilings F 90 F 90 F 90 F 90 Floor and ceiling loads at least 5,000 N/m² at least 5,000 N/m² at least 10,000 N/m² at least 10,000 N/m² Suspended ceilings if sc exist for air conditioning, difficult to ignite B2 if sc exist for air conditioning, difficult to ignite B2 if sc exist for air conditioning, difficult to ignite B2 if sc exist for air conditioning, difficult to ignite B2 Roofs in accordance with provisions in accordance with provisions in accordance with provisions in accordance with provisions Clear ceiling height approx m approx m approx m to 3.50 m approx m to 3.50 m Clear raised floor height recommended: approx m recommended: approx m recommended: approx m recommended: approx m Raised floor understructure bonded pedestals, suspended U-type stringers bonded pedestals, suspended U-type stringers bonded pedestals, bolted U-type stringers bonded pedestals, bolted U-type stringers Raised floor tile design, incl. labeling and linking of tiles (smoke detection, extinguishing nozzles, water, equipotential bonding) difficult to ignite B1 difficult to ignite B1 difficult to ignite B1/A2 difficult to ignite B1/A2 Vapor barriers for walls and ceiling of server room no requirement no requirement no requirement recommended for data rooms Doors Fire resistance class at least: T30-RS; at least: T30-RS; recommended: T30-RD * recommended: T30-RD at least: T30-RS; recommended: T30-RD at least: T30-RS; recommended: T30-RD Door dimensioning at least 1 m wide and 2.13 m high at least 1.20 m wide and 2.25 m high (recommended height: 2.50 m) at least 1.20 m wide and 2.25 m high (recommended height: 2.50 m) at least 1.20 m wide and 2.25 m high (recommended height: 2.50 m) Single person security interlocks, turnstiles, portals or other systems ensuring that only one person can pass at a time no requirement no requirement yes yes No exterior windows in the server room area if ew exist, additional measures such as foil or auxiliary frames and grilles not permitted not permitted not permitted * T30: 30-minute fire resistance; RS: smoke protection; RD: smoke-proof

5 Building Construction (continued) Design provides protection against electromagnetic radiation no requirement no requirement yes yes Intrusion-resistant doors RC 1 RC 1 RC 2 RC 2 Entry lobby Physically separated from other areas of the data center no requirement yes yes yes Fire lobby separation from other areas of the data center F 30 F 30 / F 90 F 90 F 90 Multiple building entrances with security checks no requirement no requirement yes yes Single person security interlocks, turnstiles, portals or other systems ensuring that only one person can pass at a time no requirement no requirement yes yes Traffic and transport routes at least 1.00 m wide and 2.50 m high at least 1.00 m wide and 2.50 m high at least 1.20 m wide and 2.50 m high at least 1.20 m wide and 2.50 m high Elevators (if necessary) door width at least 1.00 m, cage depth 1.60 m, height at least 2.25 m, carrying capacity at least 1000 kg door width at least 1.00 m, cage depth 1.60 m, height at least 2.25 m, carrying capacity at least 1000 kg door width at least 1.00 m, cage depth 1.60 m, height at least 2.25 m, carrying capacity at least 1000 kg door width at least 1.00 m, cage depth 1.60 m, height at least 2.25 m, carrying capacity at least 1000 kg Parking Are visitor parking and staff parking areas separated from one another? no requirement no requirement yes, physically separated by fence or wall yes, physically separated by fence or wall Are visitor parking and staff parking areas separated from the receiving area? no requirement no requirement yes yes, physically separated by fence or wall Visitor parking in the proximity of data center exterior walls no requirement no requirement 10 m minimum distance 20 m minimum distance, separated by a barrier to prevent vehicles from approaching Administration, offices, security office, building services management system, staff rooms Physically separated from other areas of the data center Fire lobby separation from other areas of the data center F 30 F 30 / F 90 F 90 F 90 Security office Dedicated security office for security equipment and monitoring, incl. protection against break-in or small arms fire no requirement recommended if security office exists recommended recommended

6 Electrical Design Electrical design Total power balance (consider reserves and extensions) Supply by transformer substations 1 1, backup by generator 1, backup by generator 2 recommended, backup by generators System supply / topology spur spur spur, ring recommended spur, ring recommended Medium-voltage switchgear (MV) 1 1 1, recommended; 2, physically separated 2, physically separated Medium-voltage switchgear (MV) Transformers 1 1 1, recommended: 2 2 Stand-by diesel generators recommended 1 1, recommended: 2 2 Fuel stock 8 hours if available 24 hours 48 hours (2 tanks, incl. cross-connections; X-) 72 hours (2 tanks, incl. cross-connections; X-) Terminal for rental unit yes yes, if not available yes, for maintenance yes, for maintenance Low-voltage main distribution (LVMD) 1 1, recommended; 2 for Cat. 2+, maintenance bypass 1, recommended; 2 for Cat. 3+, maintenance bypass 2, maintenance bypass Separation of PS (secure power supply) in LVMD (sealing) or separate room no recommended yes yes System type TN-C/TN-S TN-C/TN-S recommended TN-S TN-S Reactive-current compensation / line filter recommended recommended recommended recommended UPS systems, incl. external bypass / redundancy n n+1, physically separated where necessary n+n, physically separated 2 (n+1), physically separated Battery backup time at least 10 min. (shutdown time) at least 10 min. (shutdown time) at least 10 min., physically separated at least 10 min., physically separated Battery monitoring / monitoring no no recommended yes Load bank for UPS no no provide connection provide connection Supply paths (protected against fire and sabotage) single, no requirement single, protection recommended 1 active and 1 passive, one supply path protected 2 active, at least 1 supply path protected Transfer switches (STS) for IT systems with only 1 power supply unit (dual-powered) no yes, recommended yes yes

7 Electrical Design (continued) Data center subdistributions (SD-UPS), separate supply from LVMD 1 1, recommended: Data center subdistributions (SD normal system) lighting flush-mounted sockets Measuring instruments (residual current measurement), no residual-current circuit breaker in the data center yes, recommended yes, recommended yes yes Rack supply by means of fixed connection or interlocked plug-and-socket connections (CEE) Grounding / equipotential bonding Lightning protection calculation according to VDE 0185 Part 3 Lightning and overvoltage protection lightning (coarse, medium protection) in distribution boards or terminal strips (fine protection) Load and power data measurement / logging / monitoring / supervision recommended recommended yes, measurement and logging yes, measurement and historical values Emergency lighting self-contained luminaires and additionally accumulator hand lamps self-contained luminaires and additionally accumulator hand lamps self-contained luminaires and additionally accumulator hand lamps self-contained luminaires and additionally accumulator hand lamps Escape lighting self-contained luminaires self-contained luminaires self-contained luminaires or central battery system self-contained luminaires or central battery system Selectivity calculation / short-circuit current calculation (network calculation and switch settings)

8 Air Conditioning Total power balance (update) Generator supply of air conditioning system yes, if generator exists yes yes yes Arrangement of IT equipment hot aisle/cold aisle recommended hot aisle/cold aisle hot aisle/cold aisle, recommended: hot aisle or cold aisle containment hot aisle/cold aisle, recommended: hot aisle or cold aisle containment Centralized / decentralized chillers (physically separated if necessary) n n+1 2n 2 (n+1) Recoolers / condensers / cooling towers / pumps n n+1 2n 2 (n+1) Recoolers / condensers / cooling towers (physical protection) Recoolers / condensers / cooling towers (design) outside temperature (higher than 32 C) Computer room air conditioners (CRAC) with operator display per line n n+1 2n 2 (n+1) Installation of computer room air conditioners (optimum airflow and maintenance) in the room in the room / mesh partition at least mesh partition / engineering block at least mesh partition / engineering block Automatic restart when power supply is resumed (chillers and CRACs) Redundant power supply of air conditioning (2 distribution boards and dual-powered) no check if this is necessary yes yes Lightning and overvoltage protection installed Redundant control of all plant components no check if this is necessary yes yes Manual operator control level if control system fails (chillers and CRACs) no recommended recommended recommended Automatic failure detection of a computer room air conditioner (CRAC) Top or bottom Venetian dampers for maintenance and in the event of a failure check if this is necessary check if this is necessary check if this is necessary check if this is necessary Connection to fault signaling system / BMS Condensate drain pan or water barrier under CRAC Chilled water piping system n n n 2n (ring system) Internal/external reserve terminals for mobile reserve units or extensions available no recommended recommended yes Strainer with bypass function yes yes recommended recommended

9 Air Conditioning (continued) Leak detection of piping system (at least pressure sensor) Chilled water pumps (phase-sequence test) n n+1 2n 2n System separation / heat exchanger (primary, secondary) no recommended recommended recommended Cooling water refill unit no recommended yes yes Hot water storage tanks / hydraulic separator system-dependent no check if this is necessary check if this is necessary check if this is necessary Chilled water system standby cooling (municipal water or well water, adiabatic cooling) no check if this is possible check if this is possible check if this is possible Connection of chilled water system pumps to UPS no check if this is necessary check if this is necessary check if this is necessary Lines carrying media in the DC area that are not allocated to data center equipment permitted but not recommended permitted but not recommended not permitted not permitted Drains in the server room for condensation water, humidification check if this is necessary yes yes yes Humidity control and monitoring (indication and historical values) % RH according to ASHRAE % RH according to ASHRAE % RH according to ASHRAE % RH according to ASHRAE Redundant room temperature and humidity monitoring (separate sensors) Outdoor air supply (filter classes G4 F7) no approx to 0.5 x air change / h approx to 0.5 x air change / h approx to 0.5 x air change / h Smoke & contamination detection of outdoor air supply recommended yes, at least smoke yes, test smoke, vicinity yes, test smoke, vicinity Positive pressure in the server room and allocated areas compared to outdoor and non-dc areas no requirement yes yes yes Integration of air conditioning into fire matrix (fire, extinction) System-wide leak detection of piping system / condensate (point detector or sensing cable) Ventilation, cooling of IT infrastructure (stand-by PS system, transformer, LVMD, UPS, batteries, extinguishing system, operating & other adjoining rooms) Consideration of alternative cooling systems (high density), where applicable in-row cooling, cool walls recommended recommended recommended recommended Can the waste heat of the data center be included in other energy balances? (heat pump, heating system etc.) recommended recommended, e.g. diesel preheating recommended, e.g. diesel preheating recommended, e.g. diesel preheating

10 Telecommunications / Cabling IT and telecommunications cabling / general Cabling design developed according to EN ? Requirements planning developed (EN n)? Cabling, racks, cabinets and cable trays comply with the applicable rules and regulations Redundant supplies (2-way) to the building, separated at least 20 m from one another, non-crossing no yes yes yes Redundant provider connection multiple provider connection, central offices, providers have right of way no no yes yes Infeed points protected against sabotage (covers not labeled, signaling contact, ) no no recommended yes Second telecom wiring closet no no yes yes Second wiring closet no no no optional Redundant backbone no no yes yes Redundant horizontal cabling no no no optional Routers and switches have redundant power supply units (alt. transfer switch) and processors no yes yes yes Multiple routers and switches for redundant operation no no yes yes Patch panels, outlets and cables have to be labeled according to ANSI/TIA/EIA-606-A. Cabinets and racks have to be labeled on front and rear Patch cords and jumpers labeled at both ends with the source / destination no yes yes yes Patch panel and patch cable documentation complies with ANSI/TIA/EIA-606-A. no no yes yes Are the lines tap-proof? Do the cable lengths meet the specification (ANSI, TIA 942)?

11 Telecommunications / Cabling (continued) Cable break monitoring of high-availability lines (backup lines)? Overvoltage protection for telecommunications supply cables (CU) Signal lines of energized lines installed separately or separated by separators on the cable trays? no no recommended yes

12 Safety and Security Safety and security / fire alarm and extinguishing systems /general Fire protection concept (incl. IT/infrastructure areas) Design conforming to VdS (fire alarm and extinguishing system), particularly reduced monitor ing areas and adjoining rooms (in accordance with applicable standards and guidelines) Fire matrix, incl. fire dampers Connection to alarm signaling system / BMS (building management system) Forwarding to a permanently manned location (gate keeper, lobby, security office, fire department) Fire detection Central fire alarm system complies with applicable guidelines Fire-alarm call boxes in room and raised floor (where applicable) and IT infrastructure areas Early fire detection systems (ASD aspirating smoke detector) recommended recommended recommended recommended Early fire detection systems (laser or LED) recommended recommended recommended recommended Firefighting Appropriate quantity and type of hand-held fire extinguishers (extinguishing agent) Mobile extinguishing units, e.g. 10, 20, 30 kg CO 2 trolleys no recommended recommended recommended Sprinkler system if necessary pre-controlled (if necessary) pre-controlled (if necessary) pre-controlled (if necessary) Preaction sprinkler system if necessary if necessary if necessary if necessary High-pressure water mist fire protection system option option option option Gas extinguishing system (chemical extinguishing agents: FM200, Novec 1230; gaseous extinguishing agents: CO 2, argon, Argonite, nitrogen, Inergen), incl. loss monitoring Oxygen reduction systems option option option option

13 Safety and Security (continued) Airtightness test (blower door test) Safe control of fire dampers Intrusion detection system / general Security concept zoning / division into areas (incl. DC area and IT infrastructure) The building is used as a data center only Design conforming to VdS (intrusion detection system and field devices) in accordance with applicable standards and guidelines Alarm matrix Connection to alarm signaling system / BMS (building management system) Forwarding to a permanently manned location (gate keeper, lobby, security office, police) Open/closed monitoring of doors (windows, channels, ducts) Monitoring of supply paths and protection against sabotage Monitoring of corridors and doors (passive trap system) Monitoring of rooms (comprehensive) Intrusion detectors at selected locations External access area surveillance (perimeter protection)

14 Safety and Security (continued) Access control system / flow of material / general Security concept zone model (incl. DC area and IT infrastructure) Assignment of access authorization is described in the company s security concept and monitored accordingly through management processes (27001) Design conforming to VdS (access control system and readers / cards) in accordance with applicable standards and guidelines Access authorization is issued in accordance with company policies no recommended yes yes Logging of access requests, incl. recording of unsuccessful attempts / access denied Alarm matrix Connection to alarm signaling system / BMS (building management system) Forwarding to a permanently manned location (gate keeper, lobby, security office) Connection of door opening systems (controllers, motor locks, door openers) to a fail-safe power supply Protection of central processing unit(s) / controllers in access controlled or monitored areas IT and facility staff have access to authorized areas only no recommended recommended yes Door blocking / pass-back / multiple use of access authorization excluded Turnstile no no yes yes Materials are unpacked outside the IT area and checked for foreign objects no recommended yes yes Bullet resistant walls, windows and doors Security counter in the lobby n/a n/a SK 3/NIJ level III (min) SK 3/NIJ level III (min) Security counter in the shipping and receiving area n/a n/a n/a SK 3/NIJ level III (min)

15 Safety and Security (continued) CCTV monitoring Building perimeter and parking no requirements no requirements yes yes Generators n/a n/a yes yes Access controlled doors no requirements no requirements yes yes Server room no requirements no requirements yes yes UPS, telecommunications and equipment rooms MEP rooms no requirements no requirements yes yes CCTV CCTV recording of all events on all cameras no requirements no requirements yes, digital yes, digital The outside areas are monitored by infrared cameras Recording rate (frames per second) n/a n/a 20 frames/sec (min) 20 frames/sec (min) CCTV recording is stored recommended recommended 30 days 30 days Safety equipment and security Water leak detection no yes yes yes Pressure drop in chillers no recommended yes yes Security staff on site no recommended recommended or located close by yes

16 Maintenance Preventative maintenance program Efficient preventative maintenance program (PM / Planned Maintenance), incl. type of maintenance, execution date and maintenance reports PM program includes maintenance activities recommended by manufacturer Detailed description of measures to be implemented when switching from redundancy <_> available and in use no recommended yes yes Complete description of measures for preventative maintenance (e.g., Method of Procedure manual Quality control process installed that confirms the implementation of a) all maintenance activities and b) the quality requirements of PM All PMs are connected to and controlled by a change management system Maintenance contracts for all components and their duration and service level monitoring according to management system (27001) Designated maintenance areas are available Cleaning and cleanliness Computer room floor / raised floor free from dirt and waste The data center is free from combustible materials, mailing boxes, personal belongings such as coffee mugs, microwaves,... Cleaners provide cleaning records to ensure a clean site

17 Maintenance (continued) Maintenance management system Efficient maintenance management system (paper or electronic format) to have a status overview of all maintenance activities (complete / in progress / scheduled) Maintenance list of all components installed (brand, model, date of manufacture, date of installation, characteristics in terms of operation, warranty information / terms and conditions, etc.) Records of tools and spare parts for preventative maintenance Power recording and maintenance trends for: Equipment, history of maintenance activities, calibration necessities, list of critical spare parts and threshold for ordering stock spare parts Manufacturer support List of qualified manufacturers for normal work and emergency measures SLAs describing work package, preventative maintenance program and response times for all critical manufacturers Description of manufacturer s technical support and support for authorized and qualified technicians reporting a fault Life cycle planning Effective process for planning, scheduling and identifying end-of-life elements of important infrastructure components

18 Maintenance (continued) Fault analysis program Compilation of lists of all failures, including date, time, infrastructure components and systems involved, computer system failures, identification of the underlying problem and lessons learned Development of a procedure to avoid underlying problems, evaluation of lessons learned and corrective actions derived from this evaluation Establishment of a trend analysis process recommended recommended recommended recommended Maintenance program Preventative maintenance program and planned maintenance; implementation rate greater than (>) 90 % yes yes n/a n/a Preventative maintenance; implementation rate: 100 % n/a n/a yes yes Efficient, scheduled maintenance program Does a service level agreement with a mobile generator supplier exist and are the response times adequate? yes n/a n/a n/a Continuous / other documentation All acceptance documents are available (fire protection analysis, commissioning documents, approved building application, attestation of conformity for VDI 6022/in accordance with applicable standards and guidelines) Regular generator load takeover tests, incl. reports Acceptance tests before commissioning the data center

19 Planning & Coordination Site policy Documented policy and procedures for the following points: - Performance of employees on site for overall onsite operation (configuration changes, operation under normal, emergency or abnormal conditions) - Site configuration: Configuring site for normal conditions standard operation: Changes in normal operation (e.g., switching chillers) - emergency procedure: Having control over the site under abnormal conditions or events - Change management: a) Control and approval of changes with regard to operational guidelines and b) identification of risks of planned changes Financial management OPEX and CAPEX planning to ensure compliance with TIER requirements OPEX and CAPEX planning to ensure compliance with TIER requirements, made at non-critical site and not in relation to other buildings or groups of buildings no recommended yes yes

20 Planning & Coordination (continued) Site documentation The following documents must be available (on site or accessible) - Working drawings - Operational and maintenance documentation - Records of soil samples, statics, electrical and mechanical design - Switches, electric circuits... - Documentation describing the system settings - Warranty documents and pre-negotiated maintenance contracts - Description of automatic operational sequences (e.g., CRAC standby -> live switchover) The above-listed documents and records are always available on site Operating documents centrally available as a library for on-site staff Master copies and changes are updated and available to operating staff and manufacturers, maintenance companies and planners Capacity management Data center space master plan exists and is permanently checked/supplemented Processes for planning further expansions for space, power, cooling for, e.g., 1/6/12/24/36 months in advance Resource management for floor space, power, cooling capacity and expansion (permanently monitored) no recommended yes yes no recommended yes yes

21 Employees & Training Employees Full-time employees (FTE) or half- or full-time employees of manufacturer who can identify critical states yes n/a n/a n/a Employees or manufacturer support for single-shift operation 5 days a week no yes n/a n/a Escalation and call-out procedure installed for specific employees or manufacturer support for described critical systems or equipment no recommended yes yes Employees on site 24 x 7, at least 1 qualified FTE n/a n/a yes n/a Employees on site 24 x 7, at least 2 qualified FTE for standard support per shift n/a n/a n/a yes Support of technical fields (electrical, control system, building management system, air conditioning) according to operational and maintenance requirements Qualifications Appropriate training of employees, required by law Documented on site training in data center processes and equipment Complete formal training of all employees in a) configuration, operation, hazards for the specific technical areas b) all site regulations, processes, procedures no recommended yes yes Tasks allocated to relevant employees for mainten ance, security, training and computer room engineering Shift personnel qualified for special shift tasks individually or as a shift team n/a n/a yes yes