Keyword: TIA 942, Tier level, Nuclear Data Center. Introduction

Nuclear Data Center International Standard Towards TSO Initiative Raja Murzaferi Raja Moktar, Mohd Fauzi Haris, Siti Nurbahyah Hamdan, Saaidi Ismail,Radhiah Jamaluddin, Nursuliza Samsudin. IT Center, Technical Support Division, Malaysian Nuclear Agency, 43000 Kajang, Selangor Nuclear Data Center is the main facility for Nuclear Malaysia Agency IT infrastructure comprising of main critical servers, research and operational data storage, HPC-clusters system and vital network core equipments. In recent years, international body such as TIA Telecommunication Industry Association and Uptime Institute have came out with proper international data center standards in order to ensure data center operation on achieving maximum operational uptime and minimal downtime. The standard are currently being rated as tier level ranging from Data Center tier I up to tier IV, differentiate by facility standard and uptime/downtime percentage ratio. This paper will discuss Nuclear Data Center adopting international standards in supporting Nuclear Malaysia TSO initiative thus ensuring the critical core component of agency s IT services availability and further more International standard recognitions. Keyword: TIA 942, Tier level, Nuclear Data Center Introduction Data center is a facility design and built to house computer systems or servers. It comprises of several critical components in order to support for the servers operations. It encompass of network or data communication equipments, environmental control systems, power and backup power systems and security devices. Each of it has key components of ensuring the availability of the data center operation and the computer system reside in it. Data Centers were built as a large computer rooms during the early ages of computing industry whereas the design and hardware technology of the older days was large. ENIAC for an example was a computer for US military purpose took one whole floor of a building covering 1800 square feet [1]. As computer technology evolved, computers have become smaller yet powerful in term of processing power. It has attracted not only special users such as military or scientific purposes but expanded to nearly every aspect of human lives. As daily business operation and research tasks have become majorly relying on computers system services running on servers, the role of data center infrastructure to house the computer systems or servers have become increasingly critical whereas availability of the computer systems have become one of the main issue. This translate to any operation disruption of computer system on daily production basis means loss productivity or profits. Other than computer system reliability, the data center infrastructure housing the computer system have been increasingly critical in providing the best environment for the computer system to operate in giving maximum reliability and availability of service. Realizing this critical role, several bodies of profit and nonprofit have came out with data center design standards and best practice. One such standard were develop by Telecommunication Industry Association the TIA-942.[2] TIA-942 Data center was historically developed without a proper standard [3]. This let network or system administrator with many challenges for choosing implementations of data center development for secure and reliable data center facility on both the existing or future needs. In 2005, The TIA-942 standard was published known as TIA-942; Telecommunications Infrastructure Standard for Data Centers. It is also known as ANSI/EIA/TIA-942. It was intended to provide requirements and guidelines for designing and installation of data centers. It facilitates planning on the early stage in the development process of a data center such as architectural, facilities and IT components. It is also intended for designers to have better understanding of the data center design including facility planning, 1

cabling systems and the network design [3]. This paper will be describing the following TIA-942 components that can be applied to the Nuclear Data Center infrastructure development: Site place and layout Cabling infrastructure Environmental Considerations/Cooling Power/Electrical Tiered Reliability Level Pusat Data Nuklear Pusat Data Nuklear or Nuclear Data Center is located in block 15 in the second floor. It is comprises an area of 36 square meter with an L shape. The height is 2.2m while raised floor is 18cm. It is currently housing 34 service and production servers, a High Performance Computing (HPC) system for nuclear simulations purposes, central storage servers, Tape library backup servers and core networking facilities for the whole agency. These entire machines are mounted in 4 standard server racks, 2 non standard server racks and 2 networking racks. The data center is remotely managed through remote server software or through the control console resides within the data center itself. Research and daily operational software are run through the server within the facility on 24x7 basis. Storage system stores valuable researcher s research data and administration critical and sensitive operational data. ACSU-2 RAK L1 RAK L2 MEJA KVM PINTU 1 Fiber termination Sensor RAK A5 RAK A4 RAK A3 RAK A2 RAK A1 RAK N1 RAK N2 ACSU-3 ACSU-4 Sius Utama AC DB Server ACSU-1 PINTU 2 Figure 1: Nuclear Data Center simple layout The Nuclear Data center facility includes environment control non precision air conditioner with alternate unit operations schedule. Main air conditioner units are two 5HP with one standing unit while the other is vertical wall mount unit. Other supporting units are 1 unit of 2HP of wall mount and another 3HP ceiling mount unit air conditioner. The air circulation for hot and cold aisle was problematic during the previous setup, whereas one of the main 5HP units was not optimizedly arrange. The L shape room design was also contributing to the low cooling efficiency. After adding HPC system in the mid of 2010, the problem have become worsen whereas the inefficient cooling capacity could not cater the equipment load. This leads to major downtime during the last quarter of 2010 due to heat. The problem was resolved by adding an additional 5HP non precision air conditioner and reconfigured the placement of the existing unit to a more strategically location. But the last upgrade also marked the full capacity of the Nuclear Data Center, where the L shape design has constricted many cooling equipment upgrade options. 2

Any further additional heat generating equipment or servers that exceed the current cooling capacity should be strictly avoided. Cabling for networking infrastructure are Category 6 or Cat6 non-fully certified and non-fully managed cables (no proper tray managing cables under the raised floor). Category 5e or Cat5e cables still exist in connecting old server systems. Basically networking cabling is a non-fully structured based. Horizontal cabling is only being partially deployed on 4 server racks while another 2 racks; whereas the most recent additional servers system that was acquired was not setup with properly horizontal cabling. Vertical or backbone cabling has been laid for inter-floor networking for Blok 15 using Cat6 while backbone for main distribution blocks (Block 11, Block 44, Block 18 and Block 29) are using mixed single and multimode depending on the distance of the blocks. ACSU-2 RAK L1 RAK L2 RAK L1 RAK L2 Cold Air Hot Air Cold Air Hot Air MEJA KVM MEJA KVM PINTU 1 Sensor ACSU-1 PINTU 1 Sensor RAK A5 RAK A4 RAK A3 RAK A2 RAK A1 RAK A5 RAK A4 RAK A3 RAK A2 RAK A1 Fiber termination RAK N1 RAK N2 ACSU-3 ACSU-2 Sius Utama AC DB Server Fiber termination RAK N1 RAK N2 ACSU-3 ACSU-4 Sius Utama AC DB Server ACSU-1 PINTU 2 PINTU 2 Figure 2: Previous Cooling System arrangement; main cooling 5HP unit ACSU-1 is non-optimized located blowing cold air directly to hot aisle at the back of server rack, resulting average temperature of 27C. Figure 3: Latest arrangement with the additional 5HP ACSU4. Optimized location blowing cold air at the cold aisle in front of the server rack whereas cooling system units run at rotation of 6 hours schedule. Average temperature at 21C. Power cablings are currently not managed according towards data center standard. But powers are backed up with a non dedicated data center generator set that can supply backup power to the whole building and data center during power outage. Uninterruptable Power System or UPS are currently based on non centralized units. With current load and numbers of servers majority can sustained a backup power not less than 15 minutes; in the mean time for the generator set to start and take over for backup power purposes. Both networking and power cablings are laid through under floor cablings but without totally proper trays or trunking. Site Security is managed through card access entrance whereas only permitted staffs have the access and video camera monitoring through L-COSMOS system. While firefighting equipment, smoke and heat detectors have been deployed with Pyrogen tm gas system has been setup for fire extinguisher purposes. As for the tiered reliability based on current infrastructure setups regardless of total downtime, Nuclear Data Center is situated in tier I rating, whereas one of the major identified fallback was in the single points of failure in core facilities such as network equipments and cablings. 3

Data Center Towards TSO Currently IT services in Nuclear Malaysia has been considered as an integral part of nearly every aspects of the agency research and daily administrational operation activities, while at the heart of the IT services is the Nuclear Data Center. In order to support TSO initiatives, rapid growth of research and operational activities, the Nuclear Data Center is proposed to be redesign and upgraded according to TIA-942 standards. This is to ensure maximum availability of service can be offered to the agency. It is also to ensure the right path or base of evolving for data center infrastructure in the future roadmaps. By adopting international recognized standard, it also means that the Nuclear Data Center will be recognized and acknowledge by overseas or international counterpart. Main aspect of Nuclear Data Center that be look upon into, in order to achieve the standard are: Site place and layout Future Nuclear Data Center layout design must be acceptable to initial deployment cost and anticipated requirement in the future. Proper space allocation for data centers starts with ensuring that space can be easily reallocated to changing environments and growth. It should be design with modest of flexible empty space that can accommodate future racks or cabinets. Functional areas must be established for anticipating data center growth and environment where software or hardware upgrades or future deployment only causing minimal downtime. The functional areas are ideally to be separate rooms, but it is sometime not practical for normal organizations; whereas it can be consolidated with defined areas. Some of the functional areas are the entrance room, main distribution area, horizontal distribution area, equipment distribution area and zone distribution area. Functional Area Entrance room Main Distribution Area Horizontal Distribution Area Description Location for access provider equipment and demarcation points. The standard recommends location on the outside of the data computer room for better security. Shall be consolidated with Main Distribution Area if located in the computer room Area for core routers and switches for LAN and SAN infrastructures. It houses the main cross-connect. It may include horizontal cross-connect for Equipment Distribution Area. The standard requires of at least having one MDA and separate racks for fibers and UTP installations Distribution point for horizontal cabling and cross-connects for distributing the Equipment Distribution Area. Equipment Distribution Area Area for server racks and cabinets with the standard specifies the hot and cold aisles configurations for heat management. Zone Distribution Area Optional Area for interconnection points in horizontal cabling between Horizontal Distribution Area and Equipment Distribution Area. Table 1: Data Center Functional Area Architectural considerations such as operation center setups, door and windows specifications, UPS and batteries room, service and exit corridors, holding and staging area must also critically take into account.[4] Other consideration are on site considerations of the data center is the location whereas if possible avoid any restriction of 4

expansion, having redundant access, no exterior windows for avoiding heat and security risk, standard ceiling height, flooring and wall specifications. Cabling infrastructure Nuclear Data Center also must be design with proper cabling infrastructure. Standard cabling design should either adopt the proper overhead or under floor cabling system. The horizontal and backbone cabling must be properly deployed into functional areas accordingly. The TIA-942 also specifies networking cablings standard based on existing TIA-568 and 569 specifications. Some of the recognized permanent telecommunications cabling system that the standard provides specifications are: Standard single mode fiber 62.5µm and 50µm multimode fiber Laser-optimized 50µm multimode fiber (recommended) 75-ohm coaxial cable (recommended for E-1, E-3 and T-3 circuits) 4-pair Category 6 UTP and ScTP cabling (recommended cat6) The standard recommends backbone cabling using laser optimized 50µm multimode fiber. It can achieve longer distance and higher speeds with more cost-effective implementation than single mode fiber. While for horizontal cabling it is recommended in installing the highest media available to reduce the need for future re-cabling and the current highest media recognized by the standard today is the Category 6. A maximum distance for maximum backbone and horizontal cabling is generally 300m for fiber based cabling on backbone and 100m for copper based horizontal cabling. Figure 4: TIA-942 compliant key functional area with cabling deployment [3] Environmental Considerations-Cooling Environmental considerations for data center includes but no limited to, fire suppression, humidity levels, operations temperatures, architectural, electrical(power) and mechanical system specifications. Some of the requirements are 5

dependent on the desired reliability tier described. This paper for discussion purpose will on power and cooling issue. The physical environment of a data center must be always kept control. Data center cooling system or air conditioner is not only for controlling the temperature; it must also manage the humidity of the environment. In ASHRAE s thermal guidelines for Data Processing Environments, recommends a temperature range of 16-24C and humidity range of 40-55% with a maximum dew point of 15%.[7] Data center temperature will rise because of servers or other electrical equipments operate in it. The heat generated by the equipment must remove in order to avoid electronic equipment malfunction. Data center air conditioning also control humidity by cooling return space air below the dew point, whereas humidity if not control, it can make the environment either too much humidity, where water may condense on internal electronic components or too low humidity can result on static electricity discharge. Power/Electrical Data Center is basically powered by electricity. In order to keep the data center operation at an optimal level, electrical power must be design to be continually supplied and backup. Main in-coming power will be from the main power source that is the Power Company supplying electrical power; TNB, while for backup power it consists of one or more UPS and diesel generators [8]. In order to prevent any single point of failure in power supplying to the data center, all elements including backup systems are typically duplicated where critical electronic components such as servers, networking peripheral or the cooling systems are connected with more than one power feeds. This design is made to achieve N+1 redundancy in the system. Figure 5: Design considerations for air circulation, telecom/networking and power cablings [3] Tiered Reliability Level Nuclear Data Center shall be rated through tiered reliability whereas it has to confirm with certain specifications and technical design need. The tier reliability level also can be use by Nuclear Data Center as a roadmap of infrastructure specifications development according to organization needs. The tier level is basically to provide the means for specific data center needs in order to give certain achievable availability of service. TIA-942 standard specifies an informative annex with data center availability tiers. The tiers are based on information from the Uptime Institute (consortium providing best practices and benchmarks comparisons for data centers). The standard describes four level of tier with the higher tier having higher service 6

availability. Each tier was described in the standard annex with detailed architectural, security, mechanical and telecommunications need. Tier descriptors also include information like raised floor specifications, watt per square foot and points of failure. Whereas for redundant purposes, the standard recognized it as Need or N to indicates the level of redundant with N representing only the system need. Tier Level Requirements 1 Single non-redundant distribution path serving the IT equipment Non-redundant capacity components (N) Basic site infrastructure guaranteeing 99.671% availability 2 Fulfills all Tier 1 requirements Redundant site infrastructure (N+1) capacity components guaranteeing 99.741% availability Conclusion 3 Fulfills all Tier 1 and Tier 2 requirements Multiple independent distribution paths serving the IT equipment with one active includes redundant component (N+1) All IT equipment must be dual-powered and fully compatible with the topology of a site's architecture Concurrently maintainable site infrastructure guaranteeing 99.982% availability 4 Fulfills all Tier 1, Tier 2 and Tier 3 requirements All cooling equipment is independently dual-powered, including chillers and heating, ventilating and air-conditioning (HVAC) systems 2(N+1) Fault-tolerant site infrastructure with electrical power storage and distribution facilities guaranteeing 99.995% availability Table 2: Tier level and its requirement simplified [8] As Nuclear Malaysia Agency evolves towards the TSO initiative, the role of IT in supporting the organization has grown tremendously. IT services have to be at the peak of its performance to serve as an integral part of either research or organizations operational purposes. At the heart of the infrastructure of IT services is the Data Center. Nuclear Malaysia has come a long way on developing it data center infrastructure, as organization has increasingly grows, the need of developing high availability services has come to its critical points. Design and development need to confirm with international standard and considerations in order to achieve optimum data center operational service support for the organization, proper development roadmap and further more recognition by the international counterpart. Acknowledgements The authors wish to express their sincere thanks to Engineering Department, Electrical and Mechanical unit, of Nuclear Malaysia for the technical support on the Data Centre Environment Control issue. 7

References 1. Bellis M. The History of the ENIAC Computer: John Mauchly and John Presper Eckert, About.com Guide, retrieved from http://inventors.about.com/od/estartinventions/a/eniac.htm 2. Data centers, In Wikipedia. Retrieved July 3, 2011 from http://en.wikipedia.org/wiki/data_center 3. TIA-942 Data Center Standards Overview, ADC Telecommunications, Retrieved June 2, 2011 from www.adc.com/attachment/1270711929361/102264ae.pdf 4. Jew J. ANSI/TIA-942 Telecommunications Infrastructure Standard for Data Centers, J&M consultant Inc. 2006, retrieved from http://www.j-and-m.com/datacenterstandardspresentationmay2006j&mconsultants.pdf 5. ANSI/TIA/EIA-942 standard Guide 6. DiMinico C. Telecommunications Infrastructure Standard for Data Centers, ANSI/TIA-942, MC Communications, retrieved Jun 20, 2011 from http://www.ieee802.org/3/hssg/public/nov06/diminico_01_1106.pdf 7. ASHRAE's "Thermal Guidelines for Data Processing Environments, ASHRAE Publications, 2008 8. Detailed expanation of UPS topologies "EVALUATING THE ECONOMIC IMPACT OF UPS TECHNOLOGY" (PDF) 9. A definition from Webopedia "Data Center Tiers". Webopedia. 2010-02-13. Retrieved 2010-02-13. 8