LEVEL 3 DATA CENTER ASSESSMENT

Nationwide Services Corporate Headquarters 410 Forest Street Marlborough, MA 01752 USA Tel: 800-342-5332 Fax: 508-303-0579 www.eecnet.com LEVEL 3 DATA CENTER ASSESSMENT Submitted by: Electronic Environments Corporation 410 Forest Street Marlborough, MA 01752

TABLE OF CONTENTS PAGE SECTION 1. EXECUTIVE SUMMARY 1 SECTION 2. EXISTING CONDITIONS/EVALUATION 2 SECTION 3. RECOMMENDATIONS 11 ATTACHMENTS: Drawing E1 ONELINE SCHEMATIC Drawing E2 FACILITY FLOOR PLAN Drawing E3 PART PLANS

SECTION 1 - EXECUTIVE SUMMARY Electronic Environments Corporation (EEC) has been contracted to evaluate the existing data center located in Plymouth, MA. The goals of the study are: A. Survey and document the existing conditions in the Data Center and Telecom Rooms: 1. General conditions (dimensions, finishes, envelope integrity) 2. Equipment layout 3. Temperature, humidity and air-flow 4. Electrical and cooling loads 5. Areas of high temperature B. Describe existing conditions and identify areas of potential concern. C. Provide recommendations to improve reliability and efficiency. EEC conducted site surveys on 10-21-2010 and 11-29-2010 to gather existing conditions. The findings our outlined in the following report. In general, the data center, telecom spaces and associated infrastructure are well maintained. System loads are relatively low, allowing for future growth. Recommendations in Section 3 of this report include the following: A. Install redundant electrical components in the data center electrical distribution. B. Upgrade and install redundant cooling units as the load grows. C. Complete a full circuit audit of the electrical systems, provide updated labeling and panel circuit directories. D. Replace the existing in-rack power distribution with new metered plug strips. E. Install a site monitoring system. F. Install a water detection system in the telecom rooms. G. Consider removal of the walls separating the telecom spaces to improve air flow and install redundant cooling systems. H. Install blanking plates in server cabinets and utilize cable management systems to neaten cabling at the rear of the server cabinets. I. Verify that a pressure relief venting system exists in both the data center and telecom rooms for proper operation of the Inergen systems. J. Consider the installation of an early warning aspirating smoke detection system (VESDA). 1

SECTION 2 - EXISTING CONDITIONS/EVALUATION A. General Conditions (Please refer to drawing E3 for floor plans of the data center and telecom rooms) In general, the data center and telecom rooms are well maintained, secure and environmentally controlled environments. A card access security system limits access to each room. All three spaces utilize a raised access flooring system for conditioned air supply and cabling. Finishes in these rooms appear to be providing a sealed environment for temperature and humidity control. Exterior windows have been closed off to limit solar heat gain and provide a level of security. The data center has a water detection system installed below the raised access floor. Room Data Center Demarc Room PC Lab/IDF Room Area 1,600 Sq. Ft. 125 Sq. Ft. 300 Sq. Ft. Approximately 50% of the data center space is utilized for server equipment and the remaining 50% is used for printing operations and data center operator s work space. There are some areas within the data center that are being used for storage of server equipment and paper products. This is often a necessity in data center operations, but should be kept to a minimum to reduce cardboard and paper dust in the environment. The Demarc and PC Lab/IDF Rooms contain equipment storage as well, which should be kept to a minimum to improve workspace access and air-flow. B. Mechanical Systems and Environmental Conditions The data center and telecom rooms are conditioned by Liebert computer grade cooling units as described in the table below. Room Served Unit Year of Make & Model Name Manufacture Capacity Data Center CRAC-1 Liebert DH125A-UAAEI 2001 10-Ton Data Center CRAC-2 Liebert DH125A-UAAEI 2000 10-Ton Demarc Room CRAC-3 Liebert BU67A-AAEI 2001 5-Ton PC Lab/IDF Room CRAC-3 (see above) (see above) (see above) Data Center Systems and Environmental Conditions The two 10-ton CRAC units supply conditioned air to the data center through the raised floor plenum via perforated floor panels. The total heat load within the data center is approximately 32kW. This includes the server cabinet loads as well as heat rejection from the UPS system and power distribution unit (PDU), and heat gain from adjacent spaces. This load meets the cooling capacity of one CRAC unit. Run hours recorded on each CRAC unit on 10-21-2010 indicate that CRAC-1 runs at full capacity, while CRAC-2 runs at a reduced capacity to maintain temperature within the room. 2

Cooling Unit Fan Hours Compressor-1 Hours Compressor-2 Hours CRAC-1 1027 1022 1023 CRAC-2 1027 347 344 EEC has created a CFD model of the environmental conditions within the data center utilizing data collected during the site survey. The CRAC units are adequately sized to provide N+1 redundancy at the current server cabinet loads. This is the limit of the systems redundancy however. Additional heat load introduced to the room will require both CRAC units to run to maintain the required temperature. A thermal map of the room shown below indicates temperatures as seen at the back of the server cabinets under normal conditions with both CRAC units running (temperature values are shown in degrees Fahrenheit). These temperatures are typical for hot-aisles behind server racks. Temperatures at a plane through the rear of the server cabinets Temperatures at the front of the racks (server inlets) are well within acceptable standards as published by ASHRAE TC 9.9. In the image shown below, temperatures at the front of the racks are actually below the lower end of the ASHRAE recommended temperature range for server inlets This data must be confirmed during a final survey to validate the model. 3

Temperatures at the front of the server cabinets both CRAC units running The CFD modeling program also allows us to create predictive models during simulated CRAC unit failures. EEC ran two failure scenarios: one with CRAC-1 running and CRAC-2 off, and the second with CRAC-2 running and CRAC-1 off. The simulations indicate that the server racks remain adequately cooled with either CRAC unit turned off. Temperatures at the front of the server cabinets CRAC-1 turned off 4

In the image above, the temperatures at the front of the server racks are highest with CRAC-1 turned off. These temperatures are still within the ASHRAE guidelines however. An increase above current loads will eliminate the cooling systems redundancy. The following image was generated from a model which assumes that the load in each server cabinet increases from an average of 3kW to 5kW. The load in the IBM cabinets would remain unchanged in this model. The model also assumes that only one CRAC unit is running at a time. In the image below, CRAC-1 is off and CRAC-2 is running. Temperatures at the server inlets range from 80 F to an extreme of 110 F, well above recommended guidelines. Temperatures at the front of the server cabinets Additional 12kW load, CRAC-1 turned off Telecom Rooms Systems and Environmental Conditions A single 5-ton CRAC unit serves the Demarc Room and PC Lab/IDF Room. This unit is located in the IDF Room. Supply air is delivered through perforated tiles in the raised floor and air is returned from the Demarc Room through return air grilles in the ceiling. Unit run hours shown in the following table indicate that the unit is running at roughly 51% of cooling capacity. Cooling Unit Fan Hours Compressor Hours CRAC-3 1029 527 The heat load in these spaces is relatively low as compared to the data center. There is capacity in CRAC-3 for additional heat loads; but since there is only one unit there is no redundancy. Air flow is sufficient at current loads; however, if load were increased in the Demarc Room, the separation between the spaces would hinder air flow and temperature variations would arise. An increase in load in the PC Lab would also create temperature variations due to air flow constraints. 5

C. Fire Suppression Systems All of the spaces under review are protected by an automatic sprinkler system and an Inergen fire suppression system. One Inergen system is dedicated to the data center and the other supports the telecom rooms. Both Inergen systems are actuated by smoke detection devices located on the suspended ceiling and below the raised access floor. The Inergen suppression system is well suited for data center applications as a clean agent system with no ozone depleting characteristics. Due to the high concentration Inergen required to extinguish a fire (roughly 40% of room volume) these systems require pressure relief venting to remove the air in the room while the gas releases from the Inergen bottles. EEC was not able to identify pressure relief venting during the site audit. This needs to be investigated further to confirm the existence of pressure relief venting. D. Electrical Systems (Please refer to drawing E1 for a graphical representation of the electrical system) The building at 8 Richards Road is supplied by an 800A, 480/277V, 3-phase, 4 wire electrical service. The entire service is supported by a 500kW, diesel fueled, standby generator, via an 800A automatic transfer switch (ATS). The ATS supplies power to the building main switchboard, which distributes power to various loads throughout the building. A 400A circuit breaker in the switchboard feeds the data center main panelboard OS1. OS1 supplies power to the UPS system, data center CRAC units and other loads within the data center space. Data Center Electrical Systems A 75kVA UPS system supplies power to a 75kVA power distribution unit (PDU), which in turn supports all of the IT loads in the data center as well as some loads in the telecom rooms. Server cabinets, printers and other equipment are supplied directly off of the PDU through branch circuits in two of three available panelboards. A third panelboard in the PDU, as well as a 225A sub-feed breaker are available for future loads. A 100A breaker in panel #2 of the PDU supplies power to another panelboard in the IDF Room. A summary of the major electrical equipment serving the data center and telecom rooms is included in the table below. Equipment Generator ATS Data Center Service UPS System PDU Location Exterior Elect Rm 119 Elect Rm 118 Data Center Data Center Make & Model Caterpillar 3412 Russelectric RMTD-800 CE ITE Circuit Breaker Year of Manufacture Capacity % Load 2000 500kW Unknown 2000 800A @ 480V Unknown Unknown 400A @ 480V 18% Liebert AP366 2000 75kVA/60kW 45% Liebert PPA075C 2000 75kVA 45% 6

Loads represented as a percent of available capacity in the table are based on readings taken during the site survey. There is plenty of capacity in the data center electrical system for future loads. The UPS system incorporates a maintenance bypass cabinet which allows for isolation of the UPS system during maintenance, without disruption of power to the servers. The UPS system has one string of batteries which was replaced in May of 2010. UPS supported circuits originating in the PDU are installed below the raised access floor, with outlets located behind the server cabinets which they serve. The outlets supply power to plug strips which are located below the raised access floor as well. Power cords from each of the servers are routed down through cable openings in the raised floor tiles to the plug strips below the floor. This method of power distribution creates a large volume of cabling below the floor and tends to complicate circuit identification. EEC attempted to trace all of the circuits supplying the server cabinets during the site audit, such that the source of power for each server could be identified in the PDU panels. Only a portion of the circuits could be identified due to cable congestion below the floor and the quantity of server power cords. All electrical panels and circuit breakers supporting the data center were scanned with an infra-red (IR) camera to identify potential overload conditions or loose connections. None of the scans revealed cause for concern and all temperatures were within acceptable ranges. A list of the equipment scanned is included below. Equipment Location ATS Electric Room 119 400A Breaker Electric Room 118 Panel OS1 Data Center Panel OS1-A Data Center Panel OS2 Data Center UPS System Data Center PDU Data Center UPS Panel IDF Room 200A Breaker Electric Room 118 Panel CPA/B PC Lab IR scan images of the PDU panelboards are shown on the next page for reference. 7

PDU Panelboards 1 & 2 PDU Panelboard #1 IR Scan PDU Panelboard #2 IR Scan While tracing the circuits in PDU panelboard #2, a loose connection was found on circuit #19. This was not identified during the initial visual inspection or through the IR scanning. Arcing was noticed when electrical load readings were being taken with a digital meter. Further investigation revealed that the conductor had never been terminated under the 8

circuit breaker screw terminal during the original installation. The conductor was in contact with the terminal, held in place only by tension on the conductor. The arcing caused momentary power disruption to some servers, which failed over to their redundant power supplies. One server which did not have its redundant power supply connected to a separate circuit, experienced an outage which was logged by Rockland Trust s IT staff. Subsequent investigation revealed that circuit numbers 19 and 21 were mislabeled on the receptacles below the floor. These markings were corrected and an alternate circuit was identified for the redundant power supply on the affected server. With the redundant circuits verified, Rockland Trust authorized EEC to shut of circuit #19 and correct the loose connection. This work was performed and the circuit was reenergized without further incident. PDU Panelboard #2, Circuit #19, Loose Connection Telecom Rooms Electrical Systems Power for IT equipment in the telecom rooms originates in two panels: UPS Panel and CPA/B. UPS Panel is supplied by the 75kVA PDU in the data center and is supported by the UPS system. Panel CPA/B is powered from a 112.5kVA transformer with harmonic filtering. The loads on both panels are fairly low. The power sources for the telecom rooms are described in the table below. Panel Location Source Capacity % Load UPS Panel IDF Room 75kVA UPS & PDU 100A @ 208/120V 13% CPA/B PC Lab 112.5kVA Transformer/ Harmonic Filter 200A @ 208/120V 16% Power for panel CPA/B is conditioned and supported by the building generator, but does not have a battery back-up to protect against momentary outages. IR scans of the panels 9

and circuit breakers serving the telecom rooms did not identify any issues. UPS Panel is missing a cover plate over the main circuit breaker. A cardboard cover has been inserted to shield personnel from live parts when the panel door is open. UPS Panel in IDF Room with Cardboard Cover UPS Panel IR Scan #1 UPS Panel IR Scan #2 10

A. General Conditions SECTION 3 - RECOMMENDATIONS 1. Rockland Trust should consider the installation of a site monitoring system to monitor all critical equipment and systems. Many monitoring systems are available on the market which can report conditions and alarms through email and voice messaging. Most systems use a web page interface which can be accessed from remote locations. Systems which should be monitored include: a. Generator and UPS systems b. Cooling systems and remote temperature sensors c. Water detection systems d. Fire alarm systems e. Electrical circuits, including rack power strips 2. The telecom rooms should be monitored by a water detection system. This system would be connected to the site monitoring system as well. B. Mechanical Systems and Environmental Conditions Data Center Systems 1. As the IT load within the data center increases, the CRAC units will no longer be redundant. This will leave the data center exposed to over-temperature conditions during routine maintenance or CRAC unit failures. If Rockland Trust expects the IT load to increase, then the existing units should be replaced with larger capacity units. Newer units will operate more efficiently through the use of communications and load sharing. 11

2. Blanking panels should be installed in the front of the existing server cabinets to prevent mixing of cool supply air and heated return air. This will ensure that the heated air is not circulated back to the server inlets and will allow the CRAC units to operate more efficiently. 3. Power and communications cable management systems should be utilized at the rear of the server racks to neaten the cabling. This will improve air-flow and heat removal from the rear of the cabinets. Some of this work may require cables to be unplugged and so it would have to be performed during a maintenance window. IR Scan of the rear of Cabinet E showing cable congestion Telecom Rooms Systems 1. The current cooling system configuration does not provide redundancy. A second CRAC unit would have to be installed to achieve N+1 redundancy. 2. Ultimately, a single open space would allow for more efficient and effective cooling. The existing walls separating the Demarc Room, PC Lab/IDF Room and IDF Room create air flow barriers and complicate an N+1 cooling solution. If possible, Rockland Trust should review the security requirements for these spaces to determine whether the walls can be removed. An alternative to hard walls would be wire mesh cage material which would provide physical security while allowing air to flow throughout the spaces. C. Fire Suppression Systems 1. If a pressure relief venting system is not currently installed in the data center and telecom rooms, this should be corrected immediately. Inergen systems require a concentration of roughly 40% of room volume to extinguish a fire. Without a pressure relieve system, the pressure increase during system discharge in the room can cause structural damage to walls, windows and doors. 2. Rockland Trust should consider the installation of an early warning aspirating smoke detection system such as VESDA. These systems provide detection of combustion 12

particles at levels lower than conventional smoke detectors. Alarms can be generated at multiple programmable set points. The VESDA system can be used as a stand-alone early warning system or can be utilized as the primary smoke detection system. D. Electrical Systems and Environmental Conditions 1. The data center electrical system does not have redundant components. Redundant power supplies have been purchased with many of the servers and these supplies are powered from alternate electrical circuits in the PDU. However, there are several single points of failure in the electrical distribution which expose the servers to a potential outage. Components for which redundancy should be considered include: a. Input breaker, feeder and panel OS1 b. 75kVA UPS system c. UPS system battery string d. 75kVA PDU Ultimately, the data center would receive two separate power feeders from the main switchboard. Redundant UPS systems and CRAC units would be supplied from alternate feeders. Servers would receive power from alternate PDUs, such that their redundant power supplies each had a separate path of power back through the UPS systems to the main switchboard. This scenario would require a considerable capital investment and would consume additional space within the facility. At a minimum, a redundant battery string should be installed on the existing UPS system. This would ensure that a single battery failure does not cause a UPS failure during a power outage. The UPS system is over 10 years old and should be considered for replacement within the next few years. At that time, options for UPS component redundancy could be explored further. 2. A full circuit audit should be completed for the circuits supplied by the PDU under the raised floor. This should be scheduled during a maintenance window to minimize the potential impact of an accidental power outage. An accurate record of the existing circuits and an inventory of how they are connected to each server cabinet is essential for ongoing management and troubleshooting. Circuits, receptacle boxes and panel directories should be clearly labeled for ease of identification. The same process should be completed for the panels in the telecom rooms. New labeling should be applied to panels OS1, OS1-A, OS2 and the 400A circuit breaker supplying the data center. The cover plate for the main breaker of UPS Panel in the IDF room should also be replaced. 3. EEC recommends that the in-rack power distribution be modified and upgraded to include metered plug strips, mounted at the back of the racks. New plug strips can be 13

purchase with local metering as well as Ethernet connection for remote monitoring and control of individual outlets. The existing plug strips under the raised floor should be removed and all server power cords should be neatly trained and terminated in the new metered plug strips at the back of the racks. A minimum of two plug strips should be installed on each rack to provide two sources for the servers redundant power supplies. The capacity and quantity of the plug strips will depend on the anticipated load per rack. Prioritized Recommendations Recommendations Priority Estimated Cost Redundant Battery String High $14,725 Redundant Cooling for Telcom Rooms High $29,650 Inergen Pressure Relief Venting High $10,000 Budget Replace 10-Ton CRAC with 15-Ton CRAC (DX Unit) Medium $65,000 Budget Per Unit Replace 10-Ton CRAC with 15-Ton CRAC (Glycol Unit) Medium $75,000 Budget Per Unit Site Monitoring System Medium $15,000 - $45,000 Budget Water Detection System for Telecom Rooms Medium $7,500 Budget Electrical Circuit Audit and Overhead Distribution Medium TBD Labeling, Blanking Panels & Wire Management Low TBD 14

15

16