ITCS Data Center Co Location Study November 19, 2007 Introduction Northwestern Information Technology (NUIT) has started investigating the possibility of using a co-location facility to house university computing resources. The number of administrative servers deployed within the central NUIT data centers (392 servers in Evanston) and (85 servers in Chicago) continues to grow with 162 more servers planned for 2008. The requirement to provide data center space, power and cooling to satisfy high performance computing (HPC) clusters; 2374 nodes identified as the research community s known demand, is even more staggering. NUIT Computing Services has implemented technologies to reduce power demands by server and storage virtualization. Neither data center is currently capable of handling today s current known demand. NUIT Computing Services has investigated co-location services as a strategy to temporarily satisfy the power and cooling demands for administrative computing servers and HPC clusters. Co-location is a commercial data center that offers rack space and network connectivity to consumers. The services offered vary from vendor to vendor. Most vendors offer several options for power, network bandwidth, system administration services, remote managed services, and backup services. The co-location approach is very different from buying or leasing space to build-out and support long-term data center needs. Buying and leasing space involves longer implementation times and larger up-front costs with the likelihood of reduced on-going expenses; however, any new space would require staff time to manage. This study did not investigate a buy/lease space alternative because time is pressing. Another study could examine and compare all data center strategies. Co-location can be implemented more quickly for a smaller up-front investment, but it does have heavier on-going expenses. Co-location could be a solution to temporarily alleviate the power demand within the data centers and provide space for high performance computing research clusters and administrative computing servers. Co Location Options & Features NUIT Computing Services contacted and requested quotes from the following colocation vendors which have data centers in the Chicago Metropolitan area: Co-Location Cag 1 Shared Rack Dedicat ed Rack TABLE 1 Power Upgrade Cross Available
Provider e Suite Space Space s Conne Space Area s Allowed ct AT&T Yes Yes Yes Yes Yes Yes March 2008 Level 3 Yes Yes Yes Yes Yes Yes Nothing in Communication Chicago; Virginia is available now s Equinix Yes Yes Yes Yes Yes Yes Available now Co-Location Provider Digital Realty Trust GigEnet Switch and Data 2 TABLE 1 (continued) Cag e Area Suite s Shared Rack Space Dedicat ed Rack Space Power Upgrade s Allowed Cross Conne ct Available Space Yes Yes No No Yes Yes Available now Nothing is available today and did not provide details. Waiting for a response. January 2008 During initial phone conversations with three of the vendors, they mentioned that they are currently not leasing any dedicated space in their Chicago area data centers due to a lack of power within those data centers. One exception was Equinix, which just opened a new facility in Elk Grove Village. AT&T expects to be able to start leasing co-location space again in March 2008, when their largest customer expects to complete the build out of their own data center and will vacate the AT&T co-location facility. GigEnet expects to finish an upgrade to their UPS sometime in December, which will give them an additional 750 kva of power and allow them to begin leasing space to co-location customers again. Digital Realty Trust has 10,300 sf of turn-key data center space available at their Chicago co-location facility on Cermak Road. Digital Realty Trust will subdivide the space into four 2,575 sf suites each having 270kW of UPS load and requires a 5-year lease commitment. ITCS discovered that each co-location facility has different packages available. Most offer shared rack space, dedicated rack space, caged areas, and suites. Shared rack space is by far the cheapest option and is suitable for customers who only have a few servers, which they would like to locate in the co-location facility. The customer simply shares a rack with other co-location customers and rents their internet connectivity from the co-location vendor. The co-location vendor will normally provide 2 Amps of power to the server and rents rack space by the U. This service is very similar to how ITCS rents space to university researchers in the Chicago and Evanston Data Centers, however, the co-location vendors limit network bandwidth and will charge premiums if customers exceed them. Customers with several servers, which need to be located in the co-location facility, can rent a dedicated 42 U rack. Most co-location facilities will include a single 110v 20 amp electrical circuit to the rack. Most of the co-location vendors we spoke with built their facility to only provide 4kW to a rack and no redundancy within the rack itself. A
few of the co-location facilities will allow a customer to purchase 8kW to a rack, but require the customer to pay for additional power and cooling. The larger co-location vendors also offer a caged rack space option and suites of rack space. A caged rack space is an area within the co-location data center, which is fenced off from other customers and provides a more secure environment for your equipment. This option is primarily for those customers who have a need for a large number of racks and want a more secure area for their equipment. The customer must pay for the cage to be built, the racks to put into the caged area, Internet connectivity, and all the equipment which will go into the racks. Once again electrical services are limited and any additional requirement above 4kW will incur an additional cost for not only the electrical work but also for additional cooling charges. Suites are identical to caged areas but solid walls are built around your racks to provide an even more secure area. There are no special package rates, it is pretty much a custom build out and you pay for everything with pricing provided through custom quotes. Remote Hands service is a service where co-location technicians geographically located at the co-location site are able to perform limited troubleshooting activities guided by the customer s technical engineer communicating remotely via telephone, email, or other means with the co-location facility s engineers. In order to get an indication of the costs for co-location services, NUIT Computing Services requested a quote from each of the vendors listed in Table 1, in similar scale to the NUIT co-location area within the Chicago data center facility. Additional Cost Factors Existing university-owned equipment moved to a co-location facility requires onetime investments in networking and computing infrastructure equipment to provide our services at the co-location facility. These are investments in NU-owned equipment; costs not paid to the co-location provider and are necessary no matter which colocation facility is selected. Three-year support and maintenance contracts for the new networking and computing equipment will be bundled into the purchase price. Staff travel time to and from the co-location facility for on-site repairs, upgrades and patches are ongoing costs. One time and ongoing costs NOT paid to co location provider Additional Cost Factors - Northwestern Supplied One-Time Cost Networking routers, swiches, firewall and loadbalancing $450,000 $0 Equipment racks and cable/ fiber management $192,400 $0 Tape backup services w/ 3-year support contract $84,429 $0 Physical move of existing equipment $21,000 $0 Installation $24,000 $0 Application downtime while services are transferred to co-location TBD $0 Man-hours to patch, repair, and travel to/ from $0 $40 Total Estimated Cost 771,829 40 772,309 480 480 773,269 Notes: 1. Networking equipment at co-location facility purchased with 3-year support contract 3
2. Equipment racks and cable/fiber management at co-location facility 3. Tape backup services at co-location facility purchased with 3-year support contract 4. Physical move of existing equipment to co-location facility based on: $500 per hour * 14 hours = $7,000 for 1 mover/van/local travel $21,000 for 3 movers/van/local travel 5. NU man-hours for installation at co-location facility based on: $120 per hour *40 hours = $4,800 for 1 NU staff working 1 full week $24,000 for 5 NU staff each working 1 full weeks 6. Application downtime for services to be relocated at co-location facility based on: Expectation is a 3 day outage per service (72 hours) 7. NU man-hours for travel to/from co-location facility for 2 days per month based on: $20 round trip * 2 = $40 for 1 NU staff member 4
Evaluation of Services to Co Locate When considering services to move to a co-location facility, ITCS agrees with Burton Group that we should avoid splitting applications and services that support a given business process between internal data centers and co-location facilities. Such a split may not seem to be a problem, but disaster recovery of the business process and future upgrades to applications and services will prove to be more difficult to coordinate in a split environment. Network traffic between the application tiers may also drive up network costs by exceeding network bandwidth limits as specified in the co-location Service Level Agreement. Co-location is best suited for NU services that are self contained and do not rely upon applications located elsewhere. When moving servers with large amounts of data requiring daily backup, these will require a tape library at the co-location facility. This allows for better usage of network bandwidth between NUNet and the co-location facility. We should try to use a co-location facility in the Chicago Metropolitan area because monthly network cost are lower and the ease of moving existing equipment into and out of the co-location facility. A cage area does not lend itself easily to physical growth beyond the initial cagedoff space. Northwestern University s growth requirements for the services to be provided at the co-location should be well understood and identified at contract signing. Co-Location providers will not guarantee that appropriate space within acceptable lead times will be available for customer expansion at the co-location facility. NU Services Identified as Best Suited for Co Location Mixture of HPC research clusters and administrative computing servers15 racks 1. Weinberg Technical Services applications 4 racks 2. Departmental servers 4 racks College Sports Project for Administration and Planning The Last Expression for Block Gallery Chicago Historical Society's Electronic Encyclopedia of Chicago History for NUIT Academic Technologies Collaboratory project for NUIT Collaboratory University Relations NUINFO website, search engine and database services University Relations Content Management System University Services redundancy for WildCard services Information Technology Research Data Integration Access (RDIA) web-based services Controller's Office Effort Reporting services Kellogg Graduate School of Management Research and Numerical Analysis services Office of Research document imaging and Institutional Review Board services, animal care management services, grants management archival services, and web services 3. Social Science Computing cluster and Isilon storage system 4 racks 4. NU Library disaster recovery services ½ rack 5. Blackboard disaster recovery services ½ rack 6. Tape Backup System 1 rack 7. Networking equipment 1 rack 5
8. Future Growth/Available Rack Space 0 6
Quote Requested from Co Location Providers NU requirements for a mixture of research clusters and administrative computing Our request for a caged area with a quantity of 15 standard 42 U racks each rack with 8kW of power (a primary 208V 30 Amp power circuit and a redundant 208V 30 Amp power circuit), a connection to a NUIT-TNS preferred internet provider s patch panel, and 10 hours per month of Remote Hands service. Digital Realty Trust Location: 350 E Cermak Road One-Time Cost Space 2,575 sft. priced at 270kW for 33 Racks $29,700 $63,450 Buildout construction of cage and power to racks TBD Metered Power at 270kW a month $17,496 HVAC at cost.5 actual metered power rate of 270kW $8,748 Network - (2) 1 Gbps GigaMANs Connections (Chicago to Chicago) $10,000 Remote Hands subcontracted $2,000 Total Estimated Cost 29,700 101,694 1,250,028 1,220,328 1,220,328 3,690,684 Level 3 Communications Location: Virginia One-Time Cost Space for 15 Racks $13,500 $14,250 Power 8kW - 30a/208V $21,000 $40,500 Network - 2.5 Gbps (Chicago to Virginia) Waived $36,000 Remote Hands $2,000 Total Estimated Cost 34,500 92,750 1,147,500 1,113,000 1,113,000 3,373,500 Equinix Location: Chicago Area One-Time Cost Space for 15 Racks $24,000 $33,600 Power 8kW - 30a/208V $21,000 $35,100 Networking - (2) 1 Gbps GigaMANs Connections (Chicago to Elk Grove) Waived $15,000 Remote Hands 10 hours per month $2,000 Total Estimated Cost 45,000 85,700 1,073,400 1,028,400 1,028,400 3,130,200 AT&T Location: Chicago Area One-Time Cost Space for 15 Racks $99,000 $56,700 Power 8kW - 30a/208V $72,000 $39,704 Networking - (2) 1 Gbps GigaMANs Connections (Chicago to Lisle) Waived $17,000 Remote Hands 10 hours per month $110 $2,240 Total Estimated Cost 171,110 115,644 1,558,832 1,387,722 1,387,722 4,334,276 Plus Additional Costs Additional Cost Factors - Northwestern Supplied One-Time Cost Networking routers, swiches, firewall and loadbalancing $450,000 $0 Equipment racks and cable/ fiber management $192,400 $0 Tape backup services w/ 3-year support contract $84,429 $0 Physical move of existing equipment $21,000 $0 Installation $24,000 $0 Application downtime while services are transferred to co-location TBD $0 Man-hours to patch, repair, and travel to/ from $0 $40 Total Estimated Cost 771,829 40 772,309 480 480 773,269 7
NU requirements for HPC clusters only Our request for a caged area to put 15 12kW racks for HPC clusters similar to the design needs of Room LG-87, a connection to a NUIT-TNS preferred internet provider s patch panel, and 10 hours per month of Remote Hands service. Digital Realty Trust Location: 350 E Cermak Road One-Time Cost Space 2,575 sft. priced at 180kW for 15 Racks $21,000 $42,300 Buildout construction of cage and power to racks TBD Metered Power at 180kW a month $11,664 HVAC at cost.5 actual metered power rate of 180kW $5,832 Network - (2) 1 Gbps GigaMANs Connections (Chicago to Chicago) $10,000 Remote Hands subcontracted $2,000 Total Estimated Cost 21,000 71,796 882,552 861,552 861,552 2,605,656 Plus Additional Costs Additional Cost Factors - Northwestern Supplied One-Time Cost Networking routers, swiches and firewall $350,000 $0 Equipment racks and cable/ fiber management $192,400 $0 Physical move of existing equipment $21,000 $0 Installation $24,000 $0 Application downtime while services are transferred to co-location TBD $0 Man-hours to patch, repair, and travel to/ from $0 $40 Total Estimated Cost 587,400 40 587,880 480 480 588,840 8