Top of Rack (ToR) versus Structured Cabling Harry Forbes

Transcription

1 Top of Rack (ToR) versus Structured Cabling Harry Forbes Chief Technology Officer Nexans Cabling Solutions White Paper Nexans Cabling Solutions April 2012

2 Top of Rack (ToR) versus Structured Cabling Harry Forbes Chief Technology Officer Nexans Cabling Solutions Top of Rack (ToR) concept The ToR concept basically consists of network edge switches which are installed in the top of each server rack. The switch is basically a standalone unit and can be a layer 2 or 3 device which is typically 1u in height with 48 Ethernet network ports. Connection between the switch and server port (figure1) is in most cases facilitated by either a copper twinax or an SFP+ fibre optic assembly and is commonly known as a direct attach. Figure1. These point to point connections are only available as complete factory built assemblies and come in various standard lengths. Due to cost, copper twinax solutions have become the preferred choice over fibre in the majority of data centre deployments. A major drawback with twinax is its reach which tends to be limited to around 10m depending upon cable spec and Ethernet speed. This reach limitation is discussed further on.

3 Structured cabling concept Structured cabling utilises intermediate connections between an end or row (EoR) switch and the server network ports. Connections from the servers to the switch are via a cross-connect. Patch panels are installed in the top of the server racks. An example of this is a 3 connector model which is commonly used in data centres, main equipment rooms and satellite equipment rooms (figure2). Figure2. The key difference is the 100m reach and the ability to facilitate connection between any two devices on the network through the cross-connect which is discussed in greater detail further on. ToR design and deployment In the following example 6 servers with 4 x 1Gbe Ethernet ports and a 48 port 1Gbe switch with a 10GBASE-SR fibre up-link are installed in each server rack. There are 7 server racks in this row (figure3). Connections between switch and servers are copper twinax assemblies. Figure3.

4 In this scenario there are 24 twinax connections in total from the servers to the ToR switch in each server rack. This leaves a further 24 switch ports available for further growth, this is a very inefficient utilisation of network ports especially if future growth is slow to materialise. CAPEX for this model = 7 switches x 6120 unit cost = 42,840 Power consumption is another major cost factor that needs to be taken into account. Research has shown that unused switch ports will still consume as much as 90% of their active power (figure 4). The chart below is for RJ BASE-T ports. Figure4. The total power rating for the ToR switch is 120 watts with 1 watt at each active network port. The inactive ports will still consume 90% of their rated power which equates to 0.9 watts per port. Therefore the total power for each switch in the example above equates to 117.6w. Assuming a unit price of 0.10p per kilowatt hour for energy costs and an equipment refresh rate of 3 years then: OPEX = 7 x 117 = 0.819kWhr x 0.10 x 24 hours x 365 days x 3 years = 2,152 Total CAPEX & OPEX for this model = 44,992

5 One way to reduce costs would be to reduce the number of switches (figure5). Figure5. In this scenario a minimum of 4 switches are required to service all the servers and results in a higher utilisation (87.5%) of switch ports providing a lower cost of ownership and improved CAPEX and OPEX. However there is limited capacity for future growth and in order to provide a minimum redundancy level of n+1 an additional switch needs to be installed (figure6). Figure6. This scenario presents 30% spare switch port capacity for future growth and is an optimised design for cost, resiliency, redundancy and reach of direct attach twinax. Total 3 year CAPEX & OPEX for this model = 30, ,537 = 32,137

6 Structured cabling design and deployment In order to provide a like for like comparison each server rack is populated with 6 servers with 4 x 1Gbe Ethernet ports each. The end of row switch is a chassis with slots for 9 x 1u line cards. 7 of which can accommodate line cards with 48 x 1000BASE-T network ports with the remaining two slots for a network management card and a line card with 8 x 10GBASE-SR fibre up-links. Like the previous example, 4 line cards are installed and servers are dual homed to separate line cards for resiliency. An additional line card is installed for n+1 redundancy providing 5 in total (figure7). Figure7. CAPEX = 27,515 (11% cheaper than ToR). Note total cost includes chassis and fan. OPEX for 3 years = 1,572 (2% more expensive than ToR) Total CAPEX & OPEX = 27, ,572 = 29,087 (10% cheaper than ToR) In the examples above both ToR and Structured Cabling are equally matched for future expansion. Each have the same number of spare switch ports for new server additions. Two ToR switches can be installed in the unpopulated racks and two line cards in the spare EoR chassis slots. In reality however the following scenario is more likely to exist where there is an uneven server count per rack and growth is unpredictable including upgrades to higher speeds. In this model (figure8) the number of racks has now been extended from 7 to 12. A 48 port ToR switch is installed in each rack. The switches colour coded in red have all 48 network ports in use. The switch colour coded in blue in rack 11 has a number of unused ports. The servers colour coded in green are new additions which require connection to a switch and can only be installed in these racks due to available power. Each rack is 800mm wide. Note redundancy is not included but is discussed in detail further on as a separate subject.

7 Figure8. In this instance the twinax connections from the switch in rack 11 can only reach the servers in rack 8 and 4 due to the limited 10m reach. This requires an additional switch to be installed. The 100m reach offered by structured cabling easily facilitates connection to all three servers from any End-of-Row switch saving on additional CAPEX & OPEX. This design flexibility increases the utilisation of unused switch ports. ToR CAPEX = 13 x 6,120 = 79,560 ToR OPEX = 13 x 120 = 1.56kWhr x 0.10 x 24 x 365 x 3 years = 5000 Total = 84,560 SC CAPEX = 2 chassis + fans = 6, line cards = 37,992 2 uplink cards = 11,676 2 x management module = 10,974 Total CAPEX = 67,592 (22% cheaper than ToR) SC OPEX = 2 chassis x 0.78kWhr x 0.10 x 24 x 365 x 3 years = 5000 (same as ToR) Total = 72,592 (15% cheaper than ToR)

8 Network speed upgrades Due to varying business demands for various services which can often be unpredictable and the associated costs, network speed upgrades tend to be done as and when required. This can have an impact on cost depending upon the choice of cable media. In the following example (figure9) as before the ToR switches are all 1Gbe. The servers colour coded grey have been upgraded with 10Gbe ports. Because of the reach limitation of twinax 2 x new 10Gbe switches must be deployed. Figure9. In the case of structured cabling only 1 x 10GBASE-T line card needs to be installed in an End-of- Row switch to provide connections to all of these servers because of the 100m reach. Again this reduces both CAPEX & OPEX. Backwards compatibility A major benefit of 10GBASE-T is its multispeed capability (100Mbps, 1Gbps & 10Gbps). This feature enables connection between any two devices that work at different speeds and is achieved through the Ethernet auto-negotiation feature. For example if a 10GBASE-T switch port is connected to a 1000BASE-T server port, auto-negotiation will drop the speed of the switch port to 1000BASE-T. This means that when new servers with 1000BASE-T ports are deployed the unused ports on the 10GBASE-T switch can be used if there are no 1000BASE-T ports available. This feature is not available with Ethernet Twinax connections at any speed. Therefore new 1000BASE-T switches would be required to support roll-out of new 1000BASE-T servers.

9 Other hidden costs In the example shown in figure8 above the power consumption figures for ToR and EoR are both 1.56kW. What has not been taken into account with the CAPEX and OPEX is the provision of additional power supplies and UPS systems to each ToR. Industry reports have shown that losses through UPS systems are typically 16% which needs to be taken into account in this model. Since there are more switch devices in the ToR model, this penalty will have a higher impact on both CAPEX and OPEX than for EoR. In addition to this, research has shown that for every watt delivered to the network edge device in the data centre, 2.8watts need to be generated at the building entry. This is due to the losses through transformers, switchgear, cooling, cabling etc. Therefore additional cost penalties need to be applied in both cases with ToR having the greater impact. Another factor that deserves attention in a business environment that is under pressure to make their operations greener is the disposal and recycling of hazardous materials. Again this comes with financial penalties and favours an End of Row approach. Design flexibility Power and cooling are two of the major concerns when it comes to the design, build and expansion of data centre services. Optimised performance requires an even distribution of power and cooling. This requires a cabling infrastructure design that enables the placement of servers anywhere on the network to maintain the balance. This is easily achieved through structured cabling because of the 100m reach. However this is severely restricted with ToR switches due to the limited reach of twinax. Patching at the cross-connect enables any available port on any switch to connect to any server regardless of where it is situated on the network (figure9). This illustrates a connection between the EoR switch in row 4 and the server in row 1 rack 1. Figure9.

10 It has been widely reported that following the practice of routing cables from ToR switches to separate rows has led to a rat s nest of cabling underneath the raised floor which has caused air dams leading to inefficient cooling and potential hotspots occurring in server racks and possible failure of equipment. Redundancy In order to eliminate air dams, improve cable traceability and management it is highly recommended to restrict the routing of twinax cables to the row of server racks in which they are installed for Tor applications. Assuming a redundancy level of (n+1) then an additional switch should be installed in the centre of every 17 server racks to ensure connection to the furthest away servers. The following table (figure10) depicts redundancy levels for both ToR and EoR as the number of server racks increase per row. No. Racks Total No. redundant ToR switches Total No. redundant EoR switches Figure10. It becomes clear that redundancy has a major impact on costs for ToR switching which also includes the provision of additional power and UPS systems. Additional OPEX Like for like the ToR model will have seven switches for every EoR switch in order to provide an equal number of switch ports (336) to the network edge devices. This equates to additional costs for ToR in terms of overheads for installation, commissioning, administration, upgrades and maintenance.

11 Summary It is clearly evident that End-of-Row switching with structured cabling has many advantages over Top-of-Rack Solutions. The capital outlay and ongoing operating expenditure is much reduced through better utilisation of resources including system upgrades and expansion as new services are rolled out. The flexibility to deploy devices where and when required offers a significant advantage to optimise power and cooling as well as keep CAPEX and OPEX under control. Harry Forbes Chief Technology Officer Nexans Cabling Solutions Harry is the Chief Technology Officer at Nexans Cabling Solutions and was educated in electrical and electronic engineering. He has worked in the cabling and networking industries for the past 30 years in various technical roles, and has extensive knowledge of and expertise in enterprise systems and data centre infrastructure requirements.

12 Copyright 2012 Nexans Cabling Solutions. All rights reserved. LANmark, LANsense and GG45 are registered trademarks of Nexans. Release date: April Nexans Cabling Solutions Alsembergsesteenweg 2, b3 - B-1501 Buizingen Tel: +32 (0) Fax: +32 (0) Nexans Cabling Solutions UK and Intelligent Enterprise Solutions Competence Centre 2 Faraday Office Park - Faraday Road - Basingstoke - Hampshire RG24 8QQ Tel: +44 (0) Fax: +44 (0) info.ncs@nexans.com