The 8-Fiber Solution for Today s Data Center Bill Charuk, SR Product Manager, Data Center Solutions Introduction Data center infrastructure is constantly evolving. Fiber optic installations have used the same basic design for decades. However, to fulfill today s needs and be ready for tomorrow s higher speed standards, a new approach should be considered. The innovative network architecture using 8 fiber ribbons as the basic infrastructure building block instead of a 12 fiber subunit is examined to improve significantly data center design, performance, futureproofing, and cost savings. Background Until now, data center infrastructures have been based on the tried and true designs that have served the enterprise network so well for the last 30 years. This is not surprising, since the equipment used and data rates involved in data center construction were essentially extensions of existing enterprise network requirements. With the advent of more and more internet connections and an explosion in social media and mobile devices driving ever larger amounts of data around the internet, the speeds necessary to move the data have progressed as well. When data speeds were increased to 56Kbs for fax machines, most thought they would never need transmission beyond this amazing speed. In 1985 IEEE 802.3 published the 10BASE2 standard for 10Mbs over thin coax. In 1993 the 802.3j committee published 10BASE-F standards for 10Mbs transmission over optical fiber. 1Gbs speeds were codified in 1998 with the adoption of the 1000BASE-X Gbit/s standard by the IEEE802.3z committee. Link aggregation was standardized in 2000 and 10Gbe in 2002. Today data centers are firmly positioned in the 40Gbe/100Gbe world with 400Gbe on the horizon. With this rapid increase in data rates, industry standards have adapted to ensure not only that the network could function, but also maintain the ability to utilize products and technologies that are cost effective and known to the end user market. In 2015Sumitomo Electric Lightwave. All rights reserved.
order to facilitate graceful growth in the data center, another infrastructure option needs to be considered, one that can be retrofitted to existing networks, while simultaneously supporting future growth needs and technology requirements. What and Who is Driving Data Demand? Connections to the internet are ubiquitous today, almost akin to carrier pigeons in New York City at the turn of the century. In 2015 alone, 15 billion devices will be connected to the internet and 3 billion users will access it. Faster broadband speeds, 28Mbps in 2015 versus just 7Mbps in 2010, will be considered standard. In addition, demands for higher quality media and faster downloads, from a 1 minute video download in 2010 to a 2 hour HDTV movie in 2015, are adding to the need for higher velocity data downloads. Cumulatively, each person that is or wants to be connected to the internet is driving data demand, and consequently higher and higher data rates. Of course, a great deal of data today is stored. If the data is stored, then it will need to be backed up. The exponential growth of the amount of stored data will necessarily drive the need for higher and higher speeds to perform these operations in a given amount of time. 1 nowell_01_0911.pdf citing Cisco Visual Networking Index (VNI) Global IP Traffic Forecast, 2010 2015, High Speed Network Migration As evidence that higher speeds are proliferating in the marketplace, Chart 1 shows the recent trend in server shipments supporting 1/10/40Gbe over the last several years. By the end of 2016, 1Gbe server port shipments will become insignificant in the marketplace. Chart 1 78 T.W. Alexander Drive, Research Triangle Park, NC 27709 800.358.7378 www.sumitomoelectric.com 2
This migration will continue until the amount of data required to be transmitted over the internet stops growing. As already shown, the use of the internet continues to grow for a myriad of reasons and likely will not abate at any time in the near (and not so near) future. Chart 2 illustrates the number of port shipments for 40Gbe and 100Gbe ports over the same time period. Chart 2 2 2 OFC-Panel-2-Need-For-Speed-Final Ethernet Alliance - http://www.ethernetalliance.org/event/ofc-2013/ As these charts clearly show, the number of high speed (>10Gbs) ports will grow exponentially in the coming years. It should also be noted at this point that while 40GbE and 100GbE deployments (which will be viewed as equal technologies from an infrastructure point of reference) are growing, the next step in data transmission standards is underway; to 400Gbs. Why is this important? To answer this question, we must first examine basic networking requirements of 40Gbs networks and how, in some instances, they are fundamentally different than those that operate at lower speeds. High Speed Network Fundamentals While this may not seem as a change any different than shifts to higher speeds that have occurred previously, the change for 10Gbe network to a 40Gbe network is 78 T.W. Alexander Drive, Research Triangle Park, NC 27709 800.358.7378 www.sumitomoelectric.com 3
markedly different, unlike any that has occurred in the past. Specifically, until the advent of 40Gbs speeds, all optical transmission in premises/data center environments used serial transmission schemes. That is, the data stream consisted of information being sent one byte at a time, one after the other, at a given rate. Of course, the ability to complete this transmission and have it be able to be discerned at the receiver is a function of several system variables, most notably the speed at which the receiver is able to transmit discreet light pulses that are readily detectable at the receiver. As transmission speeds increase, serial schemes are problematic for Vertical Cavity Surface Emitting Lasers (VCSELs). The ability of the VCSEL to transmit at higher speeds, coupled with the distortion of signals by polarization mode dispersion and chromatic dispersion in the optical pathway, negates use of current VCSEL s at higher speeds in serial transmissions. However, to use higher speed lasers (which are readily available) would necessitate an unacceptable increase in the cost of transmission equipment and essentially make obsolete all installed multimode optical fiber in data center and enterprise networks. The way to utilize inexpensive VCSEL technology and meet the speed requirement is to use parallel optics. Or more simply put, aggregate several channels at lower speeds to reach higher speeds required. This is precisely what the 40G/100G standards, as adopted by IEEE 802.3ba-2010 committee, advises for networking technologies in the transmission of Ethernet frames. The standard, taken in its simplest form, identifies fiber optic link requirements for transmission at 40/100Gbs over multimode fiber. See Table 1. Table 1 78 T.W. Alexander Drive, Research Triangle Park, NC 27709 800.358.7378 www.sumitomoelectric.com 4
However, there are obvious differences in how these speeds are achieved versus previous standards. For 10Gbs transmissions and below, serial transmission was used. In figure 1, we see that the channel consists of 1 transmission link (TX) and one receiver link (RX) both operating at 10Gbs. This is typically accomplished using a duplex LC connector as shown. Figure1-Serial Transmission When using parallel path transmissions, the single lanes are multiplied to reach an aggregate speed. As seen in Figure 2, TX and RX channels are now composed of 4 lanes each. These lanes can be comprised of individual 10Gbs or 25Gbs lanes to reach 40Gbs and 100Gbs respectively. Connectivity in this case is achieved by using MPO (Multiple- Fiber Push-On/Pull-off) connectors. An MPO connector is pictured in Figure 3. These connectors are designed to terminate and align 12 fibers simultaneously. Figure 2-Parallel Lane Configuration Figure3-MPO Connector 78 T.W. Alexander Drive, Research Triangle Park, NC 27709 800.358.7378 www.sumitomoelectric.com 5
The 12 fiber MPO connectors have been in use in the network for some time and are well known. However, there is an inherent problem with their use when installed in a parallel path system. As noted from Figure 4, there are 4 TX, 4RX and 4 unused or dark fibers per connection. While not a technical problem in and of itself, this scheme presents a problem to the network operator. Figure 4-MPO Channel Assignments for Parallel Transmission The problem arises when a 40G/100G solution is required to be implemented using a 12 fiber infrastructure. As seen in Figure 4, there are 4 unused fibers in the center of the connector related directly to every 12 fiber subunit in the cable/cables being used. This equates to a 33% loss in the fiber network utilization. The loss of 33% of the physical network can be a costly concern. While there are ways to overcome this issue, which will be explored later, they do not always result in the most cost effective or best optical pathway for the data center network. Essentially, with the new standards, parallel transmission schemes provided for by the IEEE standards clearly identify the need for an eight fiber based solution. This is a paradigm shift from the way networks have been designed and installed for the past 30 years in enterprise networks and in most data centers. Physical Layer Solutions Getting to 8 Fibers The first eight fiber solution must encompass existing networks. The question at hand is how we achieve use of all the fibers in an existing 12 fiber based network. This methodology is well known and readily available. The basic premise of this work around is that the existing 12 fiber groups, usually with a minimum fiber count of 24, are rearranged into a subset of 8 fiber groupings. Therefore 24 fibers in two 12 fiber 78 T.W. Alexander Drive, Research Triangle Park, NC 27709 800.358.7378 www.sumitomoelectric.com 6
groupings are rearranged into three 8 fiber groups, thereby using all the fibers in the network. This can be accomplished practically in one of two ways. The first is to use a specifically designed cable harness that has two 12 fiber connectors on one end and three 12 fiber MPO connectors on the other end with only 8 fibers populated in the necessary locations to connect the parallel channels. This method has a disadvantage in that the harnesses can be unwieldy and difficult to dress into the fiber termination hardware due to the transition from two subunits to three. A more popular method is shown in Figure 5. In this scenario, the conversion occurs in a cassette/conversion module, which is mounted in a standard fiber optic patch panel. Note the two 12 fiber MPO connections at the rear and three 8 fiber MPO connections at the front of the module. This can also be accomplished with a single 24 fiber MPO at the rear of the module. Figure 5-8 Fiber Conversion Cassette Figure 6 below (next page) shows how this is accomplished internal to the cassette/conversion module. 78 T.W. Alexander Drive, Research Triangle Park, NC 27709 800.358.7378 www.sumitomoelectric.com 7
Figure 6 It must also be remembered that when installing this conversion on a 12 fiber infrastructure, this conversion must occur on both the TX and RX sides of the channel, so the additional costs for the two conversions must be added to the planning of a 40/100G network for every link. The question may arise, if technology exists that can convert the ubiquitous 12 fiber infrastructure to parallel transmission standards, why do I need to consider an 8 fiber infrastructure? There are two areas that need to be examined further that will explain why the 8-Fiber Solution makes more sense. These will be highlighted as we discuss the 8- Fiber Solution further. 8 Fiber Infrastructures for Data Center Applications The eight fiber solution for data centers is essentially a tailored network infrastructure designed to support 40/100G (and 400G) networks in the most efficient and cost effective manner. The major difference is obviously that the subunits of the cable infrastructure are based on 8 fiber count instead of 12. 78 T.W. Alexander Drive, Research Triangle Park, NC 27709 800.358.7378 www.sumitomoelectric.com 8
By designing a network using the 8 fiber subunits as the building blocks, direct connections without conversions are now achieved. The only item in which care must be taken is the connection at the end equipment the 8 fibers terminated in the connecting MPO connector must be aligned as shown in Figure 4, with the operating fiber separated by four blank fiber spaces (Note: this space is necessary so the fibers can attach to the TOSA and ROSA in the end equipment transceiver). This segregation can be easily made using a preterminated patch cord or a field installable Splice-On MPO connector of the correct type (i.e. one specifically designed for this application). There are several advantages to this type of installation. The first is cost. By installing a 12 fiber system for 40/100G systems, the operator will incur costs for the conversion at either end of the system. Depending on the method used, these costs can be over $100 per port. In a large installation, these costs can be a significant part of the overall fiber build. The main advantage from a technical standpoint requires review of the standards for attenuation as shown in Table 1. This time, focus needs to be not on the distances allowed, but the Maximum Channel Connector Loss. Reverting to the discussion surrounding using 12 fiber subunits and the conversion required, in either method of conversion, an extra connection is introduced on both the TX and RX ends. The conversion method adds TWO extra connector loss points to the optical pathway. Link loss savings can be 0.2 to 0.5dB by removing these two connections! As we can see above, the longer the distance, the less connector loss is allowed in the link. In order to achieve the lowest optical loss in the system, the 8- Fiber Solution should be employed. How is the 8 Fiber Solution Implemented? We have already shown that the connectivity portion of this solution is accomplished by use of 8 fiber MPO connectors designed for use in parallel path systems; however the cable subunit construction is the major consideration and the largest departure from typical enterprise structures. Cables with subunits built on 8 fiber multiples need to be 78 T.W. Alexander Drive, Research Triangle Park, NC 27709 800.358.7378 www.sumitomoelectric.com 9
employed. For smaller fiber count cables, this is easily achieved by substituting 8 fibers in a buffer tube or under a single jacket to achieve this goal. As fiber counts climb, however, this can become problematic from an installation standpoint. To achieve higher fiber counts in loose tube cables, more buffer tubes must be employed to hold the requisite number of fibers, resulting in larger and larger cables. Additionally, these cables typically have a larger diameter than corresponding high fiber count ribbon cables. It is difficult to put multiples of 8 fibers in a loose tube cable for a number of reasons. First, most cables have been qualified both mechanically and in accordance with UL testing with up to, but not exceeding, 12 fibers per tube. Additionally, industry color code standards would necessitate up to half of the fibers to be striped. This would make it difficult and time consuming to terminate the fibers correctly in the field. What is the answer then? The best cable solution for 8 fiber systems is ribbon cables. Optical fiber ribbon cables have been in use for many years and are available in the same NEC categories as loose tube cables. Due to the unique construction of ribbon cables, most have been qualified with ribbons based on 8 fiber counts, as well as the more well-known 12 fiber counts. For higher fiber count cables especially (usually >96 fiber) 16 fiber, 24 fiber, and 32 fiber ribbons are available. In some cases, the ribbons can be separated in the field into 8 fiber component ribbons. This feature markedly improves productivity while doing terminations in the field since ribbonizing, as would be required for loose tube termination, is not necessary. Figure 7 shows the cross section of a 96 fiber ribbon cable with six, 16 fiber ribbons. Notice that in this arrangement, the manufacturer has encapsulated two 8 fiber ribbons into sixteen, making them easy to separate. Also of note is that the color code of the fibers is consistent once the 8 fiber subunits are isolated, making it fast and easy to terminate. Figure 7 78 T.W. Alexander Drive, Research Triangle Park, NC 27709 800.358.7378 www.sumitomoelectric.com 10
Cables made up of 8 fiber ribbons are available in outside plant, plenum, riser and indoor/outdoor versions with every fiber type included. Figure 8 shows the various layers of encapsulation that make up a 16 fiber ribbon. Figure 8 So what? I am only running 1Gbe today Granted, some data center administrators and operators are planning on running only 1Gbs or perhaps 10 Gbs systems in their new data centers. However, 8 fiber infrastructures will support these networks completely. Additionally, if at any time in the future of the data center, should speeds need to be increased, the passive subsystem already in place need not be altered at all. New end equipment can merely be attached and the system is ready to operate at 40Gbs, 100Gbs, and beyond. What about the future? While there are no guarantees that the network infrastructure will evolve into something completely different than that of today, it is probably safe to assume that future technologies will not be of the disruptive variety in order to allow for graceful growth of the network and to take advantage as much as possible of the installed base. In fact, 400G networks, the next probable step in higher speeds in the network, will easily be supported by the same 8 fiber based solution discussed by using 8 X 25Gbs lanes or 4 X 100Gbs lanes. In many ways, the 8-Fiber Solution is the best technology option for any data center for today and tomorrow. Conclusion While removal of existing 12 fiber subunit infrastructure networks solely for the purpose of installing an 8 fiber 40G/100G solution is an alternative to complete renovation of the data center infrastructure, it is clear that for new installations, which will now or can in the future be called upon to support parallel path transmissions, it is the solution of choice. Not only is the 8-Fiber Solution more cost effective by removing the need for conversion products at either end of the optical link, but it also provides more optical head room by removing two unneeded connections from the system. 78 T.W. Alexander Drive, Research Triangle Park, NC 27709 800.358.7378 www.sumitomoelectric.com 11
By eliminating unused fibers, the unnecessary connectivity points, and conversion module hardware, the 8-Fiber Solution ultimately yields both the best possible optical performance and cost savings for your data center. Moreover, the 8-Fiber Solution optimizes 40/100G and beyond, while backward compatible with legacy networks (1G & 10G), creating a real-time, scalable, and already future-proofed network. 78 T.W. Alexander Drive, Research Triangle Park, NC 27709 800.358.7378 www.sumitomoelectric.com 12
78 T.W. Alexander Drive, Research Triangle Park, NC 27709 www.sumitomoelectric.com 800.358.7378 78 T.W. Alexander Drive, Research Triangle Park, NC 27709 800.358.7378 www.sumitomoelectric.com 13