GIGABIT ETHERNET ADVANCEMENTS EXTEND NETWORK REACH TO METROPOLITAN AREA NETWORKS DATA COMMUNICATIONS MANAGEMENT.

Transcription

1 DATA COMMUNICATIONS MANAGEMENT GIGABIT ETHERNET ADVANCEMENTS EXTEND NETWORK REACH TO METROPOLITAN AREA NETWORKS James Trulove INSIDE Gigabit Ethernet Basics; Extending the Ethernet into the Campus; Metropolitan Area Ethernet Extension; Gigabit Ethernet s Advantages over Conventional WAN Links; Gigabit Ethernet s Advantages over ATM WAN Links; Proprietary Considerations GIGABIT ETHERNET BASICS To fully understand the implications of extending Gigabit Ethernet into the Metropolitan Area Network (MAN), one needs to be familiar with the basics of the technology. Gigabit Ethernet is a natural extension of conventional 10 Mbps and 100 Mbps Ethernet technology. Through a series of extensively engineered standards, promulgated by the Institute of Electrical and Electronic Engineers (IEEE), Ethernet technology has progressed in both media types and in traffic pattern. The media evolution has included coaxial cable, twisted pair copper, and fiber. At this point, coaxial cable is not used in new installations and will soon disappear from use. How- PAYOFF IDEA Recent innovations in Gigabit Ethernet transmission have increased link distances to over 70 km. This makes possible the extension of Ethernet networks across dark fiber to interconnect an enterprise s buildings, while maintaining native Ether-packet topology. The link distances extend far beyond the limits of a simple building complex or campus into the larger local area of the Metropolitan Area Network (MAN). Link-speed increases of several orders of magnitude over conventional WAN links are possible at very attractive costs. The technique also has significant advantages over traditional WAN routing as well as ATM transport, as no conversions to link protocols or LAN emulation are needed. This reduces the complexities and costs of transporting Ethernet data packets between remote buildings, within the range of the gigabit link. In addition to simple LAN bridging, service providers can offer dramatically increased bandwidths for Internet and switched-voice connectivity.

2 EXHIBIT 1 Fiber Standards IEEE 802.3z Mode Fiber Mode Wavelength Fiber Size a Distance Limit 1000BaseSX Multi 850 nm 50/100 µm 500 to 550 m 1000BaseLX Multi 1300 nm 50/100 µm 550 m 1000BaseLX Multi 1300 nm 62.5/125 µm 550 m 1000BaseLX Single 1300 nm N/A 5 km 1000BaseSLX Single 1550 nm N/A 10 km 1000BaseELX/ZX Single 1550 nm N/A km a Expressed as core/cladding diameters in micrometers. ever, twisted pair copper cables and fiber-optic cables continue to evolve as technical refinements allow their use at higher and higher speeds. Twisted pair cable, at the Category 5e performance level, can readily carry Gigabit Ethernet within the 100-m horizontal links of the structured-cabling LAN. However, this distance is far too short to be used between most buildings, and is thus restricted to the building infrastructure. Fiberoptic cable, on the other hand, does not suffer from the severe distance restrictions of copper. A particular type of optical fiber, single-mode, is used in the extended distances of today s Gigabit Ethernet connections. In addition, the longer transmission wavelengths are used for greater range. Exhibit 1 shows the common fiber constructions and their respective usable link distances, in accordance with accepted standards. Note that the longer-distance standards are merely proposed at this time, although several manufacturers have begun manufacture of equipment meeting these specifications. The distance limitations of 1000BaseSX and 1000BaseLX-multimode are governed by the operating characteristics of multimode fiber, namely, a relatively low bandwidth-distance parameter. This parameter, expressed as the product n-mhz-km, gives an indication of the usable optical bandwidth of a particular optical fiber over a specified cable distance. Conversely, one can infer the usable distance over which a signal of a specified bandwidth may be carried. Although the 70+ km distances of the ELX/ZX standards seem enormous, in comparison to the standard Gigabit Ethernet fiber distance limits, the bandwidth capability of single-mode fiber is barely strained. It is estimated that one can carry multiple wavelengths of OC-768 (40 Gbps) over ordinary single-mode fiber to distances exceeding 80 km. Large Ethernet networks are generally constructed in a series of subnets, utilizing Internet protocol. Packets are conveyed from origin to destination by a series of switching (at OSI Layer 2) connections within pathways of a single subnet, and by routing (at OSI Layer 3) connections between different subnets. A typical Ethernet network consists of arrays

3 of switching and routing elements that serve to unambiguously guide the packets through the myriad of network connections and concentration points. The switching and routing functionality may exist in the same device or in separate modules (also called blades) within a switching chassis. The advent of Layer 3 switching capabilities in these switches has greatly reduced packet latency and increased packet throughput for routed data connections. A typical multi-floor building has a single Ethernet switched-port array on each floor to serve the workstations on that floor. Each floor switch is usually interconnected via a fiber (or more rarely, copper) high-speed link (called an uplink) to a centralized master switch, which is often a chassis-based Layer 2/Layer 3 device. The master switch and the links to each floor switch effectively become the building s network backbone. EXTENDING THE ETHERNET INTO THE CAMPUS When a series of buildings, each with an internal Ethernet network, are interconnected, they effectively form an extended local area network. In effect, the local building network of a single building is extended to multiple buildings nearby. All buildings may be on the same IP subnet, if the number of buildings is small and if there are no alternate paths for redundancy. But more typically, each building is on a separate subnet, with routing or Layer 3 switching enabled on the inter-building links. Multiple paths are allowed between buildings and the central-site for redundancy. Routing algorithms allow the data packets to be redirected through the alternate path in the event the main connection is broken, either by a fiber cut or equipment failure. Inter-building distances on a typical small campus are often within the 550-m range of 1000BaseLX (1300 nm wavelength) using either 62.5/125 µm or 50/125 µm multimode fiber. 1 For longer distances on a larger campus of buildings, or for short-range links to nearby buildings, the 1000BaseLX interface may use single mode fiber to reach distances up to 5000 m (5 km). Fiber-optic cable offers several advantages for campus Ethernet backbones. The most obvious, of course, is the extended distance over which it will operate, relative to copper cable links. In addition, fiber offers inherent immunity to electrical and electrostatic disturbances. The two most familiar of these disturbances are grounding problems and lightning. Significant electrical currents can be set up between buildings when they are not properly bonded (or grounded) together. Sometimes, these grounding problems are inadvertently a result of the buildings connection to the power grid. For example, adjacent buildings may be fed power from different power transformers, resulting in an incidental electrical potential between their respective building safety grounds. A metallic cable between such buildings might carry significant unwanted ground

4 currents, and thereby offer hazardous electrical potentials to workers and occupants. Electrical discharges, characterized by lightning in the most severe case, can easily damage sensitive (and expensive) switches and routers if connected between buildings via copper cable. Although typical switch interface can withstand 1000 to 1500 volts, the typical electrical discharge might range from 10,000 to 1 million volts. In any event, the copper cable run lengths are so restrictive that copper is usually quite unsuitable for inter-building Ethernet cabling. METROPOLITAN AREA ETHERNET EXTENSION The availability of Gigabit Ethernet links of as much as 80 km creates an opportunity to interconnect enterprise buildings over a fiber link, without using other more conventional modes of packet transport. A conventional MAN link uses a wide-area protocol, or a LAN emulation, to encapsulate Ethernet packets for transport over moderate to large distances. Examples of these transport methods are conventional T-carrier (T-1, DS-3) WAN links, SONET optical carrier (OC-3, 12, or 48) rings, and ATM links, which can use SONET physical interfaces, with or without the ring protection. In contrast, the deployment of moderately long Gigabit Ethernet links avoids the conversion, emulation, and encapsulation complexity and simply makes the MAN connections in Ethernet s native mode of transport. At the lower speeds of 10 and 100 Mbps Ethernet, it was possible to run links of up to about 2 km. To do this, the standard collision-detection mechanism of a link is turned off, and the link operates in a full-duplex mode, where data is transmitted on one fiber, simultaneously, as data is received on the paired fiber. Because collisions in a point-to-point duplex fiber environment have virtually no meaning, operating in the fullduplex mode is very compatible with the link requirements. Gigabit Ethernet links work the same way when utilized in this longdistance fashion. Thus, it is possible to extend these link distances from 70 km to 80 km, using advanced laser transmitters, high-sensitivity receivers, low-loss single-mode fiber, and low-loss connectors. Each of these technologies is crucial in making Gigabit Ethernet go the distance, so to speak. Single-mode fiber has much lower losses at 1550 nm, although the laser diodes for this wavelength are more expensive than for the lower wavelengths. Advances in single-mode fiber manufacturing have led to optimally low cable losses and higher bandwidth-distance products. In addition, the new small-form factor (SFF) fiber-optic connectors, such as the LC and the MT-RJ, have significantly lower termination and mating losses than their older ST and SC counterparts. In fact, with optimal termination methods, it is possible for the Systimax LC connectors to achieve inser-

5 tion losses of as little as 0.1 db. Tight alignment specifications for the laser/fiber interface also increase the effective range of these new links. There are generally two scenarios where Gigabit Ethernet connections are used in a MAN. In the first, dark fiber is made available to an end user for data interconnection. In the second, a service provider utilizes its private fiber network to provide Gigabit Ethernet connectivity. Take a look at these, one at a time. In many cases, public institutions (such as school districts or state and local governments) may make an arrangement with a local cable television provider to place dark fiber to each of its buildings within an area of a city. The installation of this fiber is often done in conjunction with the granting of a cable franchise, or rights of way for the cable company s lines. In essence, the provision of extra fiber to the public sector is an additional fee or cost of business that must be accommodated to win the franchise or contract. The school or municipality may simply connect the fiber link for a building directly to a switch or router with an extended Gigabit Ethernet interface. At the central site, connectivity for data, Internet, IP voice, or IP video is made to support the needs of the enterprise. The transport of ordinary data over an extended Gigabit Ethernet link is a logical progression from sending data around a LAN. Internet data is really no different, except that IP is used exclusively, whereas other protocols might be used in the LAN. Now that IP telephony is becoming quite commonplace, it is easy to envision using IP phones at smaller remote buildings, or full IP PBXs at larger ones. The bandwidth, with overhead, is slightly more than 8 Kbps per voice call. With a gigabit bandwidth, a very large number of calls may be accommodated with no noticeable effect. Likewise, IP videoconferencing over the H.323 protocol can easily be accommodated within a 1000-megabit link, because normal quality videoconferencing only requires 384 Kbps. A service provider may offer high-speed virtual networking, virtual private networking (VPN), or dark fiber connectivity to enterprises within its service area. Local telephone operating companies often run fiber cables between the serving central office and larger commercial buildings. It is quite practical to run more fibers than are actually needed, given the relative cost of the few extra fibers and the much larger cost of cable installation. These extra fibers can be converted to Gigabit Ethernet use, merely by placement of a gigabit-speed switch at the central office (CO) and at the customer premise. The Gigabit Ethernet link to the CO could provide a really high-speed Internet connection for the building, as well as eventually serving as a voice-over-ip (VoIP) path. Alternatively, the service provider could provide a virtual LAN (VLAN) or virtual private network (VPN) to the user over its IP backbone, through direct Gigabit Ethernet connections.

6 In addition to the classic Bell local service provider, a number of alternative local carriers (competitive local-exchange carriers, CLECs) are entering almost every metropolitan market. These carriers often have equipment co-located at the very same COs, and can easily operate over the same fiber loops as the telco. In addition, the CLEC can also have its own fiber run between its point of presence and potential user buildings. This scenario lends itself very well to larger downtown areas, with lots of tall buildings and lots of potential customers. If one compares the gigabit-per-second bandwidth of Gigabit Ethernet to the conventional WAN or ATM links (from 1.5 Mbps to 155 Mbps), the advantage of native Ethernet at gigabit rates is rather self-evident. Let s not look at this and some of the other advantages over conventional links. GIGABIT ETHERNET S ADVANTAGES OVER CONVENTIONAL WAN LINKS Conventional WAN links consist of a standard telecommunications circuit, plus two routers (one at each end of the circuit). The speed of the WAN link can vary from a measly 56 Kbps to a whopping 45 Mbps. The former is a synchronous digital data link over a leased circuit, or perhaps a Frame Relay circuit, at a circuit speed of 56 Kbps. 2 From there, one can jump up to a Fractional T1 circuit, often a 256-Kbps Frame Relay link, or to a full T1 at Mbps throughput. Fractional DS-3 is relatively rare; so for most users, the next jump is to 45 Mbps at a very high cost. In all these cases, some significant circuit termination equipment is needed (see Exhibit 2). One will need an appropriate DSU/CSU and a router (or possibly a bridge). In some cases, the two are combined into one unit, such as a router with a built-in DSU/CSU. Often, configuring the router is fairly complex. In addition to setting up the IP address, default gateway, and static routes, one will invariably have to set up the circuit data rate, link protocol, link authentication, SPIDs or PVCs, and a number of other parameters. One s router must be able to route the appropriate network protocols, such as IP, IPX, and AppleTalk; and there may be an additional configuration involved in these routing protocols. There are both standard and proprietary link protocols, and the pair of routers must be set up to use the same protocol or they will not link up. All this adds a layer of complexity that would not have existed in a simple Ethernet link between the two sites. In addition, the operation of these conventional routers includes latency in the processing of each packet that can cost real throughput. For example, if the protocol requires an acknowledgment before the next transmission, it will cause a delay while each router, in turn, processes the entire packet, routes it to the link interface, and decodes the packet at the other end. This latency can be as much as 100 times longer than the packet transit time through a Layer 2/Layer 3 Ether-

7 EXHIBIT 2 Wide Area and Metropolitan Area Alternatives net switch. Not having to go through that processing and translation of the Ethernet packet at either end (to encapsulate it into a variation of HDLC link protocol) could really speed things along. That is exactly what the new longer-distance fiber links for Gigabit Ethernet provide: a plain-old-ethernet-service (one might call it POES). Nothing could be simpler. If there is no need to use subnet differentiation between the ends of the link, one can literally hook up the cables and turn the power on. What could be simpler? If one wants to use subnets, then one needs a Layer 3 switch at each end; but otherwise, one eliminates about 95 percent of the setup parameters needed for the conventional WAN router. In fact, there are now simplified Layer 3 switches that learn the routes automatically. They are truly plug-and-play. The secret is that the MAN-class Gigabit Ethernet links allow one to remain pure Ethernet end-to-end. GIGABIT ETHERNET S ADVANTAGES OVER ATM WAN LINKS Another way that is becoming popular to increase the link speeds of MAN connectivity is to use an ATM backbone. In this case, the user installs an ATM access device (or perhaps an ATM blade in a router) and obtains an ATM local loop from the local carrier. This ATM service can be as low as T1 speeds, but normally DS3 (45 Mbps), OC-3 (155 Mbps), or even OC-12 (622 Mbps) circuits are used to more fairly approach the

8 speed of the LAN backbone. In some instances, the DS3 or OC-3 ATM loop can be carried over a SONET ring back to the CO. On a scale of affordability, most users know what a T1 link would cost, without the ATM formatting. One can expect the ATM line to cost 50 to 100 percent more, even at that low data rate. For most local carriers, the upgrade to DS-3 is about ten times the cost of the T1, and OC-3 is proportionally more. ATM is very expensive for the carrier to provision, and the circuit setup is likewise fairly complex. Several different classes of service are typically offered, with a decrease in price for the variablerate services. On the user side, one must contend with the complex configuration of the ATM device, and all parameters must be perfectly matched to the virtual circuit parameters of the carrier. Because ATM granularity is a 53- byte cell, the larger Ethernet packets must be broken up and reassembled at the remote end. In addition, the various features of Ethernet transport must be simulated to prevent connection problems. This process in the ATM world is called LAN emulation, or LANE services. To manage an ATM transport layer, the network technicians will require extensive additional training in this technology, in addition to their base knowledge in Internet protocol, server operating systems, and general Ethernet operation. The obvious advantage of using Gigabit Ethernet directly in the MAN is that one avoids the additional expense and complexity. Experienced network technicians and engineers have essentially all the specialized knowledge they need to design, install, and maintain an extended Gigabit Ethernet network. Also, one totally avoids the capital expense of the ATM access devices and the recurring expense of the extra-cost ATM services from one s carrier. For the limited distances (if 70 to 100 km can be considered limited) of long-reach Gigabit Ethernet links, one can achieve total network continuity, without resorting to cumbersome, expensive, and slower WAN technologies. PROPRIETARY CONSIDERATIONS At this writing, several manufacturers including Lucent Technologies and Cisco Systems have introduced long-reach Gigabit Ethernet interfaces. All of these manufacturers have the intent of making their particular method the IEEE standard, and each will insist that its version is the de facto standard. The truth of the matter is that one cannot currently guarantee interoperability between these competing standards, despite their similarities. To deploy this new technology, one should plan to install same-manufacturer equipment at each end of the fiber circuit. In the instance where a local carrier, perhaps a CLEC, is providing the connectivity, one may be required to use the products of a particular

9 manufacturer. This is because the local carrier will have to terminate one s link in a Gigabit Ethernet switch at the local exchange office, and that switch will use one of the proprietary standards. The Gigabit Ethernet equipment interface on one s premises will have to be compatible. In the near term, standards will be agreed upon to make all of these competing interfaces compatible. For the time being, use caution in choosing and specifying these new extended range interfaces. SUMMARY The new Gigabit Ethernet extended range options offer enterprises the ability to run native Ethernet connections over distances that previously required a WAN router or ATM transport. Distances of 70 to 100 km can be achieved on single-mode fiber that is designed to optimize the nm optical window. Maintaining an all-ethernet MAN eliminates some of the cost and complexity of traditional methods, increases bandwidth to the level of the LAN backbones, and may be offer network management advantages. Users of this technology should be aware that the standards for long-range Gigabit Ethernet are under development, and there may be some interim compatibility issues between competing manufacturers. Nevertheless, the advantages of this technology are significant and should be considered by anyone with a MAN application. Notes 1. The 50/100 µm multimode fiber will also support 500 to 550-m runs, utilizing the 1000BaseSX interface at 850 nm. However, this fiber size is not often found in existing multimode fiber installations. 2. In this instance, the 56 Kbps is a synchronous data circuit carried over a two- or four-wire dedicated local loop back to the CO, where it is inserted into a DS0 channel of the T-carrier network for transport to the other end. This type of circuit is quite different from the so-called 56 Kbps modem links that are popular in PCs. James Trulove has more than 25 years of experience in data networking with companies such as Lucent, Ascend, AT&T, Motorola, and Intel. He has a background in designing, configuring, and implementing multimedia communications systems for local and wide area networks, using a variety of technologies. He writes on networking topics and is the author of LAN Wiring, An Illustrated Guide to Network Cabling and A Guide to Fractional T1, and the editor of Broadband Networking, as well as numerous articles on networking.