RESILIENT PACKET RINGS

RESILIENT PACKET RINGS Rajesh Sudarsan Ramrao Institute of Technology, University of Mumbai rajeshsudarsan@hotmail.com Abstract: An important trend in networking is the migration of packet-based technologies from Local Area Networks to Metropolitan Area Networks. The rapidly increasing volume of data traffic in metro networks is challenging the capacity limits of existing transport infrastructures based on circuit-oriented technologies like Synchronous Optical Network (SONET) and Asynchronous Transfer Mode (ATM). Inefficiencies associated with carrying increasing quantities of data traffic over voice-optimized circuit-switched networks makes it difficult to provision new services, and increases the cost of building additional capacity beyond the limits of most carriers' capital expense budgets. Packetbased transport technology, a natural fit with the now ubiquitous IP protocol, is considered by many to be the only alternative for scaling metro networks to meet the demand. Gigabit Ethernet is only the first step in the evolution of packet-based transport in the MAN. Though well suited for point-to-point and mesh network topologies, it is difficult to deploy Ethernet in ring configurations due to fairness problems and the spanning tree protocol. Yet, most of the existing fiber plant in metro areas is in ring form, because the incumbent transport technology, SONET, is typically deployed over fiber rings. Ring topology enables SONET to implement a fast (sub 50ms) protection mechanism that can restore connectivity using an alternate path around the ring in case of fiber cuts or equipment failure. Unlike SONET, Ethernet does not have a built-in fast protection mechanism. There are, therefore, great benefits in a new technology that can fully exploit fiber rings (in particular, ring resiliency) while retaining all the inherent advantages of a packet-based transport mechanism like Ethernet. The emerging solution for metro data transport applications is Resilient Packet Ring technology. It offers two key features that have heretofore been exclusive to SONET: efficient support for ring topology and fast recovery from fiber cuts and link failures. At the same time, Resilient Packet Ring technology can provide data efficiency, simplicity, and cost advantages that are typical to Ethernet. Resilient Packet Ring (RPR) technology is being standardized in the IEEE (802.17 Workgroup). This paper discusses today s solution to metro access networks, including their strengths and weaknesses, need for Resilient Packet Ring networking, the technological aspects involved and its benefits. 1. Introduction Solutions for metro/access networks have been proposed by numerous vendors, using various proprietary and standardized technologies. All of these solutions try to address the ambiguity of supporting existing legacy traffic and services, while migrating to a new future-proof data-optimized infrastructure. Legacy services are predominantly TDM 1

services, both voice and data, while future data services to end-users will be carried by Ethernet technology. Ethernet, and more specifically Gigabit-Ethernet, is expected to be the interface of choice connecting users to the public network due to its low cost, native LAN connectivity, synergism with IP, high bandwidth granularity and planned bandwidth scalability. Most of today s metro/access networks consist of different network elements for transport (typically SONET ADMs), layer-2 aggregation and switching (ATM, Frame Relay or Ethernet) and layer-3 IP routing. Recently, two major alternative approaches have been introduced; Solutions generally named Next Generation SONET Solutions, as well as Ethernet-based solutions. Next-generation SONET solutions integrate in a single network element add-drop, distributed cross-connect, wavelength division multiplexing and switching functionalities, while Ethernet transport solutions drive today s enterprise Ethernet switches to the metro network, replacing SONET as the transport layer, and establishing Ethernet as the technology for the aggregation and switching layer. 2.The Carrier's Challenge Today By 2002, data is expected to make up 80% of all telecommunications traffic the vast majority based on IP-standard. Although SONET/SDH equipment represents the incumbent optical networking technology, it was designed for voice traffic and does not scale to the demands of today s exponential data demands. Today's networks simply will be unable to handle the combination of increased data, voice and video traffic of the future. The protocols used on today s fiber metro rings become inefficient as data traffic increases. Carriers are in need of a new way to gain revenue from differentiated data services while supporting their legacy circuit-based voice systems. The ability to grow communications networks, while having the ability to address the explosion of data traffic in metropolitan networks, is a key concern for many carriers worldwide. 3. Requirements for Today s Carrier Services Carriers today are rapidly building out their Metropolitan Area Networks (MAN). In order to support both toll-quality voice services and new IP-based differentiated data services, carriers are concerned with several fundamental quality concerns, including network reliability, fairness, scalability, performance and efficiency. 1. The access network must be reliable, providing 99.999% uptime, redundant hardware, ring topology support, and fiber protection and restoration capabilities. 2. The access network must provide fairness, or the ability to allocate bandwidth in a manner that is consistent with defined Service Level Agreement (SLA) policies. Fair bandwidth management and allocation must be independent of where a subscriber sits on the network relative to other subscribers. 3. The architecture must be scalable to thousands of endpoints and must support a broad range of applications. The applications themselves will require bandwidth services that can scale from tens of kilobits per second to gigabits per second. Scalability is a critical requirement for the metropolitan network, as the network must be capable of adapting to an escalating number of users and applications. 4. The access network must be capable of supporting services with deterministic and predictable performance, including real-time packet voice and video applications that 2

require minimal latency and jitter. The ability to ensure performance guarantees by traffic type will enable integrated services on a common access network. 5. The network must be optimized for the ring topology of the MAN and for the predominant traffic type it carries data. The packet access network must incorporate service protection, but it must be accomplished efficiently. The system must be cost efficient to operate. 6. The network must provide support for legacy voice services. This fundamental revenue-bearing service demands both availability and reliability, and provides a significant revenue base as carriers build out their new service model. 4.The Limitations of SONET and Ethernet in Metro Rings: 4.1 SONET Most metro area fiber is in ring form. Ring topology is a natural match for SONET-based TDM networks that constitute the bulk of existing metro network infrastructure. However, there are well-known disadvantages to using SONET for transporting data traffic (or point-to-point SONET data solutions, like Packet over SONET [POS]). SONET was designed for point-to-point, circuit-switched applications (e.g. voice traffic), and most of limitations stem from these origins. Here are some of the disadvantages of using SONET Rings for data transport: Fixed Circuits. SONET provisions point-to-point circuits between ring nodes. Each circuit is allocated a fixed amount of bandwidth that is wasted when not used. For the SONET network that is used for access in Figure 1 (left), each node on the ring is allocated only one quarter of the ring's total bandwidth (say, OC-3 each on an OC-12 ring). That fixed allocation puts a limit on the maximum burst traffic data transfer rate between endpoints. This is a disadvantage for data traffic, which is inherently bursty. Waste of Bandwidth for Meshing. If the network design calls for a logical mesh, (right), the network designer must divide the OC-12 of ring bandwidth into 10 provisioned circuits. Provisioning the circuits necessary to create a logical mesh over a SONET Ring is not only difficult but also results in extremely inefficient use of ring bandwidth. As the amount of data traffic that stays within metro networks is increasing, a fully meshed network that is easy to deploy, maintain and upgrade is becoming an important requirement. SONET Ring SONET Ring Optic fiber Logical Circuit Fig: 1. SONET Access and Fully Meshed Network 3

Multicast Traffic. On a SONET Ring, multicast traffic requires each source to allocate a separate circuit for each destination. A separate copy of the packet is sent to each destination. The result is multiple copies of multicast packets traveling around the ring, wasting bandwidth. Wasted Protection Bandwidth. Typically, 50 percent of ring bandwidth is reserved for protection. While protection is obviously important, SONET does not achieve this goal in an efficient manner that gives the provider the choice of how much bandwidth to reserve for protection. 4.2 Ethernet Ethernet does make efficient use of available bandwidth for data traffic, and does offer a far simpler and inexpensive solution for data traffic. However, because Ethernet is optimized for point-to-point or meshed topologies, it does not make the most of the ring topology. Internet 802.1 ad GigE GigE GigE Fig: 2: Ethernet ring Unlike SONET, Ethernet does not take advantage of a ring topology to implement a fast protection mechanism. Ethernet generally relies on the spanning tree protocol to eliminate all loops from a switched network. Even though spanning tree protocol can be utilized to achieve path redundancy, it recovers comparatively slowly from a fiber cut since the recovery mechanism requires the failure condition to be propagated serially to each upstream node. Link aggregation (802.1ad) can provide a link level resiliency solution, but it is comparatively slow (~500ms vs. ~50ms) and not appropriate for providing path level protection. Data optimized technologies like Ethernet, do not provide the carrier-class fairness guaranteed in RPR-based networks. Ethernet switches prioritize traffic locally, on every interface on the ring, thus creating unfair conditions for traffic that has to traverse through several nodes on its way to the destination node. Figure 3 demonstrates the lack of fairness in Ethernet-based networks. In this example a 10Gbps Ethernet ring connects between the nodes. s 4, 5 and 6 all transmit traffic to node 1, creating congestion between node 1 and node 6. Under these conditions, node 6 will get half of the total available bandwidth, while nodes 4 and 5 will each get 25% of the available bandwidth. 4

2.5Gbps 2.5Gbps 5Gbps 10Gbps Fig 3: Unfairness in Ethernet Ring 5. Resilient Packet Ring Technology: Future of Metro Transport Resilient Packet Ring (RPR) technology was designed to combine SONET carrier-class functionalities with Ethernet s high bandwidth utilization and granularity. Additionally, RPR technology offers fairness that has been lacking in today s Ethernet solutions. WAN Metro Core RESILIENT PACKET RING TRANSPORT Metro Access LAN RPR is a new MAC layer technology, being standardized in the IEEE 802.17 workgroup, that employs spatial reuse to maximize bandwidth utilization, provides a distributed fairness algorithm, and ensures high-speed traffic protection similar to SONET Automatic Protection Switching (APS). This technology leverages the scalability and reliability of optical networking to significantly enhance packet switching in the MAN and 5

the WAN. RPR is a Layer 2 technology and is therefore Layer 3 protocol and addressindependent. RPR allows full ring bandwidth to be utilized under normal conditions and protects traffic in the case of a nodal failure or fiber cut using a priority scheme, obviating the need for SONET-based protection. Additionally, because RPR MAC layer can run on top of SONET framing, RPR-based networks can provide performance-monitoring features similar to those of SONET. IEEE 802.17 will define a MAC protocol for use in local-, metropolitan- and wide-area networks, employing existing physical-layer specifications such as Gigabit Ethernet and 10 Gigabit Ethernet, as well as OC-48/STM-16 (2.5Mbps) and OC-192/STM-64 (10Gbps), for transfer of data packets at rates scalable to many gigabits per second. Using predefined standards-based physical layers will both speed up the standardization process and allow for the straightforward and seamless attachment of IEEE 802.3- and SONET-based LANs to the RPR-based MAN. Creation of the new MAC layer, optimized for MAN traffic and topologies, will enable RPR to deliver cost-effective, quickly provisioned, secure and reliable data links on a service provider's existing fiber infrastructure. End users will be able to "turn up" bandwidth on demand at rates from 1Mbps to 1Gbps and beyond. RPR technology offers all of these carrier-class functionalities, while at the same time keeping Ethernet s advantages of low equipment cost, high bandwidth granularity, and statistical multiplexing capability. 5.1 Technical aspects of RPR Networking 5.1.1 Bandwidth Fairness Fairness is one of the most important features in carrier-class networks. Ring bandwidth is a shared resource, and is vulnerable to exploitation by individual users or nodes, in the network version of the "the tragedy of the commons." Fairness is achieved when the traffic characteristics of two service flows that have the same service level agreement (SLA) are identical, regardless of their network source and destination. A fairness algorithm is a mechanism that gives every node on the ring a predetermined "fair" share of the ring bandwidth, ideally without the straitjacket of a provisioned circuit. The RPR protocol can guarantee fairness across the metropolitan network. Each node on the metro ring executes an algorithm designed to ensure that each node on the ring will get its fair share of bandwidth. However, ring nodes use the spatial reuse capability to use bandwidth that is greater than their fair share on local ring segments, as long as it doesn t affect other ring nodes. More specifically, the ring supports weighted fairness, proportional to the bandwidth each user buys. In order to avoid the complexity involved in implementing a weighted fairness algorithm, a combination of peak-rate limiting at the tributary interface (in both ingress and egress directions) and plain fairness between nodes on the ring can be employed to achieve a similar result. Figure 4 demonstrates the operation of the fairness algorithm. Before employing the fairness algorithm all nodes transmit at their peak usage rate. At some point 1 experiences congestion and requests upstream nodes to reduce their transmission rate to the allowed rate. After convergence, each node will receive its fair share of bandwidth, while nodes that can locally transmit higher rates, like nodes 2 and 3, without generating congestion on other nodes, continue to transmit their peak usage rate. 6

(a) (b) (c) Fig 4: Fairness in RPR ring (d) 5.1.2 Spatial Reusability One of the main features of the RPR protocol is its spatial reuse capability a technique that is used in ring topologies to increase the overall efficiency of the ring. This is achieved by allowing traffic to be passed bi-directionally on the ring along the spans between source and destination nodes, but without requiring the use of that same bandwidth over other segments of the ring. The destination node strips its packets from the ring, thereby providing the full bandwidth on other ring segments for utilization by other packets. Destination stripping is different from earlier ring-based protocols, such as token ring and fiber distributed data interface (FDDI) protocol, wherein traffic was removed from the ring by the source node consuming unnecessary bandwidth. Spatial reuse mechanisms in metropolitan rings are extremely advantageous. They can improve bandwidth efficiency greatly and increase the effective bandwidth that is available on the ring. 7

7Gbps 2Gbps 8Gbps Fig 5: Spatial Reuse in an RPR ring Figure 5 demonstrates how spatial reuse works. Assume the ring runs RPR on top of OC-192c, and there s a substantial amount of non-guaranteed traffic over it. In this example node 6 is sending 2Gbps traffic to node 1 and node 5 is sending 7Gbps traffic to node 1. On a traditional SONET ring, this would fill up the ring capacity, while in an RPR ring, nodes 3 and 2 can continue to utilize the full ring bandwidth. 5.1.3 Ring-level Aggregation A key feature of the RPR protocol is the ability to perform aggregation at the ring level rather than at the node level. Ring-wide aggregation allows the statistical multiplexing of a much larger number of customers, for example, 100 on the entire ring rather than 10 on a single node. This difference leads to very significant statistical gains that allow much higher over-booking ratios to be implemented. 5.1.4 Fast Protection and Restoration The feature that more than any other makes RPR a carrier-class technology is its fast self-healing capability that allows the ring to automatically recover from a fiber cut or node failure by wrapping traffic onto the alternate fiber within the SONET-class 50msec restoration timeframe. Not only does RPR technology provide carriers with SONET-class fast protection and performance monitoring capabilities, it enables them to achieve it without dedicating protection bandwidth as in the case of SONET, thus eliminating the need to sacrifice 50% of the ring bandwidth for protection. RPR uses counter rotating physical fibers to carry working traffic. Protection messages are exchanged quickly, informing other nodes of the status of the network. During protection events, the multiple traffic priorities on the ring ensure preferential handling for higher priority packets. The network will protect and re-establish connectivity automatically. A well-defined protection mechanism is needed for plug-and-play operation of RPR nodes in order to allow the insertion and removal of nodes without provisioning. Figure 6 shows an example of the data paths taken before and after a fiber cut event. Before the fiber cut, node 3 sends traffic to node 1 via node 2 (Figure 6a). When the fiber cut occurs (between node 1 and node 2), node 1 will wrap the inner ring to the outer and node 2 will wrap the outer to the inner ring. After the wrap, traffic from node 3 to node 1 initially traverses through the non-optimal path (passing through node 2, back 8

to node 3 and into the originally assigned port on node 1 Figure 6b). This is immediately done and can inherently meet the 50ms criteria, as it is a local decision of the nodes that identify the fault (in this case, nodes 1 and 2). Subsequently, higher layer protocols discover the new ring topology and a new optimal path is used (Figure 6c). This can take several seconds to converge in a robust manner, after which the double ring capacity is restored (even though not evenly distributed, depending on the location of the fault and the ring traffic pattern). Thus, unlike SONET rings that always pay the redundancy tax to achieve protection, after a failure an RPR ring reduces its capacity only for several seconds. (a) (b) Fig 6: Fast Protection in an RPR ring (c) In summary, RPR s fast protection and restoration capability prevents service loss for high priority critical traffic, just as SONET does, enabling carriers to continue to support their existing critical traffic over a data optimized network, without compromising their guaranteed 99.999% service availability. 5.1.6 Physical Layer Versatility The Resilient Packet Ring standards now in development create only a new Media Access Control (MAC) addressing scheme designed for ring-based topologies. This has the advantage of leaving Layer 1 open. Hence, Packet Ring technologies will be compatible with Ethernet, SONET, and DWDM physical layer standards. 9

IP, MPLS, Others 802.17 Resilient Packet ring Optical Transmission (SONET, Ethernet, WDM or new Fig 7. RPR are Physical and network layer independent This means that Resilient Packet Rings can operate over an existing SONET configuration, creating better data capabilities without the need to replace SONET equipment or abandon SONET for TDM voice traffic. Similarly, a DWDM configuration can make use of Resilient Packet Ring at Layer 2 for some or all wavelengths. Finally, where dark fiber is available, a Resilient Packet Ring can operate without the need to purchase expensive Layer 1 gear. 5.2 Quality of Service (QoS) Qua lity of Service (QoS) is required in order to let a carrier effectively charge for the services it provides. Furthermore, different service quality features are required for different types of applications. Data transfer applications require low packet loss rate, while real-time applications such as voice require low latency and low delay variation. There are several parameters that more than others govern the characteristics of a delivered service: Service availability, delay, delay variation, and packet loss rate. The most important parameter for a carrier selling bandwidth services is service availability. Service availability depends on the reliability of the network equipment as well as on the network survivability characteristics. Delay occurs in a network when a packet waits in a switch queue for other packets to pass. Delay variation is the difference in delay of several packets belonging to a common traffic flow. High-frequency delay variation is called jitter, and low-frequency delay variation is called wander. During a protection event, and until high layer protocols converge, the available bandwidth on an OC-192 RPR ring reduces to 10Gbps. In order to guarantee service for every traffic flow under these circumstances, a portion of each flow s bandwidth should be allocated on this guaranteed throughput. This way, even during a protection event service availability is guaranteed. Traffic delay is a minor issue in high throughput rings. An end-to-end delay of several tens of milliseconds is usually regarded as acceptable, while in an OC-192 ring, the expected delay for the longest Ethernet frames (1518 bytes) is shorter by three orders of magnitude (about 5µ sec). Delay variation control is essential to ensure proper transport of TDM services over an RPR ring. Control can be achieved by synchronizing the nodes on the RPR ring by a common timing source. Policing is another tool that lets a carrier differentiate between service classes. In an underutilized network that does not employ policing customers may get a better service than they ve paid for. A per-flow policing ensures that customers get only the service they ve paid for. 10

6. TDM Circuit Emulation over RPR RPR technology is designed to enable service providers to build carrier-class data optimized networks. However, most of the revenues of carriers today come from TDM services over their legacy TDM network. Circuit emulation of TDM services over RPRbased networks enables carriers to keep their current high-revenue TDM services as well as offer new TDM services over their packet optimized network, and gradually migrate to Ethernet-based services. Thus, delivering TDM services over RPR-based networks is an essential complementary capability to the RPR technology. Since TDM services are delay and delay variation sensitive, circuit emulation of TDM services requires strict jitter and wander control as well as end-to-end synchronization over the ring. Standardization of TDM circuit emulation over packet networks has been progressing in the IETF. Its goal is to enable end-to-end delivery of TDM circuit emulation services over metropolitan as well as long-haul packet networks. Currently, standardization is focused on circuit emulation services over Multi-Protocol Label Switching Protocol (MPLS). MPLS labels and a new circuit emulation header are used to encapsulate TDM signals and provide the Circuit Emulation Service over MPLS (CEM). Essentially, a TDM data stream is segmented into packets and encapsulated in MPLS packets. Each packet has one or more MPLS labels, followed by a CEM header to associate the packet with the TDM stream. The outside label is used to identify the MPLS Label Switch Path (LSP) used to tunnel the TDM packets through the MPLS network, whereas the interior label is used to multiplex multiple TDM connections within the same tunnel. Until circuit emulation of TDM over MPLS is fully standardized and interoperable, aggregation of the TDM traffic headed outside of the metro ring can be done in a central point, and handed over to a long-haul legacy TDM network. An RPR-based network, supporting transport of TDM services can aggregate, using a Virtual Digital Cross- Connect, DS-1, DS-3, OC-3, OC-12 and OC-48 services into a single OC-48 or several OC-12s in the carrier s PoP, and hand them over to a long-haul TDM network. Long Haul TDM Network OC48 DS3 OC12 OC3 OC3 DS3 OC12 OC12 Fig 8: Circuit Emulation of TDM services 11

Figure 8 demonstrates how circuit emulation of TDM services is supported over RPRbased networks. In Figure 5, nodes 2, 3, 4 and 5 serve TDM users with DS-3, OC-3 and OC-12 services. In this example the DS-3 services are connecting users within the same ring (between nodes 2 and 5) and the traffic is carried transparently over the RPR ring, while the OC-3 and OC-12 services, connecting users in other metro rings, are aggregated into a single OC-48 port in the carrier s PoP (node 1), and handed over to a long-haul TDM network. 7. Conclusion: Resilient Packet Rings free Ethernet traffic from the bonds of distance limitations over point-to-point links, and expands it to run over robust optical ring architectures ultimately spanning the globe. It is a key technology that enables enterprise Optical Ethernets built by enterprises or procured as managed Virtual Private Ethernet services. In either case, enterprise Optical Ethernets not only simplify the networking environment but open up opportunities for routing, storage and server centralization, low latency network computing, and improved overall knowledge worker productivity. RPR addresses the current and future trends in MAN and WAN networking. It brings a highly efficient packet-switched technology into carrier networks to address the growing data-traffic trends in a cost-effective way. It addresses the unique requirements of metropolitan, ring-based, data-transport networks by preserving carrier-class delivery of voice and circuit-switched traffic while adding capabilities for very rapid service velocity and efficient transmission of packet-switched data. These are the new capabilities that are required by the carriers to offer new-age services. The unanimous approval by the IEEE to work on a standard and the momentum that it has gained in the standardization process through the participation of users, vendors and suppliers guarantees its success as a technology that will be future-proofed and rooted in the pragmatics of evolving today s networks. 12

GLOSSARY ADM ATM CWDM CSMA/CD DS1/DS3 DSL DWDM E1/E2 IP ITU LAN LEC MAC MAN MDU MLPPP MPLS MTU OC-3/OC-12 OC-192 PDH PIM POS PPP PVC QoS SDH SONET SLA SRP T1 TCP/IP TDM VLAN WAN WDM Analog to Digital Multiplexer Asynchronous Transfer Mode Coarse Wave Division Multiplexing Carrier Sense Multiple Access With Collision Detection Digital Signal, Level 1 (1.54 Mbps) or 3 (44.7 Mbps) Digital Subscriber Line Dense Wave Division Multiplexing European Trunk 1/2 (2 Mbps/34.3 Mbps) Internet Protocol International Telecommunications Union Local Area Network Local Exchange Carrier Media Access Control Metropolitan Area Network Multiple Dwelling Unit Multi Layer Point-to-Point Protocol Multiple Protocol Label Switching Multiple Tenant Unit Optical Carrier 3/12 (155 Mbps/622 Mbps) A SONET transport with the optical carrier at 9.953 gigabits per second Plesiochronous Digital Hierarchy Protocol Independent Multicast Packet over SONET Point-to-Point Protocol Private Virtual Circuit Quality of Service Synchronous Digital Hierarchy Synchronous Optical NETwork Service Level Agreement Spatial Reuse Protocol Trunk 1 (1.544 Mbps) Transport Control Protocol/Internet Protocol Time Division Multiplexing Virtual LAN Wide Area Network Wave Division Multiplexing REFERENCES 1. Corrigent Systems, Technical note on Resilient Packet Ring Technology, 2001. 2. Nigel Cole, John Hawkins, Martin Green, Raj Sharma and Kanaiya Vasani, A report on Resilient Packet Rings for Metro Networks. 3. Tony Rybczynski, Nortel Network, Ethernet and Optics Come Together: Resilient Packet Rings, 2001. 4. Technology White Paper, River STONE Networks, Packet Ring Technology: The Future of Metro Transport, 2001 13