CARRIER-CLASS PROTECTION AND RESTORATION FOR ETHERNET MANS A White Paper from Metrobility Optical Systems
T h e C h a l l e n g e Building ultra-fast protection and restoration for mission-critical Ethernet networks Smaller staffs, greater competitive pressure and customers increased uptime requirements combined with shrinking technology budgets represent an enormous challenge for today s service provider and his enterprise IT counterpart. To find a solution to the challenge of being continuously online while dealing with these constraints, we need to examine how innovations in optical Ethernet are increasing the self-recovery capabilities of the metropolitan network (MAN) while decreasing the costs of achieving faster than SONET recovery times. This paper reviews the various line protection and restoration methodologies traditionally deployed, standards under development and newer technologies available today that far exceed basic fail-over and recovery requirements, by providing the advanced levels of protection and ultra high-speed restoration that form the foundation for fully self-sufficient networks. 2
D r i v i n g t h e C h a n g e f o r I m p r o v e d F a u l t P r o t e c t i o n a n d R e c o v e r y All of us keep wondering when business will once again return to normal. But as much as some things have changed over the last few years, there are some things that have remained constant. It s still a 24/7 world and will be for the foreseeable future. Businesses have to be continuously online in order to remain competitive. Downtime can now cost tens of thousands to millions of dollars per hour. In short, today s business climate demands self-sufficient networking infrastructures, often with five 9 s availability only 300 seconds of downtime per year. Without the network there is nothing Networking is the key infrastructure component tying together all the data storage and processing power on which industry relies. When the physical network fails and up to 45% of all network outages are caused by problems at the physical layer - user access to data throughout the entire computing environment is lost. The importance of locality in resolving network problems should not be underestimated. Preserving, protecting and restoring network operations at the level at which they occur is both more efficient and more cost-effective. So it is incumbent upon the network technology provider to deliver products that (1) guarantee high levels of fault tolerance, 24/7 availability, fast recovery and restoration; (2) are simple to provision, configure and manage; and (3) are affordable in terms of raw dollar cost and human resources required to troubleshoot and support them. The Prohibitive Cost of Downtime Downtime is simply too expensive. A recent Infonetics Research study of medium to large businesses found that, Network downtime (outages and service degradations) costs the average respondent company an estimated $32.7 million per year in lost productivity of networked employees and lost revenue from networked employees who directly generated revenue. [Source: Infonetics Research, The Cost of Downtime in the US 2000 ] According to industry analysts, the hourly cost of network downtime ranges from $74,000 per hour for the average business [Source: IDG] to $2.5 million per hour for credit card authorizations [Source: Contingency Planning Research] to an astonishing $6.5 million per hour for brokerage operations [Source: Gartner Group]. In fact, the cost of downtime can be even greater, as many companies who suffer catastrophic downtime are never able to recover and go out of business. With the costs so high, it is easy to understand the importance that customers are placing on high availability and fast restoration and recovery in their networked environments. A p p r o a c h e s t o N e t w o r k R e s t o r a t i o n Over the years, both enterprise data networks and voice-optimized carrier networks have evolved in their levels of fault tolerance driven by customer demand for increasingly greater data availability and uptime. According to surveys conducted by Telechoice/ Interactive Week over a five-year period, reliability of network service has consistently ranked as the most important issue from a customer perspective. [Source: Riverstone Networks, Technology White Paper #132]. Carriers have long required five 9 s (99.999%) availability because of the particular demands of voice traffic. That level of uptime is now needed for our data networks as well. Local area networks (LANs), which form the basis of the data infrastructure, typically experience only 99.96% uptime at best that means 3.5 hours of downtime per year. Contrast that with the 5 minutes per year of downtime in a five 9s environment. The metropolitan network is where these incompatible uptime standards collide and data and voice networks converge. SONETbased carrier implementations are known for their fast failover and high-uptime characteristics, while optical Ethernet providers coming into the metropolitan network have had to rely on slow LAN-based recovery mechanisms. Let s examine the characteristics of these incompatible traditional uptime approaches, as well as emerging standards and the 3
new technologies available to bring carrier-class line protection and restoration (LPR) technologies to the metro network. SONET Since ANSI ratified the original standards in 1988, SONET has been the leading technology for transmitting voice and data over fiber-optic networks. With restoration times in the event of a path failure of 50 milliseconds (msecs) or less, it has set the standard by which all other protection and restoration technologies are measured. Although linear configurations are supported, SONET is most often implemented in one of two ring configurations: UPSR (Unidirectional Path-Switched Ring) and BLSR (Bidirectional Line- Switched Ring). But just choosing a ring configuration doesn t ensure high levels of availability. Carrier-class availability levels of 99.999% are provided through SONET services like Automatic Protection Switching (APS) which offers 3 types of data protection over fiber: 1+1 (one to one), 1:1 (one for one) and 1:n (one to many). SONET, as a fiber-based technology, has a number of advantages also shared by optical Ethernet: low BERs (bit error rates), a wide range of available bandwidth options (see chart), APS protection, fast (50 msec) failover and restoration, sophisticated failure mode detection and management tools and the ability to support both voice and data over a single fiber. For all its advantages, SONET, unlike optical Ethernet, is not perfectly suited to the metro network. SONET was not designed to deal with high-speed data traffic. Incremental bandwidth allocation is not available (e.g., 1 Mb increments). But even with SONET s weaknesses, it constitutes a daunting challenge for metropolitan network providers to achieve the same level of fault tolerance and meet the uptime requirements of carrier-class network service that SONET provides. And optical Ethernet providers have had to do so using tools originally designed nearly two decades ago for data networks when 24/7 network availability was not an issue for most businesses. Spanning Tree The original Spanning Tree standard (IEEE 802.1D) was developed to solve a LAN problem: the looping and broadcast flooding that occurred in redundant bridged (up to 7 hops) network configurations. For many years, Spanning Tree Protocol (STP) has been employed by Layer 2 switch vendors to avoid loop backs in fault-tolerant configurations, where multiple paths are configured through an Ethernet network, with more than one link at every point. At the time that it was developed, Spanning Tree accomplished what it was meant to do. And, what now seems excessively slow, recovery and re-convergence times of 30 to 60 seconds were quite acceptable at a time when networks were much slower and usage was only a fraction of what it is today. But the limited scalability of STP, coupled with its unacceptably slow failover, its inability to handle VLANs efficiently, and its inefficient utilization of bandwidth in optical ring configurations, all combined to make STP non-viable in the gigabit networks of the metro area. Emerging failover and restoration solutions Just as Ethernet evolved enormously over the last decade and is now a faster, more robust and efficient data transport technology, new failover and restoration mechanisms are emerging to replace STP. These new data-optimized protocols are designed to rival SONET levels of robustness. The IEEE has been working to define Ethernet standards that are the equal to today s SONET-based solutions. And there are even some new optical Ethernet products that far exceed the high availability and self-recovery features of SONET-based solutions. Two Layer 2 technologies, Rapid Spanning Tree Protocol (RSTP) and Resilient Packet Ring (RPR), both seek to overcome the limitations of STP in metro Ethernet failover and recovery. Rapid Spanning Tree Protocol (RSTP) RSTP (IEEE 802.1w), ratified in 2001, builds upon the original 802.1D standard and addresses many of STP s limitations, while still remaining compatible with legacy STP equipment. RSTP offers much faster recovery times (generally one to three seconds) than its predecessor, but still falls short of SONET restoration times of 50 msec or less. RSTP by itself does not fully overcome the shortcomings of STP. To achieve fast restoration times in the 10-msec range, RSTP must 4
be used in conjunction with the IEEE 802.3ad Link Aggregation Protocol. Even so, most implementers remain convinced that RSTP recovery is rarely much below the one-second mark far slower than the carrier standard. [Source: Terri Gimpelson, Metro Vendors Question Spanning Tree Standard, NetworkWorld Fusion, 8/06/01] Another weakness of RSTP is that it was designed for mesh, not ring topologies. Because it is basically just an evolved STP implementation, RSTP continues to have limitations in scalability based on STP s original design support having been limited to networks of small diameter. As attractive as RPR seems, particularly when compared to RSTP, it also has it weaknesses. Until the standard is finalized (and for some period afterward), implementations of RPR will remain vendor-proprietary. RPR also requires a new MAC addressing scheme. And since it is implemented at the switch layer, RPR still doesn t address the large number of network failures, which occur at the physical layer. T H E R E H A S T O B E A B E T T E R WA Y Resilient Packet Ring (RPR) Although RPR (IEEE 802.17) is still in the draft stage, it is scheduled for full ratification as an industry standard in 2003. RPR defines a SONET-like alternative for metro data networks, with support for dual counter-rotating ring configurations optimized for packet-based data traffic. With features similar to SONET Automatic Protection Switching (APS), RPR meets both the requirements of robust fault tolerance and high-speed (sub 50-millisecond) failover. The goals of the 802.17 RPR working group also include full compatibility with existing standards (802.1D, 802.1Q and 802.1f), as well as the adoption of existing physical layer medium. The ability to operate over existing SONET equipment and support for packet broadcasting and multicasting are also planned. As we mentioned earlier, nearly half of all network outages are caused by problems at the physical layer. And yet, the solutions that we have considered up to this point provide restoration and recovery services at higher layers in the network. Physical layer issues like loss of link can be simply resolved with physical layer switching. No complex algorithms or overhead-intensive Layer 2/3 protocols are needed, in fact they may exacerbate the problem. A far better solution to physical-layer induced outages in metro networks is to install optical Ethernet devices equipped with autoconfigurable LPR (Line Protection and Restoration) and SONAR (Switch on No Activity Received). This not only ensures locality in the restoration process, which dramatically accelerates restoration times, but also reduces the cost and complexity of equipment and Fiber Switch Switch A Layer 2/3 Switch Copper or Fiber Switch A Metrobility's chassis-based redundant interface line cards SONET 51.84Mbps 155.52Mbps 622.08Mbps 1.244Gbps 10Gbps 13.27Gbps 40Gbps Switch B Working/Primary Protection/Secondary Ethernet 10Mbps 100Mbps 1Gbps Unused Path Switch B Switch C Switch C In this scenario, Switch B is communicating with the fiber switch. APS uses the protection fiber only as a backup, so the protection fiber may be idle or it might even be used to carry less important data. If the working fiber fails, data is switched over to the protection fiber, which has become the working fiber. Failover time for SONET APS is 50 milliseconds. If both paths fail, the network is down. This Gigabit Ethernet configuration is designed to support mission-critical broadband Ethernet applications in a metro network with 24/7 availability and fast recovery and restoration requirements. When the primary fiber link breaks, traffic automatically reroutes to the secondary link. No STP software rerouting is necessary and failover takes less than 200 microseconds. If both paths fail, bridging protocol switches traffic to the unused path. 5
maintenance, when compared with SONET, spanning tree and resilient packet ring solutions. Auto-configurable LPR is available today in products like Metrobility s patented redundant twister. This is equivalent to dual homing in ring architectures and provides a secondary path for data should the primary path fail. Operating in Dynamic Recovery Mode, restoration occurs in under 200 microseconds. Similar to SONET Automatic Protection Switching (APS), LPR is ultra-fast in failover and restoration (200 µsec vs SONET's 50 msec) and can be implemented at a far lower cost. LPR dynamic recovery is essentially a Layer 1 implementation of APS 1:1 mode, where there is a one-to-one relationship between the primary (or working) line and the secondary (protection) line. In both APS and LPR, the protection fiber is in standby mode as a backup. The only difference is in the timing of the failover. In SONET APS 1:1 mode, there is a 20msec delay before data is switched from the working fiber over to the protection fiber after a failure has occurred. With LPR in dynamic recovery mode, the failover is virtually instantaneous at 200 µsec. Metrobility s SONAR is a feature that actually goes beyond SONET failure detection modes - Loss of Signal (LOS), Loss of Pointer (LOP), and Loss of Frame (LOF) - to provide Loss of Activity (LOA) failure detection. SONAR not only detects loss of link on the active (or primary) port, but it also monitors for traffic for the active line. If the active port remains idle with no traffic detected during the user-defined LOA level [0 to 31 seconds in 1 second increments], the secondary port will be checked. The data is then automatically switched over, even though no specific line failure on the primary has been detected. SONAR proactively preserves and protects the network against a switch or switch port failure creating an absence of traffic, which could bring the entire network down. With Metrobility s LPR and SONAR features, networks enjoy the benefits of traffic engineering services, path diversity and fast reroute / recovery (sub 200 microsecond) capabilities, as well as a higher level of sensitivity in link viability. A new level of network self-sufficiency is achieved with intelligent, automated Layer 1 recovery mechanisms. THE METROBILITY EDGE Metrobility s innovative line protection and restoration, LPR and SONAR, technologies leverage the company s years of mission-critical networking experience to deliver Ethernet-based metro networking solutions, with carrier-class availability and path restoration times far faster than SONET. High-Availability Highlights Quality-of-equipment monitoring allows proactive remote monitoring of power levels and temperature for Metrobility network components. Quality-of-line monitoring enables real-time analysis and non-disruptive adjustments to the network to accommodate the traffic. Hot-swap, load-sharing power supplies and hot-swap line cards Auto-failover to restore broken links User-configurable LPR technology and Dynamic Recovery Mode enable line protection with fast failover and restoration (200 microseconds at 100Mbps and 1Gbps) SONAR monitors for both loss of link and loss of traffic to protect against a switch failure from bringing down all or part of the network. Why Metrobility Solutions? Simple to install configure and manage Automatic layer 1 protection and restoration Self-healing, self-recovering network components Increases network uptime with proactive elimination of outage causes Faster-than-SONET failover at a fraction of the cost. Proactive management with NetBeacon element management software NEBS Level 3 certified chassis and power systems Used by stock exchanges, brokerage houses, 911 systems, hospitals and banks, Radiance products were designed to operate in mission-critical environments. Metrobility is the pioneer in providing affordable fault tolerance and high availability to metropolitan networks. 6
N o r m a l W h a t s T h a t? Once upon a time, seemingly unlimited IT budgets were normal. Then, things changed dramatically to give us today s new normal IT budgets are severely constrained both by decreased revenues (which necessitate lowered IT expenditures) and by fear of the unknown (what s in store next for the economy?). And finally, service providers and their customers in the metro area must deal with increasingly complex networks, provide increasingly robust network service and meet ever more stringent uptime requirements with fewer trained personnel. The Next Normal -- As we See It Carrier-class optical Ethernet is critical to the success of metro networks. Cost-effective, high availability networking is the future. No longer do only a few companies and isolated applications rate full fault-tolerant (FFT) configurations. Now, most companies and soon, all companies will need at least five 9s of network availability. But with companies more cost- and value-conscious, high availability solutions must be affordable. Intelligent physical layer solutions can reduce complexity and cost, while increasing locality of restoration and delivering substantially faster network recovery times. Self-sufficient network devices with enough intelligence to troubleshoot, correct and heal themselves will be developed more extensively as customers demand equipment that can be easily installed, configured and maintained by smaller, overworked IT staffs with more work and less training. Five 9s may be only the beginning in terms of the reliability demands that we are putting on our networking infrastructures. Six sigma and other higher standards for availability are even now being demanded in some critical environments. In order to add value in the metro networking arena, we face an ongoing challenge not only to provide increasingly higher speed links with greater bandwidth, engineer exponentially faster failover/ recovery mechanisms and meet an ever higher availability standard, but also to make our networks more self-sufficient able to virtually install, troubleshoot and heal themselves. New technologies in the form of new standards, more advanced hardware components and more sophisticated algorithms and intelligent firmware will continue to appear. And we at Metrobility are committed to an architecture that allows us to interoperate seamlessly with emerging Layer 2/3 standards, while we strive to advance the art in building Layer 1 solutions that offer the features and benefits of the more complex and costly higher-layer devices with which we interoperate. For additional information on line protection and restoration for high-speed Ethernet networks, please contact: Metrobility Optical Systems 25 Manchester Street Merrimack, NH 03054 1.603.880.1833 1.877.526.2278 www.metrobility.com Copyright 2003 Metrobility Optical Systems. All rights reserved 7