MPLS WAN Explorer Enterprise Network Management Visibility through the MPLS VPN Cloud
Executive Summary Increasing numbers of enterprises are outsourcing their backbone WAN routing to MPLS VPN service providers. However, MPLS VPN WAN services come with some serious network management liabilities that can be quite costly. Once MPLS VPNs are deployed, IT loses end-to-end routing and traffic visibility across the WAN backbone. This loss of visibility makes it more difficult to keep service providers accountable for service quality, causing costly finger-pointing when problems occur. More importantly, the lack of end-toend routing and traffic visibility greatly impairs key network operations and engineering processes, which increases the cost of managing the network while causing application delivery to suffer. Packet Design s MPLS WAN Explorer restores much-needed network-wide routing visibility to enterprises that utilize MPLS VPN services for their WAN backbone. MPLS WAN Explorer extends Packet Design s industry-leading route analytics technology, which leverages the network s live routing protocols as a source of network management information. With MPLS WAN Explorer, enterprises can now see beyond the traditional borders of their internal networks and understand their end-to-end network, even across MPLS VPNs. MPLS WAN Explorer greatly improves network monitoring and troubleshooting processes with network-wide routing visualization, Layer 3 network reachability monitoring and alerting, re-windable troubleshooting history, end-to-end path tracing, and detailed analysis tools. This paper reviews how Layer 3 MPLS VPNs work, and explores the network management challenges introduced by deploying them. The paper then introduces MPLS WAN Explorer, the route analytics technology that powers it, how it works across MPLS VPNs, and illustrates how MPLS WAN Explorer can help enterprises increase the efficiency and accuracy of key network management processes, keep service providers accountable for service quality, and more successfully deliver end-users application traffic. Layer 3 MPLS VPNs A Brief Overview Layer 3 MPLS VPNs are delivered by service provider IP/MPLS networks that are organized into a core of provider or P routers, and a layer of customer-facing provider edge or PE routers. PE routers are configured to handle multiple VPNs through separate virtual routing and forwarding (VRF) tables. Each customer s VPN is handled by dedicated VRFs on various PEs located in different geographies across the service provider s network. Interconnectivity between VRFs is delivered by a mesh of MPLS tunnels, with a special extension of BGP providing control plane mapping of tunnels to VRFs. When using MPLS-based VPN services, enterprise customers are responsible for connections from each site to the service provider network, by connecting their Customer 1
Edge (CE) router to a PE router and enabling routing, typically using the Border Gateway Protocol (BGP). MPLS VPNs are an IP Routing Service the Implications Service provider MPLS VPNs are an IP routing service where the service provider not only takes responsibility for providing a Layer 2 link and getting traffic across it, but also for delivering the enterprises internal IP routing updates properly and privately across its shared VPN network. Because this routing aspect of the MPLS VPN service is delivered via a standards-based MPLS VPN service architecture that blocks enterprises from seeing into the service provider network, IT completely loses end-to-end visibility across the enterprise network. Technically speaking, the lack of end-to-end network visibility means that IT engineers are blind to a key function of IP networks routing reachability. The role of routing protocols in IP networks is to ensure that IP subnets (represented by routed prefixes) attached to routers across the network can communicate with (or reach) each other. With a MPLS VPN obscuring backbone routing, IT engineers can no longer tell if the network is operating correctly at an IP routing level. This poses a fundamental monitoring challenge, since SNMP management systems can show all devices and interfaces being up, while application traffic may be dropped or delayed due to routing-layer issues that are occurring within the service provider network cloud, or at the complex BGP peering interface between PE and CE routers. Without any detailed information on end-to-end routing reachability, troubleshooting the network aspect of an application problem also becomes even more of a challenge than normal, often getting stuck in finger-pointing between IT and the service provider. Finally, it becomes even easier to introduce errors into the network during routine network changes since engineers don t have any detailed insight into the actual state of network operations. The bottom-line impact of the lack of visibility into routing reachability is that key operations and engineering processes such as monitoring, troubleshooting and planning the network to ensure application delivery become much more timeconsuming, and much less accurate. Ultimately, these inefficiencies cause operations costs to rise in the face of ever-increasing demands for networked applications needed to drive business automation. MPLS WAN Explorer Visibility through the MPLS VPN Cloud Packet Design offers a unique solution called MPLS WAN Explorer, which is designed to help enterprises regain end-to-end network routing and traffic visibility across MPLS VPNs. MPLS WAN Explorer utilizes Packet Design s industry-leading route analytics technology that is deployed in hundreds of large enterprise, government and service provider networks worldwide. Route analytics solutions listen passively to routing 2
protocol exchanges on the network and deliver a router s eye view of Layer 3 connectivity and reachability, providing network engineers with previously unavailable intelligence on the end-to-end Layer 3 operation of an IP network. Route analytics works by forming passive (listen-only) peerings with key routers in the network using standards-based routing protocols such as BGP, OSPF, IS-IS and EIGRP, recording every routing protocol update, and creating a model of the network that is as accurate as the routers themselves understand it. In the case of MPLS VPNs, MPLS WAN Explorer extends route analytics by peering via IBGP with the CE routers and receiving all the routing updates that the CE routers exchange with other CE routers via the MPLS VPN PE routers. By combining route analytics' understanding of both BGP and IGP, MPLS WAN Explorer provides visibility into the end-to-routing topology across MPLS VPNs, significantly improving the accuracy and efficiency of key enterprise IT processes. MPLS WAN Explorer provides a variety of monitoring, troubleshooting, and other analysis tools that help network managers make sense of what is happening to their WAN MPLS VPN Reachability Monitoring, Alerting and Visualization One of the key missing ingredients in MPLS VPN SLAs is any provision for guaranteeing IP reachability. MPLS WAN Explorer helps IT ensure that the backbone routing managed by the service providers is working properly by creating and maintaining a moving window baseline of per-vpn and per-site routing reachability. Based on user-defined thresholds, it can monitor and alert on any loss of routing reachability across one or more (redundant) MPLS VPNs. An intuitive network-wide topology view including the VPN cloud provides at a glance detection of sites that have lost reachability or are experiencing other problems such as routing policy violations where sites are connected to a VPN that they aren t supposed to be. Figure 1: MPLS WAN Explorer provides end-to-end WAN topology visualization 3
Easy to use monitoring and analysis reports provide detailed reachability information on a per VPN, site and prefix basis, as seen in Figure 2. Figure 2: The Reachability from Other Sites report shows a list of VPN sites, their announced prefixes and percentage of reachability to those prefixes from other sites Fast Detection of MPLS VPN Routing Outages and Instabilities MPLS WAN Explorer not only monitors and alerts on per site and per VPN prefix reachability issues, but can also monitor and alert on VPN site routing outages and instabilities. Watchlists of paths between data centers and their satellite user sites can be monitored and alerts triggered if any path fails or changes. User-set thresholds to monitor for excessive routing protocol activity (churn) as well as prefix and link flapping can trigger alerts if the network experiences harmful instabilities that can impact application traffic. Often times, when enterprises utilize two MPLS VPN service providers for fault tolerance, IT managers have no idea if the primary has failed and the secondary VPN is active, simply because there has historically been no way to monitor the level of redundancy in the network. MPLS WAN Explorer provides early warning of increased continuity risk in the network by alerting on per-vpn loss of reachability. This early warning system helps network managers quickly alert their service provider of problems so that redundancy can be restored in order to avert a potentially disastrous failure of the network should the secondary VPN experience a problem. Knowledge of these failures also helps network managers keep their service providers accountable and can even aid enterprises during contract renegotiations. Rewindable Troubleshooting History One of the biggest challenges with managing complex, redundant IP networks is understanding precisely what happened in the past, whether five minutes or five days ago. 4
This is no less true of trying to troubleshoot what happened in a MPLS VPN service problem. Fortunately, MPLS WAN Explorer continuously records all routing events and provides a History Navigator that allows engineers to rewind the network back to the point in time when a problem was occurring to understand the network operation at that moment. MPLS WAN Explorer even allows historical analysis on a per-site basis. Figure 3: Engineers can rewind the network for more effective troubleshooting using the History Navigator End-to-End Path Tracing and Detailed Routing Analyses Once engineers have rewound the network to the time that a problem was occurring, they can utilize MPLS WAN Explorer s end-to-end path tracing to localize the portion of the network that carried the application traffic, and thus should be examined during the troubleshooting process. MPLS WAN Explorer s path tracing provides visibility between sites across the VPN, including ingress and egress PE routers, and can even traverse multiple VPNs. 5
Figure 4: MPLS WAN Explorer provides path tracing across MPLS VPNs MPLS WAN Explorer provides a variety of reports to aid troubleshooting analysis. Detailed routing analyses with flexible drill-down views allow engineers to further pinpoint the source of problems within the network. An example troubleshooting scenario is shown in Figures 5-7. In this case, several sites have lost prefix reachability to the Chicago-1 site. 6
Figure 5: A summary reachability report shows reachability problems to Chicago-1 In Figure 6, a drill-down report on site reachability shows that there is variable reachability to the Chicago-1 site. For example, Atlanta-1 has lost all reachability to Chicago-1. Since most other sites have retained most of their reachability to Chicago-1, it's most likely that the source of Atlanta-1 s problems are local to Atlanta-1, perhaps due to a down condition or instability in the EBGP peering between its CE router and the service provider s PE router. Figure 6: Flexible drill-downs such as the site reachability report allow engineers to identify the per-site location of problems in the network A more complex task is to understand what has happened to sites such as Boston-1, which have partially lost reachability. With MPLS WAN Explorer, engineers can utilize further drilldown reports to look at prefix-level reachability and see if individual prefixes can be reached by any other sites or not, as seen in Figure 7. In this case, one Chicago-1 prefix is reachable by 17 sites, but not by Boston-1, which means that the source issue is problem at Boston-1. By contrast, another prefix is not reachable from any sites, meaning the problem is local to Chicago-1. 7
Figure 7: Detailed routing reachability analyses allow engineers to further localize the source of reachability issues on a per-prefix basis. Powerful BGP Troubleshooting Tools for VPN Peering Problems As with all other router to router connections, problems in the BGP peerings between service provider PE routers and per-site CE routers can and sometimes do occur. Since these issues occur at the boundary between two networks, and because BGP is a very difficult routing protocol to understand, enterprise engineers need powerful tools to understand what has happened. MPLS WAN Explorer s rewindable history and topology visualization allows engineers to easily tell if there was a peering issue by visually verifying if any CE site was isolated from the VPN. Per-CE router BGP event history tables also allow engineers to verify if a peering was reset by the CE or the PE router. Monitoring and Alerting on Breaches in MPLS VPN Privacy MPLS WAN Explorer can help network managers ensure the privacy and integrity of an enterprise s MPLS VPN backbone by alerting on significant changes in the number of prefixes in the MPLS WAN VPN. Since an enterprise WAN should be relatively stable in the number of its advertised prefixes, if a large number of prefixes are advertised into the network in an unexpected manner, then it is possible that the service provider has inadvertently mixed customer VPNs. Once an increase of prefixes has been detected, engineers can look at the History Navigator s histogram of levels in advertised prefixes in the network and find the time when the prefixes entered the network by looking for a jump in the prefix graph. MPLS WAN Explorer also provides a list of all known prefixes, which can filtered to show any routes that are advertised but not in the baseline, as seen in Figure 9. 8
Figure 9: Detailed views of new, non-baseline BGP prefixes help detect foreign routes In addition, in cases where the Internet routing table isn t being advertised into the network, engineers can also see whether there are unknown BGP Autonomous Systems associated with routes in the network. When connecting to a provider s layer 3 VPN service using BGP, each of the enterprise s sites must have a unique Autonomous System Number (ASN), typically private ASNs assigned by the service provider. These ASNs in effect represent the list of VPN sites. The service provider s network should never inject routes into the customer s VPN that are from an unknown ASN, as this would indicate that another customer s VPN has inadvertently been connected into the VPN. MPLS WAN Explorer provides a Routing Information Base (RIB) Browser tool that can analyze BGP routing based on a number of attributes including ASN, and thus show if there are any unknown ASNs in the network, as seen in Figure 10. Drill-down analyses to historical event details show when and where the foreign routes were introduced to the VPN. Figure 10: Listing of ASNs and their respective advertised route counts. If an unknown ASN appears in this listing, then network managers know that the privacy and integrity of their VPN service has been compromised. Monitoring of Remote Site IGP Routing Issues Some CE sites have extensive IGP routed networks behind them, perhaps with multiple OSPF/IS-IS areas or EIGRP AS. In these cases, network managers also need to be able to get insight into routing issues within those IGP domains, especially in cases where WAN 9
reachability issues are traced to the IGP domain behind the CE. MPLS WAN Explorer provides extensive OSPF, IS-IS, and EIGRP monitoring, historical analysis and even scenario modeling. For more details on how route analytics can be used for a variety of network management purposes, please visit Packet Design s white paper library at: http://www.packetdesign.com/resources/white-papers Scalable Monitoring of Satellite Sites MPLS WAN Explorer can also monitor thousands of smaller, satellite WAN sites that consist of a CE router with perhaps a single routed prefix and no IGP domain, without requiring a BGP peering to each CE. Monitoring of paths to important satellite sites allows real-time alerting to reachability issues to those sites. In addition, MPLS WAN Explorer provides a real-time updated satellite site reachability report to aid monitoring and analysis. Integrated Routing and Traffic Analysis across MPLS VPNs MPLS WAN Explorer not only provides network-wide, end-to-end understanding of routing and IP reachability dynamics, but when combined with Packet Design s Traffic Explorer, enables integrated routing and traffic monitoring, historical analysis, network modeling and capacity planning across MPLS VPNs. When enabled by MPLS WAN Explorer, Traffic Explorer provides MPLS VPN specific site-to-site traffic reports that work hand in hand with MPLS WAN Explorer s site reachability analysis reports. For more information on Traffic Explorer, please visit Packet Design s website at www.packetdesign.com MPLS WAN Explorer Benefits MPLS WAN Explorer offers enterprise IT managers a number of benefits when deployed to help manage MPLS VPN services and ensure application delivery across the WAN: More responsive monitoring due to real-time alerting on critical network events. Unlike SNMP, routing protocols operate with milli-second response times. Since MPLS WAN Explorer s route analytics sees network events at the same speed as routers, network managers get the benefit of real-time alerting on critical network issues such as CE to PE peering outages or lost redundancy, lost site reachability and suspicious additions of routes to the VPN. Faster troubleshooting and higher network quality. MPLS WAN Explorer increases IT engineers ability to localize the network problem domain and reduce finger pointing. Rather than wasting time wondering who s to blame for a problem or waiting for the provider to respond, network managers can now proactively find the source of issues that impact application delivery. Historical problem analysis prevents past or intermittent problems from continuing to plague application delivery over time. Intelligence to keep service providers accountable. Without MPLS WAN Explorer, enterprises have no visibility to understand whether service providers are providing the level of routing service quality that is needed to support critical networked applications. In the case where a provider has caused a reachability problem, 10
network managers now have a complete forensic history and powerful visualization and reporting tools to aid them in holding their provider accountable for service outages and instabilities. Conclusion MPLS WAN Explorer provides enterprise IT managers with the intelligence needed to ensure that MPLS VPN deployments don t impede key network operations and engineering processes and cause costly application delivery problems. With network managers increasingly being graded on application delivery and cost savings rather than just basic infrastructure availability, MPLS WAN Explorer s Layer 3 visibility is a must-have capability to ensure successful and cost-effective WAN management. To learn more about Packet Design, MPLS WAN Explorer and route analytics, please visit us online at http://www.packetdesign.com, email us at info@packetdesign.com or call us at 408-490- 1000. 11