A P P L I C A T I O N

Transcription

1 A P P L I C A T I O N N O T E Faster Resolution of IP/MPLS Service Provisioning Fall-Outs and Customer-Reported Problems Integrated Service, MPLS Infrastructure and Control Plane Troubleshooting for Network Operators

2 Executive Summary The IP and MPLS control planes provide the critical topology knowledge for simplifying endto-end service provisioning, and also automatically re-route around network failures. Yet when provisioning or re-routing processes fail, network operators must be able to understand why. In most of today s operations, control plane troubleshooting requires expert IP routing knowledge to extract a current view of the network topology from the nodes using command-line interface (CLI) commands. Beyond the fact that such expertise is a scarce commodity, manual troubleshooting prolongs the mean time to repair (MTTR), raising the prospect of servicelevel agreement (SLA) violations. It would be expensive and impractical for service providers to retrain all their network operators to become IP routing experts in order to scale their IP/MPLS operations. The ideal alternative would be to equip IP/MPLS network operation teams with management tools that simplify the troubleshooting of service provisioning and customer-reported problems. The Alcatel-Lucent 5620 Service Aware Manager (SAM) makes it possible to efficiently provision and test Layer 2 and 3 services; the Alcatel-Lucent 5650 Control Plane Assurance Manager (CPAM) enables real-time control plane visualization, control plane configuration validation and proactive surveillance. Together, this pair of management tools helps network operators accelerate service-problem diagnosis by collapsing the affected service, MPLS infrastructure and control plane topology into one simple view that reveals, intuitively, the relationship between the layers.

3 Table of Contents 1 Introduction 2 Alcatel-Lucent 5620 SAM and 5650 CPAM Meet the Challenge 3 Troubleshooting Workflows 4 Troubleshooting Services or MPLS Infrastructure Fall-Outs 4 Troubleshooting LSP Path and Service Tunnel Provisioning Fall-outs 5 Troubleshooting MPLS Protocol (LDP, RSVP) Peering Fall-Outs 6 Verifying the MPLS Infrastructure 7 Troubleshooting Control Plane Impact on Existing Services 8 Proactive Service Monitoring 9 The Alcatel-Lucent 5650 CPAM Solution Architecture 10 Conclusions

4

5 Introduction In a typical service provider s commissioning process, field installers set up the physical network infrastructure and IP routing experts implement the IP routing configuration. Network operators can then start commissioning the MPLS infrastructure and begin provisioning services. The IP control plane facilitates MPLS tunnel commissioning and service provisioning. Each individual node understands the overall network topology and has knowledge of the forwarding paths to reach target destinations. During service provisioning, this standard approach allows operators to select service end-points and service profile templates before letting routers automatically instantiate the service across all intermediate nodes in the service path. It is certainly a great improvement over non-control plane technologies in which nodes have no knowledge of their neighbors. Provisioning an end-to-end service in a non-control plane environment requires each node in the path to be provisioned independently based on topological knowledge stored in static inventory databases that are often out of date. In most cases, MPLS service tunnels and services can be turned up in minutes. There are, however, occasional provisioning failures caused by control plane misconfigurations, the absence of existing paths between sources and destinations, or unavailable bandwidth across such paths especially if routing is constrained to specific traffic-engineering values. In such cases, those responsible for provisioning the MPLS infrastructure and services cannot see the IP routing topology. When MPLS tunnels or services fail to become operational, or become operational using the wrong path, network operators are left with two choices [Figure 1]: Isolate and diagnose the control plane issues manually by retrieving routing tables and parameters from the nodes, correlating them into an end-to-end topology, and computing each next hop in the path just as the router would do; or Rely on highly paid, typically scarce IP experts to track down the problem basically handing off the convoluted manual process and prolonging MTTR. The same challenges exist when customers report problems on previously operational services. Whether problems are caused by the control plane s inability to restore the service after rerouting, by traffic-dropping due to congestion of remaining paths, or by any other factors, network operators must rapidly repair service faults reported by customers. Yet isolating and resolving problems is not easy if they turn out to be caused by the control plane. Manual diagnosis makes it difficult for operators to troubleshoot without extensive routing knowledge and CLI administration. Building that knowledge internally by providing intensive IP routing training to their entire workforce is hardly a cost-effective proposition for service providers. To scale IP/MPLS operations economically and reduce MTTR, service providers require management tools that simply and intuitively present the routing plane s relationship to both the MPLS infrastructure and layer 2 and 3 services within a simple, intuitive view that facilitates troubleshooting without requiring a certified IP routing expert. Alcatel-Lucent Faster Resolution of IP/MPLS Service Provisioning Fall-Outs and Customer-Reported Problems 1

6 Figure 1. Lack of IP Topology Visibility Service Troubleshooting Everything looks good from here. Why is delay exceeded? Network Operator No Visibility of Underlying IP Topology Only available via CLI Failed link caused a re-route over wrong path due to misconfiguration Alcatel-Lucent 5620 SAM and 5650 CPAM Meet the Challenge The Alcatel-Lucent 5650 Control Plane Assurance Manager (CPAM) offers graphical IP routing topology visualization; superimposed MPLS protocol peering visualization; control plane, IP route and MPLS route-change monitoring; and a powerful path computation query tool based on specific parameters. Seamlessly integrating the MPLS infrastructure and layer 2 and 3 services management offered by the 5620 Service Aware Manager (SAM), the 5650 CPAM allows network operators to highlight label-switched paths (LSPs), service tunnels and services in the IP routing topology view. This new management model introduced by Alcatel- Lucent collapses multiple layers onto a single view, equipping network operations personnel with the necessary tools to operate an IP/MPLS network and reduce troubleshooting delays. 2 Alcatel-Lucent Faster Resolution of IP/MPLS Service Provisioning Fall-Outs and Customer-Reported Problems

7 Troubleshooting Workflows Different troubleshooting workflows are followed depending on the problem being investigated. The same tools can be used, but the starting point is variable. The following table illustrates three possibilities. T R O U B L E S H O OT I N G L S P S O R S E R V I C E T U N N E L S T H AT A R E D O W N Checkpoint the IP topology map to establish a baseline from which any IP topology changes will be flagged. Verify that an IP path exists between source/destination addresses. Use the 5650 CPAM IP path computation (shortest path first: SPF). The complete or partial path will be highlighted. If the SPF path cannot reach its destination, the problem is an unreachable destination. Use the Next Hop feature to identify other routers that might not be able to reach the same destination. If the IP route SPF succeeded, the problem is not at the IP layer. At the MPLS layer, use SPF for GRE/LDP tunnels and unconstrained RSVP LSPs. For CSPF LSPs, results will show the shortest path that meets constraints such as specific bandwidth, admin group, etc. The MPLS view can be used to identify RSVP/LDP misconfigurations, and to highlight IP links that meet or do not meet RSVP constraints. Use RSVP/LDP SPF computation with the required class of service (CoS) and bandwidth requirements. If there is no path found, lower the bandwidth requirement to see if bandwidth is the issue. Change the CoS also to see if a path can be found. T R O U B L E S H O OT I N G A FA I L E D AT T E M P T AT P R O V I S I O N I N G A S E R V I C E If a selected service tunnel is not able to make a service operational, the wrong tunnel might have been selected, the selected tunnel might not have enough remaining bandwidth, or the tunnel may be down. Highlight the target service tunnel on the IP topology map to ensure it reaches the appropriate destination. Use the filtered search to find out if there is enough bandwidth for a specific priority on the service tunnel. If there is not enough, use RSVP SPF to find the path the control plane would use to route the service tunnel. A user can change the target service tunnel to one that corresponds to the highlighted path, or can create a new one if none exists. If no SPF path can be highlighted, the destination may be isolated (i.e., have no IP connectivity). Try the IP route SPF to verify. T R O U B L E S H O OT I N G E X I S T I N G S E R V I C E S W I T H O P E R AT I O N A L S TAT U S U P Consider a situation in which a service problem is reported by a customer or a Threshold Crossing Alarm is issued by the 5620 SAM Service Test Manager through its SLA monitoring. Extract the LSP path used by the service from the 5620 SAM. Navigate from the LSP to the 5650 CPAM LSP monitor, which records all route changes as they occur. From the timestamps, locate whether a route change precedes the reported SLA test failure. Use the 5620 SAM Service Test Manager to issue an OAM service tunnel trace. Highlight the OAM service tunnel trace with a previous OAM trace result on the IP topology map. Alcatel-Lucent Faster Resolution of IP/MPLS Service Provisioning Fall-Outs and Customer-Reported Problems 3

8 If the OAM service tunnel traces overlap, the service tunnel did not re-route. Set a checkpoint on the IP topology map to ensure the route is not flapping. If not, the problem is not caused by the control plane. If OAM service tunnel traces diverge, there has been a recent service tunnel change which might be the cause of the problem. Verify that the forward/reverse paths match. Verify that the number of hops is not dramatically different. If so, use the 5620 SAM OAM tests to verify packet drop rate, jitter and delay. Troubleshooting Services or MPLS Infrastructure Fall-Outs The 5650 CPAM provides a rich suite of functionality that can help reconcile serviceprovisioning failures in an IP/MPLS environment. To effectively troubleshoot IP/MPLS services, it is necessary to drill down from one layer to the next and understand their impact on each other. Services depend on having operational tunnels between sources and destinations, with sufficient bandwidth to deliver the appropriate quality of service. Service tunnels depend on having operational LSPs established by the MPLS protocols. The MPLS protocols (LDP, RSVP) depend in turn on appropriate routing information from the IP protocols (ISIS, OSPF, ISIS-TE, OSPF-TE, BGP). The following sections describe the tools used to troubleshoot each layer. Troubleshooting LSP Path and Service Tunnel Provisioning Fall-outs Fall-out when creating a service tunnel can be caused by inadequate remaining bandwidth in the targeted LSP path. A simple SPF query specifying the service tunnel s CoS and bandwidth requirements might reveal that the shortest path currently corresponding to the specified requirements is different than the service tunnel selected in the service-provisioning form [Figure 2]. It is then important to determine if the selected path truly should not have been used, or whether a control plane configuration mistake on the selected path has inadvertently introduced a divergence from planned traffic engineering. Figure 2. Divergence Between Selected and Possible Service Tunnel Paths 4 Alcatel-Lucent Faster Resolution of IP/MPLS Service Provisioning Fall-Outs and Customer-Reported Problems

9 To identify possible misconfigurations, the 5650 CPAM offers a powerful control plane parameter search tool with configurable filters. The filters can be set to highlight links that differ from the planned control plane configurations [Figure 3]. These filters could be retrieved from a filter list made available by routing architects to help network operators isolate their provisioning problems. Figure 3. Highlighting IGP Links Deviating from Standards Configuration Troubleshooting MPLS Protocol (LDP, RSVP) Peering Fall-Outs MPLS protocols rely on IP to identify the network topology. They use that knowledge to provide next-hop information when users provision LSP paths across the network. Each link to be used for MPLS forwarding must have both interior gateway protocol (IGP) and MPLS protocols enabled in each direction (called peering). The 5650 CPAM offers an MPLS topology view that superimposes IGP and MPLS links in one view [Figure 4]. Parallel IGP, MPLS RSVP and LDP links are displayed to discover misconfigurations such as unidirectional links or IGP linkdown statuses that affect MPLS protocols. As mentioned above, the 5650 CPAM can also highlight IP links that meet or do not meet RSVP constraints for bandwidth, admin groups, and the like. Figure 4. Superimposed IGP and MPLS Topologies Show Missing MPLS RSVP (purple) and LDP (yellow) Peering Alcatel-Lucent Faster Resolution of IP/MPLS Service Provisioning Fall-Outs and Customer-Reported Problems 5

10 Troubleshooting IP Routing-related Fall-Outs Service and MPLS layers depend on an operationally sound IP routing layer. To troubleshoot this layer, the 5650 CPAM offers a number of features that isolate IP connectivity problems. The 5650 CPAM enables checkpointing of IGP databases, making it easy to detect whether changes such as added or deleted links impair service or service-tunnel provisioning. Using the service overlay on the IP routing topology view could reveal that a failed service site is isolated from a routing perspective. Network operators can also highlight the current shortest path first (SPF) for a given source/ destination combination [Figure 5]. Should an end-to-end path not be found, the partial IP path is displayed allowing the operator to find where routing failed. For more information on IP routing diagnosis, please read the Alcatel-Lucent application note, Troubleshooting the IP Routing Layer with the 5650 Control Plane Assurance Manager. Figure 5. SPF Computation Demonstrating End-to-End IP Route, Including Equal Cost Multi Path (ECMP) Verifying the MPLS Infrastructure Once an LSP, service tunnel or service has been successfully provisioned, the 5650 CPAM service overlay function can be used to verify the path used across the network. Figure 6. VPLS Service Overlay Showing Different Forward/Reverse Paths Using MPLS or service overlays in the 5650 CPAM IP topology view can quickly reveal if a newly created service tunnels follow a consistent forward/reverse path through the network [Figure 6]. 6 Alcatel-Lucent Faster Resolution of IP/MPLS Service Provisioning Fall-Outs and Customer-Reported Problems

11 This step is especially important when verifying MPLS infrastructure provisioning involving multiple alternate paths as a way of ensuring that each possible path follows planned routes [Figure 7]. Inconsistencies are obvious in the 5650 CPAM IP topology view; a network operator can rapidly identify the source of an error by focusing on the point of failure. Figure 7. Highlighting Primary and Alternate MPLS Paths to Verify Routes Troubleshooting Control Plane Impact on Existing Services The relationship between the dynamic IP/MPLS infrastructure and layer 2 and 3 provisioned services is made visible by laying 5620 SAM objects over the 5650 CPAM IP topology. A key value of the combined 5650 CPAM and 5620 SAM for operators is that navigation is seamless between the two. Consider the example [illustrated in Figure 10] of a three-site VPRN service with a hub-and-spoke topology displayed by the 5620 SAM. Highlighting this service over the checkpointed 5650 CPAM IP topology view shows that an OSPF link failure (in red) has forced all three service tunnels to take an alternate path (orange). The figure further illustrates that one service tunnel has its reverse/forward paths on separate links. This multi-layer, superimposed visualization helps network operators instantaneously understand: that a failure caused the control plane to start a re-route process; that underlying service tunnels were established over new paths; and what the new service topology is. To proactively identify whether a change in the control plane might have affected services, the 5620 SAM s Service Test Manager can be used to issue regular OAM service tunnel tests that will report the pass/fail status of various SLA metrics such as packet loss, delay and jitter. Once a failed test is reported, the 5650 CPAM IP path and LSP path monitoring feature can be used to see whether a routing change occurred in the same timeframe. Both OAM traces and historical path records can be highlighted on the 5650 CPAM IP topology view to visualize previous and current paths. Alcatel-Lucent Faster Resolution of IP/MPLS Service Provisioning Fall-Outs and Customer-Reported Problems 7

12 Figure 8. Cause and Effect Between Control Plane Changes and MPLS Service Tunnel Route Proactive Service Monitoring The 5650 CPAM supports proactive assurance of IP/MPLS networks and services through a number of surveillance features integrated tightly into the 5620 SAM: IP routing topology monitoring of added/deleted IP links from a user-selected baseline; threshold-crossing alerts (TCAs) that report abnormal control plane behaviour; and IP/MPLS path monitoring and path-change recording. The seamless integration of the 5620 SAM s proactive SLA monitoring feature and 5650 CPAM control plane surveillance can provide advance detection of servicequality or control plane health variations, allow network operators to understand the causes and effects of control plane changes and appreciate their impact on services. For more information on proactive control plane and service monitoring, please refer to the Alcatel-Lucent application note titled Proactive Control Plane Management to Offer Consistent Service Delivery and Adhere to SLAs. 8 Alcatel-Lucent Faster Resolution of IP/MPLS Service Provisioning Fall-Outs and Customer-Reported Problems

13 OSS, Traffic Engineering, 5750 SS-C (Service Admission Control) SAM-O OSS Open Interface CPAM-O Extensions 5620 SAM Server 5650 CPAM Server Stats Mgmt Fault Mgmt Common Frameworks (Models/GUI/etc.) Prov Mgmt OAM Mgmt/Info. Alarms and Status Service Adjacency LSP/Tunnel Adjacency East/West API TCA and RCA CPAA Mgmt IGP and LDP/RSVP Topology SPF Paths Path Watch Map and Overlay Protocol Validate Mediation Interfaces 7701 CPAA 7701 CPAA VLL Area 1 Area 2 VLL Area 3 The Alcatel-Lucent 5650 CPAM Solution Architecture The Alcatel-Lucent 7701 Control Plane Assurance Appliance (CPAA) provides realtime knowledge of control plane topology and the operational status of all routers in the network including third-party routers. It processes user-generated requests for real-time path computation spanning multiple areas. All 7701 CPAAs in a network are centrally configured and managed by the 5620 SAM. The 7701 CPAA is a 4U rack-mountable appliance with a VxWorks operating system that non-intrusively 1 participates in control plane signaling, executes distributed requests originating from the 5650 CPAM, and performs element-management functions originating from the 5620 SAM. Control plane protocols are based on Alcatel-Lucent s tested and proven service router operating system SR OS to guarantee a high degree of interoperability in IP customer networks. The 5650 CPAM server provides the necessary framework to coordinate data collection by multiple 7701 CPAA sources before merging it into a usable format for resident applications. It also coordinates user requests for multi-area path computations across 7701 CPAAs before merging the results for analysis, display or export. The 5650 CPAM is integrated seamlessly with the 5620 SAM to provide in-context navigation from provisioned services, tunnels, label-switched paths (LSPs) and historical operation and maintenance (OAM) traces. It reuses the 5620 SAM physical infrastructure and user interface menus, allowing service providers to leverage their 5620 SAM server platform, database and training investments. The well-established 5620 SAM OSS interface (SAM-O) is extended to provide a single, common northbound interface for both products. This reduces the complexity associated with introducing a new application into the network environment. Traffic engineering tools and session admission control applications can also leverage the 5650 CPAM OSS interface to have a real-time source for their control plane topology and computation needs. 1. SR OS has been modified to issue only one LSA (link-state advertisement) to announce one active link to the area. The 7701 CPAA is not seen as a transit node since it only has one active link to any given routing area.

14 Conclusions Service providers must acquire simple management tools that enable operators to manage IP/MPLS-based networks effectively, without requiring extensive IP routing training. The 5650 CPAM delivers instant control plane change visualization, real-time path computation, layer 2 and 3 service and MPLS infrastructure overlays on the IP topology view, configuration deviation searches and many more features that can be used and re-used in various combinations to support the appropriate troubleshooting workflow. The 5650 CPAM s seamless integration with the 5620 SAM enable troubleshooting of each layer individually and in relation to each other. This powerful yet simple multi-layer view pre-correlates all the necessary information to help reduce troubleshooting time for serviceprovisioning fall-outs and customer-reported service problems. It represents the simplified IP/MPLS management solution service providers need to empower their network operators and scale their IP/MPLS operations. 10 Alcatel-Lucent Faster Resolution of IP/MPLS Service Provisioning Fall-Outs and Customer-Reported Problems

15

16 Alcatel, Lucent, Alcatel-Lucent and Alcatel-Lucent logo are trademarks of Alcatel-Lucent. All other trademarks are the property of their respective owners. The information presented is subject to change without notice. Alcatel-Lucent assumes no responsibility for inaccuracies contained herein Alcatel-Lucent. All rights reserved (06)