Network-Wide Change Management Visibility with Route Analytics

Network-Wide Change Management Visibility with Route Analytics

Executive Summary Change management is a hot topic, and rightly so. Studies routinely report that a significant percentage of application and service delivery problems in IP networks are caused by misconfigurations and other human errors -- usually made during routine changes to the network. Yet while a variety of change management solutions are available, they are all concerned with controlling changes from a device-centric point of view whether or not command syntax is being typed in correctly, or whether the authorized version of operating system (OS) or drivers is being applied. While extremely useful, these solutions still lack a major component: the visibility to help IT departments working with large or complex networks to easily understand precisely how the entire network including routing and traffic flows will behave when changes are made. Given that delivering applications with high availability and performance is the network's purpose, the uncertainty involved in such routine network operations creates a significant gap in IT s ability to meet its service level commitments. The root of this uncertainty is primarily technological: The lack of network management tools that can maintain an accurate model of the as-running network s routing and traffic behavior as the basis for network engineering and operations. Without this fundamental knowledge, network managers are frequently operating in the dark, making decisions based on inaccurate or missing information. Route analytics technology bridges this gap in network management intelligence, providing IT with an always-accurate model of the as-running network's routing and traffic. Using this automatically self-maintaining model, network managers can "rewind" recorded network history to examine exactly how routing and traffic behaved at any point in the past. And they can perform highly accurate change modeling to simulate how the network would behave in the event of a future network change planned or unplanned. This white paper reviews the limitations to traditional change management tools and strategies, explains how route analytics works and how it can be used simulate and plan not only simple, routine maintenance change operations, but also complex network changes such as a data center migration in a large, complex enterprise network. IT departments stand to gain vastly increased visibility that increases the accuracy of change operations and reduces errors, resulting in higher and more predictable application availability and performance. Ultimately, they can also realize improved utilization of existing network assets and reduced operational costs. Why Device-Centric Change Management is Insufficient The fundamental limitation of change management tools today is that, while they provide helpful process control for executing device commands correctly and enforcing version controls across multiple devices, they offer no insight as to whether a configuration on a particular device will have its intended effect across the network. In large, complex and redundant networks which usually are controlled by routing protocols such as OSPF, IS-IS, EIGRP and BGP this is critical, because the countless variations in routing behavior make the implications of a routing or traffic change very difficult to understand. As any experienced network engineer will attest, it is hardly uncommon for unforeseen consequences to result from a change. Even changes that are well managed from a device point of view can and will regularly exert an unintended butterfly effect on some other part of the network. 2013 Packet Design, Inc. Page 2 of 18

Artificial Solutions Don t Cut It Of course, this is nothing new. That s why IT and network engineering organizations sometimes shell out the capital and software costs to either build a lab to test network changes before they re rolled out, or to buy network modeling software. While there are merits to both of these measures, both of them work on artificial, offline models which can't provide accurate, networkwide visibility into the impact of planned or unplanned changes. Labs are by definition artificial and can't come near approximating the complexity of the production network, so they are primarily useful for testing device behavior such as interoperability or performance. Network modeling software is a good deal better in that, with a lot of work, an engineer can create a model that looks much like the actual network. But it s still a manually created model that can t tell how routing and traffic would actually behave, and updating it takes so much time and effort that it's impractical for day-to-day and week-to-week change operations. Network modeling solutions can also be extremely expensive. At the end of the day, traditional network modeling solutions are mostly useful for long-range, major planning exercises. Route Analytics Seeing the Network like a Router Now there is an answer to the need for network-wide network change modeling and visibility. Route analytics is a technology that leverages the intelligence in routing protocols to automatically build and maintain an always-accurate routing map of any IP network, across AS, areas, and multiple protocols. Route analytics solutions peer with key routers in each AS or area, passively listening to and recording every routing update communicated throughout the network. Route analytics implements the same algorithms that run on routers to calculate and deliver a routing topology map as accurate as the network s actual routers understand it. Based on this routing map, route analytics then intelligently integrates NetFlow data collected from a small subset of network interfaces to create an integrated, dynamically updated routing and traffic map. This integrated view is distinct from traditional methods of NetFlow analysis, which do no more than present link-by-link flow data in separate tabular reports. Route analytics collects NetFlow records from routers handling the majority of traffic flows into a network, such as at data centers, Internet peerings and key WAN links. It then utilizes its always accurate knowledge of routing paths to map each traffic flow across the precise links it traverses in the actual network. The result is a network-wide map of all links with accurate real-time and historical traffic utilization and flow details. This map can be rewound to past moments in time to perform forensic analysis on the exact state of traffic and routing at that time; it can also be used as the basis for simulating routing and traffic changes, showing exactly how the whole network would behave in response. Route analytics delivers this comprehensive intelligence with a minimum of overhead, since it collects NetFlow records from only a few exporting routers. Route Analytics in a Simple Change Management Scenario: Adding a New IGP Route or Routes Route analytics empowers IT with visibility to make even mundane network changes and updates far more accurate. In this example, route analytics is applied to enhance best practices for the relatively simple operation of adding new routes to a router. When adding new IGP routes, the following best practice steps are recommended: 2013 Packet Design, Inc. Page 3 of 18

Pre-Implementation Work 1. Assign the prefix to the project 2. Determine the date for application to the network 3. Verify that the prefix does not exist in the routed infrastructure (unless the new route is a backup route) 4. If it is a backup route, then confirm if it is a primary, secondary, or equal cost path 5. Assign the correct metric to route if different from default 6. Determine if the summary block assigned is summarized for the area or region 7. Determine if the summary block needs a summary statement and where it is needed 8. Note which routers will be affected by the route addition 9. Confirm that no other maintenance is needed on this router in conjunction with this addition (e.g., upgrade to new version of code, addition of hardware for this interface or other future additions) Installation Plan Definition 1. Verify that the correct router is being changed 2. Re-check that the prefix has not been used in the routed infrastructure 3. Add the prefix as primary or secondary to interface or add static 4. Add the network statement to the IGP process 5. Confirm that the prefix is active and in the correct format (i.e., prefix and mask are correct) Back-Out Plan Definition 1. If the prefix exists on another router, either attempt to get another prefix assigned and replace the current prefix with the new prefix, or abort the maintenance until a suitable prefix can be assigned. 2. If the prefix will not come up in IGP, then assure that all the maintenance steps have been completed. Look at routing tables for advertisements. Remove the new interface configuration, or document the change and shut the interface down. Remove static routes if applicable. Approval Process 1. Obtain approval through the local process for the maintenance with the above plan specifications Perform Maintenance 1. Notify the Network Operations Center (if applicable) that maintenance is beginning 2. Work through the maintenance procedure to conclusion 3. Test all new prefixes to confirm that they are advertised correctly a. If not, then troubleshoot b. If it cannot be fixed, back out 4. If everything checks out correctly, then close the maintenance with NOC 2013 Packet Design, Inc. Page 4 of 18

How Route Analytics Helps in Adding IGP Routes Route analytics has full knowledge of all routed prefixes, metrics and netmasks, and provides a network-wide prefix list (see Figure 1) that shows all routes that are active in the network; this provides 100% assurance that the prefix and mask do not already exist in the infrastructure. Figure1: Network-wide prefix list aids in ensuring that a proposed added route doesn t already exist somewhere in the network. Using the same prefix view, network engineers can also confirm the summary block that holds this prefix and the source from which it is advertised. This will ensure the summary will be advertised correctly. Before the maintenance, network engineers can view relevant routing event streams by router and/or area, to confirm that the router and region to be changed are not experiencing significant churn. For example, they can look at a particular router (Figure 2) to launch the events list (Figure 3) and view the level of activity on the router. If there is a high level of churn, the engineer can work with the responsible person on call to return the router and region to a more stable state. This step is important because starting maintenance while other problems are occurring in a region can impact diagnostic efforts for another engineer who might be working in the region. (It also prevents the change operation from becoming the unwitting scapegoat for problems that were already occurring pre-maintenance.) 2013 Packet Design, Inc. Page 5 of 18

Figure 2: Route analytics visualizes the entire network topology, showing the actual state of routing at any point in recorded time, and offers full interactivity including the ability to drill down by choosing a router and seeing its prefixes, events, and neighbors. 2013 Packet Design, Inc. Page 6 of 18

Figure 3: Events view for an individual router, over a specified period of time. When the route is added to the protocol on the router, the same events view can be used to verify that the router has propagated the change. In addition, route analytics can be used to increase the accuracy of the change operation by executing the following steps: Examine prefix lists from within the area or level, and from outside, to see if the summary is working correctly. Verify from which router the route originated, where it is summarized, what netmask and metric are associated with it, and how traffic from other parts of the network will use the route to move data across the network. If the route is not behaving as desired, route analytics can be used to model metric changes to route traffic correctly. The network engineer highlights a routed path across the network in the topology view, then clicks on the relevant link, adjusts the metric, and finally observes how the path changes. Figure 4 shows an example of highlighting a path and changing a metric in an EIGRP network. 2013 Packet Design, Inc. Page 7 of 18

Figure 4: Metric changes can be modeled with route analytics by selecting a path (highlighted in yellow above) and simulating new metrics. Once the metrics are changed, route analytics displays precisely how the routed path will change as a result. Route Analytics Applied to Data Center Migration Route analytics can also be used for more significant network operations such as simulating the effects of a planned migration of servers from one data center to another. The network illustrated in this case study is that of a multi-national Fortune 500 enterprise running EIGRP. The key question was to determine what would happen to link utilization on core WAN circuits if servers were moved to a different data center. For the sake of scope, a particularly important circuit was chosen for focused analysis, as shown in Figure 5. 2013 Packet Design, Inc. Page 8 of 18

What happens to link utilization of core WAN circuits if servers are moved from this location to this location? Route analytics can show the effect on all circuits, but this circuit will be used to illustrate in detail Figure 5: Route analytics helps engineers understand how WAN circuits would behave after a data center migration. Since route analytics records and stores a complete forensic history of all routing and traffic changes, network managers were able to "rewind to a point in time where there was a historically high level of WAN circuit utilization, and use this as the basis for the planning and simulation exercise. A histogram of traffic utilization across any selected link or any portion of the network in aggregate is seen in Figure 6. 2013 Packet Design, Inc. Page 9 of 18

Route analytics lets users view the state of routing and traffic at any point in the recorded database In this case, the network manager moves back in time to a period of high utilization (to see worst case-scenario) Figure 6: Network managers found a past point of high utilization to use as the basis for their data center migration simulation exercise. Clicking on any link in the network map reveals rich details on routing and traffic state at that moment in time. In this case, network managers were able to see the precise utilization of the WAN link in question, in both directions, as seen in Figure 7. 2013 Packet Design, Inc. Page 10 of 18

At the selected time period: Southbound traffic on this link is 37.72Mbps (it is experiencing high utilization, so half the link is red) Northbound traffic utilization is negligible, so the northbound half is grey (indicating low utilization) Figure 7: Route analytics network map provides rich details on routing and traffic for links, paths, and routers, such as WAN link utilization. Having selected a particular point in time, the network manager can simulate routing and traffic changes based on the state of the network at the time being viewed. The network manager selects the router in the data center that delivers the traffic from the servers that are to be moved, and displays a list of network addresses (or prefixes). Then he simulates the downing of the prefix representing the server cluster, as seen in Figure 8. 2013 Packet Design, Inc. Page 11 of 18

For the data center router, the network manager displays the list of networks (prefixes) it supports... then simulates downing the prefix to be moved Figure 8: Route analytics allows network managers to view comprehensive information such as a list of all routed prefixes, which eases network change simulations such as downing a prefix. Next, the network manager opens the router in the data center to which the servers are being moved, and adds the downed prefix from the first data center to simulate the server cluster being brought up in that data center, as seen in Figure 9. Route analytics simulates the re-routing of all traffic destined for this prefix to the new network location. This simulation is completely accurate, since it is based on the precise state of routing and traffic at that moment in time. 2013 Packet Design, Inc. Page 12 of 18

Traffic Explorer can simulate moving servers from the first data center to the second one by moving the prefixes, thus re-routing traffic to the new datacenter. Figure 9: Moving the server cluster is modeled by adding the previously downed prefix from the first data center to a router handling traffic at the new data center. Route analytics now displays a network map, as seen in Figure 10, which shows that utilization in the southbound direction on the WAN link has decreased slightly, while the northbound utilization has changed from negligible to high. This is important intelligence for the network engineering team, which can now perform further analyses to find a way to accommodate the proposed change without incurring new costs. 2013 Packet Design, Inc. Page 13 of 18

After the modeled change: Southbound traffic on this link has DECREASED to 37.67Mbps from 37.72Mbps (it is still high utilization, so half the link is red). #2 For Northbound traffic, utilization is HIGH, so the Northbound half is RED. Figure 10: Route analytics shows precisely how the network would behave after a planned change. In this case, northbound link utilization for the WAN circuit in question would be too high, so further planning adjustments will be needed before this change is implemented. For the specific WAN link examined in this case, an unacceptable utilization level would occur as a result of this data center move, prompting the network managers to further analyze their options. Route analytics provides comprehensive link utilization analysis capabilities to ease analysis of the before-and-after effects of specific change scenarios. 2013 Packet Design, Inc. Page 14 of 18

Network managers can examine how ALL of the links in the ENTIRE network have been affected by this move via Link Utilization, Link Traffic and Available Capacity Reports. Figure 11: Route analytics provides comprehensive link utilization information on a before-andafter simulated change basis. 2013 Packet Design, Inc. Page 15 of 18

We can see available capacity across the ENTIRE network after simulating the data center move. Figure 12: Route analytics provides insight into network-wide available capacity for real-time monitoring as well as for historical analysis and modeling purposes. Using this intelligence and further network change modeling and simulation, the network managers may be able to find a way to adjust routing to accommodate the traffic without incurring additional WAN circuit costs (e.g., by modifying certain router metrics). Upon performing these simulations, they may further find, by looking at link traffic analyses and finding links with low utilization, that certain WAN links can be decommissioned or cost-reduced (see Figure 13). 2013 Packet Design, Inc. Page 16 of 18

This circuit bandwidth can now be reduced to save cost! Figure 13: Network managers can find ways to optimize use of existing network assets to contain or reduce costs by modeling changes and finding under-utilized or unutilized links. Benefits of Route Analytics to Change Management Processes Route analytics provides a number of benefits for IT change management operations: Increases accuracy of network planning and change operations Mitigates risk of impacting application performance due to unexpected behavior when executing planned changes Helps maximize existing network assets and contain or reduce WAN bandwidth costs Improves IT productivity by automating previously highly manual tasks such as maintaining an up-to-date network map 2013 Packet Design, Inc. Page 17 of 18

Conclusion As pressure increases on IT to deliver predictable application availability and performance, especially with the advent of more sensitive converged services such as VoIP, network managers need more than just device-centric approaches to ensure that change management operations are accurate on a network-wide basis. Route analytics provides the global network visibility needed to complement and strengthen existing change management tools and processes, ensuring that they are accurate and efficient. With network-wide visibility of traffic behavior, network managers can deliver higher service levels while maximizing IT assets. To learn more about Packet Design and its industry-leading route analytics solutions, please: Email us at info@packetdesign.com Visit Packet Design s web site at http://www.packetdesign.com Call us at 408.490.1000 Corporate Headquarters Packet Design Inc. 2455 Augustine Drive Santa Clara, CA 95054 Phone: 408.490.1000 Fax: 408.562.0080 http://www.packetdesign.com 2013 Packet Design, Inc. Page 18 of 18