A How-To Guide to BGP Multihoming

Transcription

1 Version 1.2 Lane Patterson A How-To Guide to BGP Multihoming Lane Patterson <lane@equinix.com> <louie@equinix.com> Equinix, Inc. How To Use This Paper This paper is geared toward readers who have a reasonable background in corporate and Internet networking, but are not experts in BGP or backbone Internet routing architectures. If you are evaluating BGP Multihoming, and need to justify and plan for it internally, Section 2 and 5 will be of interest. If you are ready to plan router configurations for BGP Multihoming, then Section 3 and 4 will be of interest. Executive Summary BGP Multihoming is a method that many medium and large enterprises and content providers use to attach their networks to two or more Internet Service Providers using Border Gateway Protocol version 4. Since there is a lack of up-to-date public information on BGP Multihoming, many organizations have either avoided it, or have implemented severely limited BGP Multihoming architectures, and have thus been unable to maximize the value of this technology. This white paper provides an updated overview of BGP Multihoming, including prerequisites, basic implementation examples and router configurations, and advanced techniques and technologies for optimizing cost, performance, and flexibility of your multihomed Internet platform. We begin with a review of multihoming prerequisites. We discuss the requirement to run BGP with your providers, and review the basics of this well-known routing protocol. We provide an overview of the process of registering an Autonomous System Number (ASN), which is a required identifier that allows BGP to group sets of routes by the organization that announces them. We review the way routes are propagated throughout the Internet by ISPs, and why this impacts your IP addressing plans. In particular, if you obtain public IP address space from an upstream ISP, you need a minimum of a /24 address block to implement BGP Multihoming. If your organization has a requirement for at least a /20 (Equivalent to sixteen /24 s), you can register for address space directly from an addressing authority such as ARIN ( and enjoy the benefits of portable address space. Finally, we present the basic router performance requirements for running BGP requirements that are increasing within the scope of standard enterprise routers. Three complete BGP Multihoming examples are presented, including network diagrams, commentary, and router configurations. We consider the most common cases: single site, single router; and single site, dual routers. We show the difference between using provider-assigned address space and portable address space. We show how to verify BGP policy propagation by using public route views servers. Advanced BGP techniques for static load balancing are presented. For outbound load balancing, we provide examples of preferencing routes based on large, well-known intermediate ASNs. For inbound load balancing, we present prefix engineering, AS-path prepending, and provider-supported BGP communities, as available tools. Despite all these techniques, we point out that standard BGP tweaking is not based on real time performance, traffic, or cost considerations, but simply manipulation of a routing protocol designed to ensure network connectivity. We present some building blocks of advanced multihoming strategies and technologies, designed to provide better cost, performance and flexibility efficiencies. On the business side, we note the important role that ISP bandwidth contract negotiation can have, and the benefits of carrier-neutral colocation for eliminating local loops and providing a competitive bandwidth market. In such an environment, it is Page 1 of 1

2 possible to move from a scenario of single-homed to a single, expensive provider towards a scenario of multihomed with majority of traffic going through an inexpensive competitive provider, and a lowcommit burstible backup to a larger, more expensive, provider. We present two new technologies that introduce cost and performance benefits. The first are multihoming platforms: services at carrier-neutral locations that automated the process of multihoming, and allow you to connect to multiple ISPs through a single router port 1. The second technology is known as route control. For sites that are multihomed, route control solutions seek to overcome the limitations of standard BGP by continually measuring the performance and cost of bandwidth through each provider link, and dynamically preferencing BGP routes based on performance and/or cost. 2 In conclusion, we note that BGP Multihoming is continuing to evolve in ways that can help organizations meet their Internet performance, reliability, and redundancy goals, without increasing costs. We plan to continue to publish revis ions of this white paper based on feedback and market developments, in order to promote the most effective use of BGP Multihoming. 1. Introduction This white paper was developed to promote effective BGP Multihoming by medium and large enterprise and content sites. Despite the strategic significance of BGP Multihoming, it remains a poorly documented and often misunderstood topic for a number of reasons: ISPs do not promote multihoming because it reduces customer dependence on their services Many prior online multihoming tutorials are incomplete or have not been kept up to date as living documents Organizations that are prime candidates for multihoming may avoid it due to perception of cost, complexity, or technical expertise required. This paper is geared to readers who have a reasonable background in corporate and Internet networking, but are not experts in BGP or core backbone routing architectures. It assumes you work for an organization with mission critical Internet requirements and sizable Internet traffic. Given today s realistic environment of cost control, we maintain a practical focus on how to use BGP Multihoming to do more with less, including bandwidth and local loop cost reduction strategies and new route control technologies. We have aimed to provide the kind of How-To information that will allow you to evaluate, budget for, propose and defend, design, implement, and optimize, a BGP Multihoming solution for your organization. This paper is organized as follows: Section 2. Multihoming Prerequisites: This section presents items that need to be considered in preparation for BGP Multihoming: router performance requirements, IP address space, basic BGP and routing concepts, and obtaining an Autonomous System Number (ASN). Section 3. Complete Multihoming Examples: This section presents diagrams, router configurations (in Cisco IOS format), and commentary on the three most common BGP Multihoming configurations. It presents techniques and resources for verifying and troubleshooting a multihomed routing policy. 1 For example, a product called Equinix Direct is an automated platform for multihoming through a single router port. Its strengths include no bandwidth commits, month-to-month terms, web-driven selection of providers by price and brand, and automated configuration when switching providers. 2 For example, Equinix Intelligent Routing Service is a route control service based on an appliance from netvmg ( Sockeye and Route Science are examples of other route control solutions. Page 2 of 2

3 Section 4. Advanced BGP Techniques--Static Load Balancing: This section discusses standard BGP configuration techniques and examples for load balancing inbound and outbound traffic across multiple provider links, and identifies the limitations of these techniques. Section 5. Multihoming Strategies and Technologies: This section discusses practical business and cost considerations when multihoming, such as bandwidth contract negotiation and optimization, ISP selection and switching, and elimination of local loops in carrier-neutral facilities. It presents some pitfalls of 95 th percentile traffic billing that can lead to double-billing if not managed properly. It discusses new BGP Multihoming technologies such as Intelligent Routing and Equinix Direct that can help lower cost and improve performance. Section 6. Conclusions. 2. Multihoming Prerequisites So you want to know what it takes to multihome? Let s start by making sure we are clear of the definition of multihoming, then we ll talk about what it takes to do BGP, IP addressing requirements and options for multihoming, and how to make sure your router is beefy enough for the task. 2.1 Definition of Multihoming BGP Multihoming: maintaining links to multiple Internet providers (usually 2 or 3) and using BGP to send routes and to receive full routing tables from these providers. 2.2 BGP Requirement BGP is the routing protocol required to connect to multiple Internet providers according to common multihoming practices 3. Its purpose is for propagating routing information between Autonomous Systems separately administered networks, whether they are large or small ISPs, or multihomed content providers or enterprises. BGP is an Exterior Gateway Protocol (EGP), and so is designed for the routing policy requirements that exist when two or more separate networking organizations interconnect. By contrast, an Interior Gateway Protocol, such as OSPF 4, is designed to propagate routes within a network that is under the control of a single organization, and does not have the level of sophistication and policy control to represent and control routing among multiple organizations. Each Autonomous System must have a unique Autonomous System Number (ASN). Think of an ASN as a meta-label that is applied to the set of routes from an organization. Whether you are a small multihomed organization with one or two public Internet routes, or a large ISP with 25,000 Internet routes, BGP will tag all the routes from an AS with the ASN, to keep track of the network entities that must be traversed to reach a destination network. BGP is not rocket science and is bundled in most routing software running on popular routers. But it does require some conscientious effort from a network engineer and an awareness of existing best practices and configuration guidelines, to set up a BGP routing session with a provider. BGP replaces the default route common in single-homed configurations with a protocol that dynamically informs your router of every specific routing prefix in use on the Internet (aggregated according to CIDR best practices), and the associated BGP attributes for each route. BGP as it pertains to common multihoming examples is described in more detail in the section BGP Requirement: Overview and Concepts later in this document. 3 Border Gateway Protocol version 4 (BGP) is an Internet Standard routing protocol described in RFC 1771 ( ). 4 Open Shortest Path First version 2 (OSPFv2) is an Internet Standard routing protocol described in RFC 2328 ( ). Page 3 of 3

4 2.3 ASN Requirement: HOW TO for Getting an ASN The first step in implementing multihoming is to obtain a publicly registered Autonomous System Number (ASN). In North America, the Caribbean and sub-saharan Africa, ARIN ( is the organization delegated the responsibility of assigning ASNs 5. Other countries and regions have their own registries for obtaining ASNs. 6 The process of registering for an ASN from ARIN is described in detail at While ARIN is the final authority for this process, here are some pointers and highlights: 1. Complete and submit the ARIN ASN Request Template. Make sure you take the time to review this carefully, and if you are unsure of anything, get outside review from consultants or sources that are knowledgeable about the process. 2. ARIN should give you acknowledgement and a tracking number for you submission, and respond within 3 working days. 3. To save time, start your review and preparation of the ARIN Registration Services Agreement and the ASN Billing Account Form. Once your ASN Request Template is approved, ARIN will ask you to submit these. 4. The Credit Card option is the fastest form of payment. 5. Once ARIN has received and processed everything, they will issue your ASN. You should be able to query the ARIN database for your ASN and point of contact (POC) information within a couple days of receiving notice from ARIN. Take time to verify this at 6. You will probably find it useful to register your maintainer ID, ASN, and routing announcements with a routing registry. In North America, RADB is commonly used ( Many providers use the RADB to determine valid routing announcements, and may not listen to your routes unless they match the policy you register in the RADB. 2.4 IP Addressing/Routing Requirement Unless you have sufficient IP addressing requirements, you probably have received your IP subnets (a.k.a. routes, prefixes, netblocks) from an upstream ISP. These subnets are commonly part of a larger block of address space that your ISP has been assigned by a registry such as ARIN. This type of IP address prefix is known as PA-space, for Provider-Assigned. Organizations within the ARIN territory that have demonstrated a requirement for more than a /21 (Eight /24 s) can request a minimum of a /20 (Sixteen /24 s) of IP address space directly from ARIN, according to their guidelines at This type of IP address space is known as PIspace, for Provider-Independent. Just as with the process for requesting an ASN, be prepared to familiarize yourself with ARIN s process for documenting IP address requirements and correctly reviewing and submitting applications. If your documentation is clean, you will find that it is a smooth process. The minimum prefix you must have to multihome is a /24. If your current provider-assigned subnet is smaller, such as a /27, then you will need to work with your provider to request a full /24 assignment, justified by a request to run BGP and do multihoming. Since this may involve re-numbering, you will want to pick the provider you use for your /24 assignment carefully. Regardless of which other providers to which you multihome, you will necessarily have to retain the provider that supplies your address space. 5 ARIN s policies for registering an Autonomous System Number are published at 6 Links to registries around the world can be found at Page 4 of 4

5 You also need to consider the routability of your address blocks, something that ARIN and other registries don t guarantee (they just assign the space routing it is up to you and your providers). Routability simply means that you announce your prefix(es) via BGP to your upstream providers, and that they in turn announce them to their peers so that they are visible throughout the Internet. However, you must always keep in mind that the most specific route always wins. For example, if your primarily provider only announces your /24 as part of a /13 prefix, and your secondary provider announces it as a specific /24 prefix, then all your inbound traffic will come in through your secondary provider. If this is important to you, get agreement from your providers on how they will propagate your route announcements, and whether any major peers are likely to ignore them. To re-cap IP addressing and routability issues with Provider-Allocated (PA) space: 1. Make sure your existing provider supports your requirement for a minimum of /24 for the purpose of BGP Multihoming 2. Determine fees and policies and procedures for the address space and BGP setup with each provider 3. Ensure that your existing provider is willing to propagate your BGP announcement of your discrete address block (e.g. /24) to their peers, in addition to the entire aggregated prefix (e.g. /13) that they would normally announce. This need not apply if you have no requirement to load balance inbound traffic, and don t mind if your inbound traffic comes in primarily through your secondary provider(s). 4. Ensure that your secondary provider(s) support your requirement to run BGP with them to propagate your prefix(es) to their peers. (*Note: their peers are not necessarily obliged to accept these more-specific announcements, and may filter such that they only listen to the aggregated prefix announced by your primary provider. This is not a widespread practice, but is known to be followed by two large ISPs) If you are able to justify Provider-Independent (PI) space from your address registry, you will not be anchored to one primary ISP, and can freely switch ISPs without renumbering your addresses. You will not need to worry about the issues above that apply to PA-space. 2.5 Router Hardware, Memory, and CPU Requirements The final requirement for multihoming is router horsepower. For starters, you ll need extra ports on your router for multiple ISP connections, sized according to your current and planned traffic requirements. The other main requirement is for RAM. Receiving full BGP routing tables from your providers requires at least 128 MB RAM, and preferably 256 MB RAM. RAM requirements depend on how many transit providers you will have (this is a variable you control), as well as how much natural growth occurs in the BGP routing tables (this is a variable you cannot control, and it pays to allow for growth room of 30-50%). You will also want to keep a minimum of at least 30 MB free RAM. Of course RAM requirement is also a function of any other features you choose to run on your router. Here s one example of RAM utilization: In early 2003, a Cisco 7206 VXR with 256 MB RAM is configured with minimal feature use and full BGP tables from 4 different providers. The size of the BGP tables averages 118,000 routes. RAM utilization is observed to be about 108MB. Router resources won t be an issue for a long time. A third requirement to keep in mind is CPU utilization. Running BGP, and related processes like BGP route flap dampening (a common feature to reduce instability in your network caused by unstable routes elsewhere on the Internet), take a certain amount of CPU processing power. On properly sized routers, this is usually not a problem. But don t cut it too close. I saw one example of a company multihomed to 3 providers using a Cisco 3640 with 128MB RAM. Since the Cisco 3640 doesn t have hardware-based forwarding to the extent that other router makes/models do, CPU utilization was also influenced by their daily traffic peaks. During traffic of mbps, the router was comfortable at 35% CPU utilization. But when traffic occasionally peaked at mbps, the CPU became saturated, and an immediate router Page 5 of 5

6 upgrade was required. By contrast, routers with fully distributed hardware-based forwarding usually see very low CPU loads, even when running BGP to multiple upstream providers and pushing traffic at line rate. This brings us to the topic of traffic throughput. Keep in mind that just because a router can be configured with a FastE or GigE interface, doesn t mean that the router has the horsepower to fill up that interface. For instance, a Cisco 7200 uses PCI busses internally for packet forwarding, and you should generally limit total traffic throughput to less than 300 mbps. Extreme and Foundry switches, while excellent at forwarding at line rate as a layer 2 ethernet switch, are generally much more CPU-intensive when used as IP routers. Be careful to do your research on what to expect when you combine your BGP, IP routing, traffic volume, and feature requirements on the same box, so you don t get surprised. At the high end, Juniper has an excellent performance to price ratio on the M5 and M10 platforms, and has a proven reputation for being able to run full line rate with important features enabled. Cisco also has newer, more robust, routers such as the 7300 family, with PXF hardware acceleration and 2 built-in GigE ports. If you are handling more than mbps, this platform may give you room for growth without requiring a costly GSR purchase. The 7600 series is also generally a good value for high-end access needs, and provides a single box to combine the switching and routing requirements of many end sites. This is by no means a complete analysis, and it won t take long for this information to become obsolete, so make sure to get current recommendations at the time you develop your multihoming requirements. 2.6 BGP Requirement: Overview and Concepts So you are ready to run BGP This section is not meant to be an exhaustive BGP tutorial there are plenty of more in-depth resources available for that. What it does cover is the high level concepts that may be new to you if you are used to a single-homed setup. Default Routing: This is an easy way to route traffic when there is only one path available for all destinations. You simply configure the same next hop router for ANY destination. For example, hosts such as desktop and server PCs are usually configured with a default gateway. Full Routes: This is the term for receiving every routable Internet prefix explicitly, so that you do not need a default route to your Internet providers. As of early 2003, there are approximately 120,000 routes in a full BGP table for Internet transit. Depending on the routing policies of a given ISP, this number can vary between approximately 105,000 routes and 135,000 routes. Given that there are between million Internet users world wide, routes typically represent highly aggregated sets of end users. Default routing does not work with multihoming, since the whole point is to pick between multiple possible links for every Internet destination. Receiving full routes gives your router the information needed to effectively multihome. Asymmetric Routing: Internet routing is a not a symmetric proposition. In order for you to receive inbound traffic, you or your providers must announce the route to get to you. In order for you to respond with outbound traffic, you must receive and accept routes back to others. The two processes are completely decoupled and independent. Whereas most single-homed customers are used to thinking of all their inbound and outbound traffic going through the same path, the complete traceroute inbound versus outbound is almost always different. Asymmetric routing is the norm in the core of the Internet. Announcing Routes: Outbound route announcements influence inbound traffic. Running BGP means that your route announcements are now dynamic, not static. It is up to you to announce your prefixes to your upstream providers, and for them to pass these prefixes on to other networks. The distribution of your routing information throughout the Internet allows end networks to learn how to get to you. This is a benefit, for instance, when your link to one ISP fails, because BGP will automatically withdraw route announcements through that path, and your inbound traffic will fail over through the remaining path(s). There are specific BGP techniques available (discussed more in Section 4.) that can help you influence which inbound path will be used to reach your site. But remember that ultimately, these techniques only pass on hints to other networks, and the final decision how to route inbound to you is really up to them. Page 6 of 6

7 Best Practices for Announcing Routes: The most important best practice for announcing routes is to make sure you apply a prefix-filter to the routes you announce to your providers that restricts the announcements to your specific aggregated netblocks. This prevents two important things that would otherwise occur with BGP: (1) announcing routes you learned from provider A back out to provider B (which would inadvertently make you an alternative path between provider A and B) and (2) announcing both your aggregate prefix, and all those nifty internal subnets you ve broken it into for internal purposes. Accepting Routes: Routes received and accepted influence outbound traffic. When each of your transit providers is announcing valid routes to all destinations on the Internet, you have the ability to make choices about which provider to choose, which affects your outbound traffic load balancing. Unlike routing of inbound traffic, you are in complete control of the next -hop selection of outbound traffic. How you set preferences on the routes you receive will determine which outbound transit provider is preferred for a given destination. There are a number of ways to set preferences on received routes. BGP has a standard selection process for determining which route is preferred, when several are available to the same destination prefix 7. This decision process allows plenty of hooks for you to set your own choices, commonly via the BGP local preference attribute. For example, you select subsets of received routes based on BGP communities (e.g. all routes with the customer community value set by your upstream provider), AS-path operations (e.g. all routes destined to AOL s ASN), or even specific prefixes. Once you match these subsets of routes, you can perform an action on the subset, such as setting local-preference higher or lower, to help prefer or avoid the given path. Best Practices for Accepting Routes (from Transit Providers): We must distinguish the practices for accepting routes from transit providers versus peering partners, as they are very different. The most important things to filter out when accepting routes from transit providers are (1) bogon networks, such as RFC1918 private networks (e.g /8) that you may use internally, and which should never be routed across the public Internet, and (2) limiting the maximum number of routes you receive from your transit provider, so a mistake on their part does not starve the memory on your router. (e.g. set max-prefix to 140,000 if you commonly receive 120,000 routes from your provider. Update these numbers every few months.) Other operations you will commonly perform on accepting routes and setting routing preferences fall into the performance tuning or traffic engineering category, and are not mandatory best practice. Many good best practices with router configuration examples and explanations are described at ibgp mesh requirement: When you setup BGP to an external ASN, such as your upstream providers, it is known as External BGP (ebgp). If you have multiple Internet routers in your own network, you also have a requirement to run Internal BGP (ibgp) between them. ibgp sessions are generally more simple to setup, but they are absolutely required. All routers that will be routing from your ASN to the public Internet must have a full mesh of ibgp sessions among them. For a pair of Internet routers, this is simply one ibgp session. But for 4 Internet routers, this adds up to 6 ibgp sessions to complete the full ibgp mesh. We have presented important high-level BGP concepts, without getting into specific mechanisms for accomplishing them. In the next section, we will show some complete examples of common multihoming cases, including example router configurations for addressing and routing. 7 The BGP Decision Algorithm is presented in Page 7 of 7

8 3. Complete Multihoming Examples This section covers the most common basic multihoming scenarios. Each scenario consists of a network diagram, description of the scenario, and an example of the BGP router configuration required to implement it. Cisco IOS configuration commands are used. Note that these are bare bones configs no advanced load balancing or preferencing route maps are applied. These will be described in separate config snippets in the Advanced BGP Techniques section. 3.1 Example 1: Single Site, Single Router, PA-space, Multihoming to Two ISPs Background: In this example, a single organization is connecting at a single site, using a single router, to two ISPs (without regard to which one is primary and/or secondary). It is receiving full transit BGP routes from each provider, and is allocated Provider-Assigned (PA) address space from the first provider, ISP 1. Figure 1 shows the network architecture and relevant details for this example. Note: When PA-space is used, the provider that owns the address block (in this case, ISP 1) will announce their aggregate block to the rest of the Internet, as well as (if you arrange it properly with them) propagating your specific /24 route announcement originated from your ASN. The caveat with this case is that other Internet peers are not required to accept the more specific route. While this is not common, it may occur that a 3 rd party ISP could only reach you through the ISP1 path, based on the aggregate route announcement. If your link to ISP1 goes down, these 3 rd party ISPs could end up still trying to route through ISP1, unaware that a more specific route exists through ISP2. Note: ISP1 originates its own aggregate route, but ALSO propogates specific Customer route from ASN /16 AS-Path /24 AS-Path 900, 200 "Internet" "Internet" /24 AS-Path 900, 300 Note: Since ISP2 does not own the address block, it only passes on the Customer-originated specific route ISP 1 ASN 200 ISP 2 ASN /30 Full BGP Routes Link: /30 BGP Announce: /24 AS-path /30 Router /30 Full BGP Routes /30 Link: /30 BGP Announce: /24 AS-path 900 Multihomed Customer ASN 900 Prefix /24 (assigned from ISP 1's aggregate of /16) Figure 1. Multihoming: PA-space Addresses, Single Site, Single Router, Two ISPs Page 8 of 8

9 Example 1 Router Configuration Single-router, PA-space, Basic Config Define your BGP ASN on your router autonomous-system 900 Define static NULL route to networks that will be announced to providers via BGP. ip route Null0 200 Define ANNOUNCE prefix list, of your netblocks to announce via BGP to your providers. You will apply this prefix- list outbound on the BGP session to each provider. ip prefix-list ANNOUNCE description Our External Netblocks ip prefix-list ANNOUNCE seq 10 permit /24 Define BOGONS prefix list, of bad netblocks you need to block from being accepting from your providers. Don t just trust your provider to run a clean network You will apply this prefix-list inbound on the BGP session to each provider. ip prefix-list BOGONS description Bad Routes to Block In ip prefix-list BOGONS seq 10 deny /8 le 32 ip prefix-list BOGONS seq 15 deny /8 le 32 ip prefix-list BOGONS seq 20 deny /8 le 32 ip prefix-list BOGONS seq 25 deny /12 le 32 ip prefix-list BOGONS seq 30 deny /24 le 32 ip prefix-list BOGONS seq 35 deny /16 le 32 ip prefix-list BOGONS seq 40 deny /3 le 32 Prevent someone else from announcing your own prefix(es) back to you, for security: update this with YOUR actual announced block(s) ip prefix-list BOGONS seq 1000 deny /24 le 32 Accept any other routes bigger or equal to /27. You can tweak this up to /24 if you like. ip prefix-list BOGONS seq 9999 permit /0 le 27 router bgp 900 don t require your IGP to be in synch with BGP, synchronization has been outmoded for some time. no synchronization tell your router to log changes to your BGP sessions, you ll want to be concerned with BGP sessions when they go up and down, it s just as important to your routing as a link up/down. bgp log-neighbor-changes enable BGP dampening to minimize adverse impact of flapping routes (routes that are announced and withdrawn repeatedly). bgp dampening define your BGP network statements: these are the aggregate external IP blocks you will be announcing to the Internet. Note that the network statement will not be effective unless there is an underlying route for the network, which is why we defined a static NULL route for this block above. network mask Page 9 of 9

10 define our BGP session with ISP-1 (ASN 200) neighbor remote-as 200 description allows you to put add a text label neighbor description BGP Transit to ISP-1 hard-code version 4 to short-cut BGP version negotiation neighbor version 4 send-community is nice if you will be setting communities on routes you announce to influence how your upstream provider re-announces the routes to the Internet. Many providers support sophisticated community sets to allow this kind of customer control. neighbor send-community Soft reconfiguration is nice, it prevents complete withdrawal and relearning of routes when doing clear ip bgp command. But it does require enough RAM to cache an extra copy of the table. neighbor soft-reconfiguration inbound Filter out bogus prefixes from your upstream. Don t trust your ISP to do this for you. neighbor prefix-list BOGONS in Limit your announcement just to your public prefix(es). This enforces aggregation, and prevents you from announcing ASN 200 s routes to ASN 300, which would accidentally make yourself a transit between the two ISPs. neighbor prefix-list ANNOUNCE out enforce max-prefix limit: just in case your provider blows up their routing tables, this keeps your router from melting under the stress by shutting off the mis-behaving BGP session instead. Once your ISP fixes the problem, you can re-enable with a clear ip bgp... neighbor maximum-prefix define BGP session with ISP-2 (ASN 300) neighbor remote-as 300 neighbor description BGP Transit to ISP-2 neighbor version 4 neighbor send-community neighbor soft-reconfiguration inbound neighbor prefix-list BOGONS in neighbor prefix-list ANNOUNCE out neighbor maximum-prefix end 3.2 Example 2: Single Site, Dual Routers, PA-space, Multihoming to Two ISPs Background: This example is the same as Example 1, except the customer is using two routers, and connecting to one ISP on each router. The purpose of this example is to show how ibgp is used to internally propagate BGP routes between the two routers. Page 10 of 10

11 Note: ISP1 originates its own aggregate route, but ALSO propogates specific Customer route from ASN /16 AS-Path /24 AS-Path 900, 200 "Internet" "Internet" /24 AS-Path 900, 300 Note: Since ISP2 does not own the address block, it only passes on the Customer-originated specific route ISP 1 ASN 200 ISP 2 ASN / /30 Full BGP Routes Link: /30 BGP Announce: /24 AS-path / /30 ibgp Router1 Router / /30 Multihomed Customer ASN 900 Prefix /24 (assigned from ISP 1's aggregate of /16) Full BGP Routes Link: /30 BGP Announce: /24 AS-path 900 Figure 2. Multihoming: PA-space Addresses, Single Site, Dual Routers, Two ISPs Example 2 Router Configurations Router 1: Dual-router, PA-space, Basic Config autonomous-system 900 ip route Null0 200 ip prefix-list ANNOUNCE description Our External Netblocks ip prefix-list ANNOUNCE seq 10 permit /24 ip prefix-list BOGONS description Bad Routes to Block In ip prefix-list BOGONS seq 10 deny /8 le 32 ip prefix-list BOGONS seq 15 deny /8 le 32 ip prefix-list BOGONS seq 20 deny /8 le 32 ip prefix-list BOGONS seq 25 deny /12 le 32 ip prefix-list BOGONS seq 30 deny /24 le 32 ip prefix-list BOGONS seq 35 deny /16 le 32 ip prefix-list BOGONS seq 40 deny /3 le 32 Prevent someone else from announcing your own prefix(es) back to you, for security: update this with YOUR actual announced block(s) ip prefix-list BOGONS seq 1000 deny /24 le 32 Page 11 of 11

12 ip prefix-list BOGONS seq 9999 permit /0 le 27 router bgp 900 no synchronization bgp log-neighbor-changes bgp dampening network mask neighbor remote-as 200 neighbor description BGP Transit to ISP-1 neighbor version 4 neighbor send-community neighbor soft-reconfiguration inbound neighbor prefix-list BOGONS in neighbor prefix-list ANNOUNCE out neighbor maximum-prefix define ibgp session neighbor remote-as 900 neighbor description ibgp to Router2 neighbor version 4 neighbor send-community neighbor soft-reconfiguration inbound make sure you are using a Loopback0 /32 on each router, and propagating this route internally. This is important to the stability of your ibgp sessions, so they are not tied to a physical interface address. neighbor update-source Loopback0 end Router 2: Dual-router, PA-space, Basic Config autonomous-system 900 ip route Null0 200 ip prefix-list ANNOUNCE description Our External Netblocks ip prefix-list ANNOUNCE seq 10 permit /24 ip prefix-list BOGONS description Bad Routes to Block In ip prefix-list BOGONS seq 10 deny /8 le 32 ip prefix-list BOGONS seq 15 deny /8 le 32 ip prefix-list BOGONS seq 20 deny /8 le 32 ip prefix-list BOGONS seq 25 deny /12 le 32 ip prefix-list BOGONS seq 30 deny /24 le 32 ip prefix-list BOGONS seq 35 deny /16 le 32 ip prefix-list BOGONS seq 40 deny /3 le 32 Prevent someone else from announcing your own prefix(es) back to you, for security: update this with YOUR actual announced block(s) ip prefix-list BOGONS seq 1000 deny /24 le 32 ip prefix-list BOGONS seq 9999 permit /0 le 27 router bgp 900 no synchronization bgp log-neighbor-changes bgp dampening network mask define BGP session with ISP-2 (ASN 300) neighbor remote-as 300 Page 12 of 12

13 neighbor description BGP Transit to ISP-2 neighbor version 4 neighbor send-community neighbor soft-reconfiguration inbound neighbor prefix-list BOGONS in neighbor prefix-list ANNOUNCE out neighbor maximum-prefix define ibgp session neighbor remote-as 900 neighbor description ibgp to Router1 neighbor version 4 neighbor send-community neighbor soft-reconfiguration inbound make sure you are using a Loopback0 /32 on each router, and propagating this route internally. This is important to the stability of your ibgp sessions, so they are not tied to a physical interface address. neighbor update-source Loopback0 end 3.3 Example 3: Single Site, Dual Router, PI-space, Multihoming to Two ISPs Background: This example is the same as Example 2, except the customer has qualified for and is using an ARIN-allocated /19 address block, instead of a provider-assigned /24. Figure 3 shows the difference in how this routing announcement propagates outward throughout the Internet, since it is not covered by an overlapping aggregate announcement by ISP 1. Page 13 of 13

14 "Internet" "Internet" /19 AS-Path 900, /19 AS-Path 900, 300 ISP 1 ASN 200 Note: Since the customer owns its own portable address block, both ISPs simply propogate this customer-originated route. ISP 2 ASN / /30 Link: /30 BGP Announce: /19 AS-path 900 Full BGP Routes / /30 ibgp Router / /30 Router2 Full BGP Routes Link: /30 BGP Announce: /19 AS-path 900 Multihomed Customer ASN 900 Prefix /19 Figure 3. Multihoming: PI-space Addresses, Single Site, Dual Routers, Two ISPs Example 3 Router Configuration Router 1: Dual-router, PI-space, Basic Config autonomous-system 900 static Null0 route for /19 ip route Null0 200 ip prefix-list ANNOUNCE description Our External Netblocks ip prefix-list ANNOUNCE seq 10 permit /19 ip prefix-list BOGONS description Bad Routes to Block In ip prefix-list BOGONS seq 10 deny /8 le 32 ip prefix-list BOGONS seq 15 deny /8 le 32 ip prefix-list BOGONS seq 20 deny /8 le 32 ip prefix-list BOGONS seq 25 deny /12 le 32 ip prefix-list BOGONS seq 30 deny /24 le 32 Page 14 of 14

15 ip prefix-list BOGONS seq 35 deny /16 le 32 ip prefix-list BOGONS seq 40 deny /3 le 32 Prevent someone else from announcing your own prefix(es) back to you, for security: update this with YOUR actual announced block(s) ip prefix-list BOGONS seq 1000 deny /19 le 32 ip prefix-list BOGONS seq 9999 permit /0 le 27 router bgp 900 no synchronization bgp log-neighbor-changes bgp dampening network mask neighbor remote-as 200 neighbor description BGP Transit to ISP-1 neighbor version 4 neighbor send-community neighbor soft-reconfiguration inbound neighbor prefix-list BOGONS in neighbor prefix-list ANNOUNCE out neighbor maximum-prefix define ibgp session neighbor remote-as 900 neighbor description ibgp to Router2 neighbor version 4 neighbor send-community neighbor soft-reconfiguration inbound make sure you are using a Loopback0 /32 on each router, and propagating this route internally. This is important to the stability of your ibgp sessions, so they are not tied to a physical interface address. neighbor update-source Loopback0 end Router 2: Dual-router, PA-space, Basic Config autonomous-system 900 static Null0 route for /19 ip route Null0 200 ip prefix-list ANNOUNCE description Our External Netblocks ip prefix-list ANNOUNCE seq 10 permit /19 ip prefix-list BOGONS description Bad Routes to Block In ip prefix-list BOGONS seq 10 deny /8 le 32 ip prefix-list BOGONS seq 15 deny /8 le 32 ip prefix-list BOGONS seq 20 deny /8 le 32 ip prefix-list BOGONS seq 25 deny /12 le 32 ip prefix-list BOGONS seq 30 deny /24 le 32 ip prefix-list BOGONS seq 35 deny /16 le 32 ip prefix-list BOGONS seq 40 deny /3 le 32 Prevent someone else from announcing your own prefix(es) back to you, for security: update this with YOUR actual announced block(s) ip prefix-list BOGONS seq 1000 deny /19 le 32 ip prefix-list BOGONS seq 9999 permit /0 le 27 Page 15 of 15

16 router bgp 900 no synchronization bgp log-neighbor-changes bgp dampening network mask define BGP session with ISP-2 (ASN 300) neighbor remote-as 300 neighbor description BGP Transit to ISP-2 neighbor version 4 neighbor send-community neighbor soft-reconfiguration inbound neighbor prefix-list BOGONS in neighbor prefix-list ANNOUNCE out neighbor maximum-prefix define ibgp session neighbor remote-as 900 neighbor description ibgp to Router1 neighbor version 4 neighbor send-community neighbor soft-reconfiguration inbound make sure you are using a Loopback0 /32 on each router, and propogating this route internally. This is important to the stability of your ibgp sessions, so they are not tied to a physical interface address. neighbor update-source Loopback0 end 3.4 Route Announcement Verification and Troubleshooting Congratulations, if you ve made it this far, you ve successfully gone through the basics of setting up BGP Multihoming Let s look at some tools that can help you determine if your BGP announcements are being correctly received by (1) your direct providers (from you), and (2) other providers in the Internet (from peering sessions with your providers). Many providers maintain either a web-based Looking Glass site or a BGP router that is publicly available for telnet, so that customers and ISPs can remotely verify BGP announcements using commands like show ip bgp x.x.x.x. After all, the only way for you to know if your routing announcements are making it out across the Internet is for you to be able to remotely check BGP tables in other providers networks. A good listing of these resources can be found at any of the following links: For you to properly verify that your routing announcements are being propagated via BGP out to the Internet, you should: 1. Ask your direct providers for access to Looking Glass URLs or public route views that are available. Verify what your prefix announcements look like within your provider using show ip bgp x.x.x.x. Look for both specific announcements (and verify from the AS-path that they originate from your ASN) as well as aggregate prefix blocks that are announced from you provider s ASN, in the case of PA-space. Do this for all of your transit providers. 2. Pick at least 2 or 3 other major providers in the Internet, and use their Looking Glass resources to verify that your routes are being correctly passed on to them. Determine if these providers are receiving all the available paths (e.g. one from each of your direct providers). Page 16 of 16

17 3.5 Real World Examples of Organizations that Multihome Using Looking Glass tools, and public WHOIS servers from ARIN, it is possible to see other enterprises and organizations that multihome. Here are a few examples, at the time of this writing, based strictly on this type of public information. Name (ASN) Alibris (ASN 20198) CollegeBoard (ASN 16919) Costco (ASN 11283) Ebay (ASN 11643) EDS (ASN 2165) FedEx (ASN 7726) Juniper Networks (ASN 14203) International Paper (ASN 16988) Paypal (ASN 17012) Schwab (ASN 6949) Wells Fargo (ASN 10837) Providers Used to Multihome UUnet and PBIS UUnet and InterNAP Single homed to InterNAP (Here s an example of a large enterprise who chose to forego multihoming, and chose a single provider who indirectly provides some of these benefits.) XO, C&W UUnet, Sprint Sprint and AT&T UUnet and C&W Sprint and AT&T UUnet and InterNAP AT&T, InterNAP, and UUnet AT&T and Qwest 3.6 More Complex Multihoming Examples We won t cover more complex examples of multihoming in this white paper, but we will mention some: Diverse Sites, Same Metro Area, Interconnected By MAN Link: example a mid-sized or large corporation with major on-line presence has a hot backup data center 30 miles away from its headquarters, on the other side of town. They wish to multihome to a different ISP at each location, and use the MAN link as a backup if either ISP goes down. Diverse Sites, Long Distance (e.g. East and West Coast), Interconnected By WAN Link: example the online e-commerce portal of a diversified conglomerate has some of its brands served out of its San Jose data center, and the rest served out of its Chicago business unit. The company already has good WAN capacity between the two sites for database backup purposes, and wishes to leverage this for Internet transit redundancy as well. Diverse Sites, Long Distance, Interconnected By Internet Tunnel or VPN Link: example a medium sized company in Los Angeles acquires another entity in New York. They wish to consolidate internet presence without additional WAN costs. Augmenting Your IP Transit With Settlement-Based or Free Peering: example an online gaming company experiences high growth in bandwidth requirements. But due to its popularity with broadband users, it is able to strike settlement-free peering deals at a neutral exchange point to route directly to customers of major DSL and Broadband Cable consumer companies. This saves it 40% in bandwidth costs in year 1. We also do not discuss some of the alternatives to doing BGP Multihoming: Multi-addressing: using a different subnet and static route for 2 or more providers, and using multiple addresses on web servers or load balancers to use these providers simultaneously. This usually involves some extended DNS functionality to be done right. Multiple links to the same provider, such as primary and backup links or two parallel load balanced links. Page 17 of 17

18 Links to multiple providers, but using multiple static routes or a 3 rd party solution (e.g. Radware LinkProof) to avoid full BGP routing. 4. Advanced BGP Techniques: Static Load Balancing The previous section provided examples of basic activation of BGP Multihoming to two providers. These examples work fine for basic failover and redundancy, but they do not address some of the critical questions of how to load balance traffic across the two links. The general point is that BGP is just another routing protocol it provides alternate paths, but does nothing by default to help you split your traffic sensibly between these paths. This section provides some configuration snippets and pointers for how to load balance traffic. We first take a look at outbound load balancing and then inbound load balancing, using standard BGP techniques. 4.1 Outbound Load Balancing When you receive two different BGP routes (one from each provider) to the same destination prefix, there is a well-defined BGP Decision Algorithm for picking which route is best 8. For standard ebgp routing, only ONE route can be active at any given time for a given prefix: 1. Do not consider ibgp path if not synchronized (this step usually disabled) 2. Do not consider route if there is no valid route to next hop 3. Pick the route with the highest weight (local to router) 4. Pick the route with the highest local preference (local to ASN) 5. Pick the route with shortest AS-path 6. Pick the route with lowest origin code (IGP < EGP < incomplete) 7. Pick the route with lowest MED 8. Prefer an ebgp route over an ibgp route 9. Prefer route with lowest next -hop metric 10. Tie breaker: route with lowest Router-ID is preferred. Assuming no special modifications, the usual first decision point that decides which provider to use for a route is the length of the AS-path. AS-path is the number of backbone hops, or administrative networks that must be traversed to get to the destination. Consider the following example: router# sh ip bgp BGP routing table entry for /24, version Paths: (3 available, best #3, table Default-IP-Routing-Table) Not advertised to any peer [route #1] [as-path] from ( ) Origin IGP, metric 10, localpref 100, valid, external Community: 2914: : :3000 [route #2] [as-path] , (received-only) from ( ) Origin IGP, metric 0, localpref 100, valid, external Community: 2914: : :3000 [route #3] [as-path] A good description and explanation of the BGP Decision Algorithm is provided at Page 18 of 18

19 (metric ) from ( ) Origin IGP, metric 10, localpref 100, valid, internal, best Notice that 3 BGP routes are received, and only the last one with AS-path is selected best, because the AS-path length is 4. This kind of behavior may work OK if you get service from two roughly equal-sized providers. But more often then not, your outbound traffic will not be well-balanced. Consider the case of using one large Tier1 provider at an expensive price, and one Tier2 provider at a much cheaper bandwidth price. By default, the Tier1 network will generally have a shorter AS-path length to Internet destinations, since it is global and closer to the core of the Internet (through better peering). So without modification, your outbound traffic will prefer the more expensive provider. Fortunately, there are tools to override default BGP behavior and define your own preferences to influence outbound traffic. The most common parameter used for this is BGP Local Preference, or localpref. One common approach relies on the fact that you can move a lot of your traffic just by changes routing preferences for a few of the largest ASNs, rather than worrying about setting preferences for many small ASNs. The topology shown in Figure 4 illustrates the Boulders and Pebbles concept: a small number of large networks usually account for the majority of your traffic. Small ISP Small ISP Small ISPs Internet International Medium ISPs Medium ISPs ISPs NTT/Verio (AS2914) Level(3) (AS3356) Medium ISPs Medium Regional ISPs ISPs Qwest (AS209) AOL (AS1668) Medium ISPs Medium ISPs Medium ISPs Small ISP Small ISP Small ISPs C&W (AS3561) Sprint (AS1239) UUnet (AS701) ATT (AS7018) ISP 1 ASN 200 ISP 2 ASN 300 Multihomed Customer ASN 900 Figure 4. Major Internet Destination ASNs Based on this observation, outbound load balancing can be achieved with the following steps: 1. Select a number of well-known large ASNs (excluding your directly attached providers). If possible, base this selection on actual traffic statistics. If not possible, expect to do some Page 19 of 19

20 experimentation with the steps below, and different ASN selections, until you achieve satisfactory results. 2. On the links you want to shift traffic to, use Cisco route-maps (or other vendors equivalent features) to match a subset of routes received from each provider, depending on whether one of these ASNs appears in the AS-path. 3. For these matching routes, increase the localpref to force these routes to be preferred. 4. Examine the resulting outbound traffic pattern, and add or delete more ASNs to shift outbound traffic until you are satisfied with the result. 5. Make sure to maintain this approach by periodically revisiting your choice of ASNs in step 1. Some ASNs may shrink over time, and others may grow (e.g. broadband providers in the U.S.) While there are more scientific ways to do outbound load balancing, using techniques such as Cisco NetFlow analysis, these techniques all require some up-front measurement infrastructure to be developed (or purchased) and deployed 9. In practice, the empirical approach presented here will probably be sufficient for many multihoming situations. Here is a recipe for this approach, based on the network diagram used in Figure 3 in the previous section. Note that the ASNs chosen for this example are U.S.-centric. For international situations, you will need to consider the major ASNs for the region in question. Router 1: Construct an AS-path access list to match major provider ASNs that appear anywhere in an AS-path (regular expression _ASN_) These are the provider ASNs that you wish to prefer through your link to ISP-1 (ASN 200). NOTE: when applying this example to your real-life situation, make sure that none of the ASNs below match any of your directly attached providers, or this will not work ASN 3561 is the main C&W ASN ASN 1239 is the main Sprint ASN ASN 7018 is the main AT&T ASN ASN 209 is the main Qwest ASN ASN 3356 is the main Level3 ASN ASN 7132 is the main SBC ASN ASN 6128 is the main Cablevision ASN ASN is a major Comcast ASN ip as-path access-list 51 permit _3561_ ip as-path access-list 51 permit _1239_ ip as-path access-list 51 permit _7018_ ip as-path access-list 51 permit _209_ ip as-path access-list 51 permit _3356_ ip as-path access-list 51 permit _7132_ ip as-path access-list 51 permit _6128_ ip as-path access-list 51 permit _22909_ Now create route maps, which apply these AS-path access lists. Remember, default localpref is 100, and higher means more preferred. 9 One popular off-the-shelf tool for integrated BGP and traffic analysis is from Adlex, although many route control products such as netvmg offer similar capabilities. If you do not mind using UNIX-based open source packages, take a look at a combination of cflowd ( an open-source NetFlow data collector) and flowscan ( a display and graphing tool for cflowd data). Page 20 of 20