renesys WHITE PAPER Practical BGP Security: Architecture, Techniques and Tools Laris Benkis A practical approach to identify weaknesses and highlight strategies for the detection of internetwork failures and defense of internetwork routing. Written for network engineers responsible for provisioning and protecting BGP, and for the IT manager who needs a technical overview of routing threats to the enterprise network.
At 12:48AM EST on the 14th of May 2003, the IP netblock 192.153.43.0/24 was announced to the via BGP through the service provider Verio s network from a small Florida ISP located in Boca Raton (1). This was peculiar because the block in question was registered to Northrop Grumman, a large US military contractor. Northrop Grumman has a very large IP network and its own autonomous system connected to the Sprint network. Therefore, anyone paying attention at the time might have concluded that something was amiss. Unfortunately for Northrop Grumman, no one was paying attention at least not until the spammers who had hijacked their IP block began using it to send email. As a consequence the IP block was listed in the SpamHaus (2) and SPEWS (3) spam blacklists. The hijacking and spam continued until complaints to Northrop Grumman alerted them to the hijack; they then took steps to reclaim ownership of their block and some two months after the start of the incident, the announcements were blocked. Sadly, the Northrop Grumman hijack was not an exceptional occurrence. The technique utilized by the hijackers was one of several standard methods used by spammers. They first identified an IP block which had been allocated by ARIN but was either not in use or not being announced on the. They then took ownership of the block by re-registering the DNS domain identified in Whois as the contact point for the block owner. Being able to send and receive mail from this domain allowed them to prove ownership of the block and request their upstream service provider to carry their announcements. This incident highlights numerous security weaknesses existing in the processes and technologies used to operate the. This paper will not review these weaknesses, but rather focus on the aftermath something bad has happened to your network on the from a BGP routing perspective; how do you recognize and respond to it? Better yet, what can you do to prevent incidents like this from happening, or at least minimize their impact? This paper discusses some BGP routing operational and security best practices that help to minimize network problems that can occur and a tool to help recognize and respond to some of the issues. We will start with a discussion of internal network configuration, presenting a sample network to illustrate some best practices and the issues they are designed to mitigate, next talk about network failures and problems that are visible externally to a network, and then discuss a detection method for these problems, and finally we will touch on some aspects of how to respond to incidents. We will not provide specific router configurations, but rather discuss at a higher level what the routers should be configured to do. A familiarity with BGP routing concepts is assumed on the part of the reader. The paper is written from the perspective of a mid-to large-sized autonomous system (AS) that peers with one or more transit providers and one or more private peers essentially an edge AS which may also have some smaller customers that it provides transit to. The routing security issues of a large transit provider are different enough that are not discussed in detail here. A simplified diagram of the target network environment is shown in the following diagram. Private Peers Transit Peers Customers 1
BGP Routing Principles An edge autonomous system, in this case, is a routing security domain with a perimeter defined by its interfaces to its EBGP peers. The peers are classified as transit, private, or customer. A large part of the operational and security challenge posed by BGP is controlling the flow of information across this boundary transmitting only that which you want to send, and accepting only what you want to receive. The information sent and received, of course, is BGP routes, and what you will accept and what you will send is very much dependent on the class of peer. An essential step in building a BGP network is to have a clearly defined routing policy. The following two tables indicate what classes of routes are typically sent and received from each class of peer; this defines a very simple routing policy for note that this is an example only and the recommendations that follow may not apply in every situation and therefore should be adjusted as required. Table 1 Sent to EBGP Peers Your AS Transit Private Peer Customer Transit Peers Private Peers Customers 1 1 sent to a customer AS can vary significantly depending on requirements and the agreement with the customer Table 2 Received from EBGP Peers Martian Address Space Unallocated Address Space Your AS Transit Private Peer Customer Transit Peers Private Peers Customers Martian address space consists of blocks that have special meaning which precludes their use in the public. The generally accepted list of Martian blocks is given in Table 3 on the next page. This table does not, as a general rule, change; therefore a Martian filter can be configured on routers and then left alone. Unallocated address space consists of address blocks which IANA has not yet allocated to a Regional Registry. Maintenance of the unallocated list by is complicated by the fact that changes are not under the control of. The list is updated as IANA allocates new blocks and they are routed publicly. As IANA allocates blocks, announcements are made on various network operator forums to let the operator community know that a new allocation has been made and that the appropriate filters should be modified. An organization that tracks the unallocated address blocks very closely is Team Cymru (4). They provide a current copy of the list in a number of useful formats. As an aside, for those readers interested in learning about BGP security best practices, a thorough review of the material on the Team Cymru website would be time well spent, in particular the secure IOS and JUNOS router templates. 2
Table 3 Martian Address Space Netblock Note 0.0.0.0/8 Default route and special-use 127.0.0.0/8 Loopback block 192.0.2.0/24 Net-Test allocated to network testbeds 10.0.0.0/8 RFC 1918 Private address space 172.16.0.0/12 RFC 1918 Private address space 192.168.0.0/16 RFC 1918 Private address space 192.18.0.0/15 RFC 2544 Network Testing 169.254.0.0/16 DHCP autoconfiguration 240.0.0.0/4 Class D (Multicast) and Class E (not used) for address space allocated to should never be accepted from EBGP peers any announcement of routes from an external peer indicates a network hijack. External BGP routes have a very high priority in route selection on a router. If a route for hijacked netblock were accepted in, it could seriously disrupt communication within in addition to the disruption already caused externally. Keeping this list up-to-date is relatively easy since BGP changes have to be made if the netblocks announced by change, so the filter list can easily be updated at the same time. Implementing a routing policy has two fundamental components. that are received into the AS have to be classified appropriately (routes received by an AS come from EBGP peers or are locally generated), routes that are sent have to be selected. The best tool available in BGP to denote a classification and to make a selection is the community tag. A community tag is a large integer number with no intrinsic meaning (except for few well-known tags with pre-defined values) which is attached to a BGP route. The network designer configures the BGP speakers in the network to announce or not announce routes based on the meaning assigned to their attached tags. Once routes have been tagged appropriately controlling their announcement becomes much easier, and inadvertent leaks of routes becomes less likely. So, for the example, let s define four tags: YA, TR, PP, and CU for generated, transit, private peer, and customer routes respectively. Now assume that has two transit peers, one private peer and one customer. Received routes would be tagged as follows: Table 4 Accepted Route Tagging YA TR PP CU Generated Transit Peer 1 Transit Peer 2 Private Peer 1 Customer 1 3
3. Provision Route Practical BGP Security: Architecture, Techniques and Tools Given a policy definition (Table 1) and an appropriate tagging of routes (Table 4), determining which routes to send to each external peer is straightforward. However, advanced routing strategies, such as load balancing and failover precautions, can complicate this process. In the interest of focusing on overall policy, we will disregard these advanced strategies. The interesting questions arise when trying to implement the policy in Table 2; on receipt of routes, how do you decide what you will accept and what will be rejected? More specifically, what prefixes should I accept from the peer, and what AS paths? The answer to this question involves tradeoffs which essentially balance security and maintainability. The problem boils down to the fact that the more specific the filters applied to a peering session, the more quickly they become outdated and require reconfiguring which can become an issue for large numbers of peering sessions and for smaller operations teams with limited manpower. This quandary leads very naturally to a discussion about automating this process. Updating route filters has several distinct steps, each of which must be addressed by the automation process. The filter information must be retrieved from a repository of some sort. Based on the filter information, a filter-list must be generated using syntax appropriate for your particular brand of router. Finally, the generated filter must be installed on the routers. The repository that contains route information is called a route registry. There are a number of registries operating on the, the best known being RADb (5) and RIPE (6). ASes which register in the registry list the netblocks which they originate as well as their customer ASes; their customer ASes, in turn, should also register and provide their relevant information. An AS which peers with a registered AS can retrieve this data and construct prefix and AS path filters that can be applied to the peering session. The filters will only allow the prefixes and AS paths of the peer AS and its customers. A useful tool that can query a route registry for the necessary data and construct the appropriate filter expressions for a number of different router platforms is RtConfig, and is a part of the IRRToolSet maintained by Systems Consortium (7). RtConfig can be used to periodically generate the necessary filters. The filters can then be applied manually or via some automated process. Another tool for producing and maintaining configurations based on routing registry data is irrpt (8). Note that Team Cymru maintains filter objects fltr-martian, fltr-unallocated, and fltr-bogons (the combination of fltr-martian and fltrunallocated) in the RADb and RIPE registries. This allows the martian and unallocated space to automatically be updated if a registry is used as a source to generate the filter. The following diagram illustrates the sequence that is followed for a private peer of to announce the prefix v.w.x.y/zz. 2. Retrieve Prefix Information Route Registry 1. Register Prefix Filter 4. Announce prefix 5. Validate announcement with filter Prefix v.w.x.y/zz Private Peer The above scenario, where automated processes generate and update router configurations to ensure optimal security, is a highly recommended operating method. However, it suffers from two problems. The first is that many networks do not keep up-to-date routing policy information in any registry. It is therefore impossible to keep updated filter configurations for those networks in any automated fashion. The second problem is that it requires considerable initial work to set up, and someone with scripting skills to implement. Many smaller 4
operations with few peering sessions to manage, or operations that lack the necessary skills, will often forgo the security benefits because of the complexity and work involved. To encourage networks to implement some measure of security in their peering configurations, the following simplified solution is recommended which, while not quite as thorough as automatic generation, still provides protection from a variety of significant threats. Filter Martian address space. The Martian list doesn t change and thus requires no maintenance. Filter space Consider filtering unallocated space Filter peer AS and peer customer AS paths. For example, given a private peer with AS# 65001 who has a customer AS# 65002, the following AS path filter template is a nice way of implementing this. The template allows the peer and the peer s customer to prepend their AS as many times as they like in their announcements. Since the actual prefixes themselves are not filtered, this filter does not require any subsequent maintenance. permit ^(_65001)+$ permit ^(_65001)+(_65002)+$ Implement prefix limiting. Called maximum-prefix on Cisco IOS and prefix-limit on Juniper JUNOS, this sets a ceiling on the number of route announcements that will be accepted from a peer. If the limit is exceeded, the session is disabled. Since accepted prefixes are not being filtered explicitly, this provides a mechanism to abort a session if a peer decides to re-advertise the full routing table as originated by itself. (Unlikely as it may seem, this has happened; see the recent Renesys presentation regarding the massive leak by Turkish Telecom [9]. Also, perusing the results of a Google search on AS7007 makes for an interesting read). The setting of the prefix limit should be based on how many prefixes the peer needs to announce. A generous margin of 50% above the current total allows room for growth while still providing protection from catastrophe. The above recommendation provides a low-maintenance configuration with reasonably robust security. Purists will not be impressed, but if all operators implemented at least this level of filtering, the autonomous system operational standards bar would be raised significantly. Failure Modes Let s now turn our attention to what can happen to from the perspective of the. The question of perspective is very important here what you see depends on where you are looking from. will announce a number of routes and the expectation is that all ASes on the will have visibility of these routes. While the reasons why this expectation may not be met are endless, the failure modes in terms of how route visibility does not meet this expectation can be summarized by a few distinct cases: 1. is not announcing all the routes that it should. This can result in parts of being unreachable from the or, if a covering aggregate exists for the missing prefix, traffic may be entering by a different path than was planned. Recall that morespecific prefixes are preferred by the BGP route selection process. This may overload circuits and impact costs by shifting traffic unexpectedly between transit providers. An example on the next page illustrates this. has a large server farm addressed in the 2.0.1.0/24 address block. This block is announced on an OC48 peering link. The rest of encompasses the entire 2.0.0.0/8 block, which is served by an OC3 peering link. If for some reason the 2.0.1.0/24 announcement is blocked, all the traffic to the server farm will follow the 2.0.0.0/8 announcement and potentially saturate the OC3 connection. Although the example is simplistic and contrived, it illustrates the point that announcement failures that may not be visible internally can cause unexpected and potentially serious traffic shifts. 5
Lots of Traffic OC3 2.0.0.0/8 OC48 Server Farm In 2.0.1.0/24 2.0.1.0/24 2. is announcing more routes than it should. This can indicate a configuration error in, or it may be leaking private peer routes unintentionally. In this example private peers with some other AS and accidentally leaks the peer s prefix. In this example the private peer transit link is an OC48 while only has an OC3. peers via a DS3. The extra traffic drawn by the leaked prefix will potentially saturate the private peering link and the transit link. OC48 3.0.0.0/8 Private Peer AS OC3 Lots of Traffic 3. routes are resulting in traffic on unintended paths. This can indicate a configuration problem in which you are announcing certain routes to transit providers which you did not intend to. It could also indicate that a private peer or a customer is leaking routes to their transit provider. This example is the reverse of the previous. In this case your private peer is leaking your prefixes and drawing unintended traffic. Resulting link saturation will impact both the private peer s customers, the customers in the leaked prefix, and the customers accessing the Private Peer network. Lots of Traffic OC3 Private Peer AS OC48 2.0.0.0/8 6
4. Some portion of the cannot see some or all of routes. This can be indicative of a failure at some point on a remote AS, where such a failure could be a malfunction or a configuration issue. This example is very common. Somewhere on the, a failure occurs. The failure is in some AS on which other ASes depend to reach, therefore is unreachable for some portion of the. The rest of the Very Important AS 2.0.0.0/8 5. routes are visible originating from some AS other than. This indicates a network hijack. Intentionally or unintentionally, another network is announcing routes. This failure mode describes what happened to Northrop Grumman. The effects can vary depending on the network block that is hijacked. In the Northrop Grumman case, an inactive block was used and consequently Northrop Grumman was blacklisted. If the hijacker were to announce a block that is actively in use in, communications from to the originating within that block could be partially or totally disrupted. If does not filter its own address space on announcements received from its transit provider, communication within could also be disrupted. 2.0.0.0/8 2.0.0.0/8 HijackingAS The final case is not, strictly speaking, an issue of visibility of routes: 6. autonomous system number is announced by some other network. Since it is not possible to know that the false announcement is actually a separate network, to the this would appear that has some other transit peering other than the legitimate ones, and may be announcing prefixes which are not part of the assigned address space. From a detection perspective, this then becomes a special case of failure mode #3 and possibly #2. 7
Depending on the routes announced by the hijacker, the impact on will vary. If address space is announced, then the impact is similar to that of case 5. If routes outside of address space are announced then there would be no disruption to operation. However, any actions taken by the hijacker would then be attributed to and reputation will suffer accordingly. AS #XXX AS #XXX HijackingAS The total number of distinct failure modes is not that large. The difficulty is that detecting these kinds of failures from the perspective of can be very difficult. Many networks have no external visibility of their status and simply rely on customer complaints to alert them to problems. While it doesn t require any financial outlay, this approach has the disadvantage of not providing complete coverage of all the failure modes; moreover, customer dissatisfaction can prove costly to the network operator in the long run. An alternate approach, a service called GRADUS offered by Renesys Corporation, provides a cost-effective means of detecting occurrences of these error conditions in real-time, allowing an operator to react quickly to correct them before customers are impacted. In many cases the failures causing traffic shifts are caused by network maintenance or changes performed during change windows when the network is operating at a low utilization; these traffic shifts may not be sufficient to saturate circuits at the time, and therefore if they can be detected immediately they can be corrected with no customer impact. GRADUS GRADUS is a real-time BGP monitoring system, connected (at the time of writing) to 100 BGP peers in autonomous systems scattered around the globe. The peers provide Renesys a comprehensive real-time view of the global routing tables. A GRADUS customer installs a Java client on a workstation in their network which connects to Renesys via a secure HTTPS connection. If GRADUS detects any of the alarm cases described previously for prefixes that the customer specifies, the client will generate a visible alarm and optionally send an SNMP trap to a network management station (NMS) and an email notification to a list of target addresses. Renesys Peer GRADUS Client Renesys Peer Renesys/GRADUS Renesys Peer Renesys Peer NMS Route Announcement Renesys Peer Gradus Notification SNMP Trap Email Notification 8
The diversity of the GRADUS peers provides a comprehensive view of how the sees, and the alarms alert a network operator the instant a significant event happens. This provides continuous real-time verification of the correctness of the BGP routing design and external surveillance for problems not visible to. In the case of Northrop Grumman, with GRADUS they would have known within seconds of the event that their netblock had been hijacked. Fast action then could have prevented any significant amount of spam from being released into the, and their being blacklisted. In addition to its monitoring capability, GRADUS has an archive of all BGP announcements it has received since January of 2002. The Java client allows this database to be queried, making it a powerful tool to debug network problems, analyze failures, evaluate potential transit providers, conduct forensic network investigations, and a host of other network analysis tasks. The data is displayed graphically, and as a raw listing of BGP announcements. The preceding image shows a depiction in GRADUS of the Northrop Grumman hijack. Each horizontal line represents a different GRADUS peer, a green mark indicates a route change with no loss on connectivity, and a red mark indicates a loss of connectivity. The raw announcement and withdrawal data provide much more detailed information. We can see that all peers first received an announcement for the route at 12:48AM on the 14th of May, the AS path from each peer to the origin AS of the netblock, and all the AS path perturbations that occurred at each peer until the route was finally withdrawn. If we were investigating the incident, the data from peer TS would certainly be interesting, as this peer kept seeing routes to the netblock until the 18th of July, while it was withdrawn from all other peers on the 26th of June. This is an example of the benefit provided by the diversity of perspective that GRADUS offers; a cursory look from an looking glass or route server would probably not have revealed that after the 26th of June the route was still active. 9
normal route to Yahoo SCD via Yahoo US backbone AS 10310 66.94.224.0/20 26101 Yahoo! 66.94.230.0/24 10111 AS for DataOne (Malaysia) 4788 TMnet, Telekom Malaysia 4436 Santa Cruz Community (scruz-net) 19151 WV FIBER LLC 3491 CAIS 5511 France Telecom 1239 Sprint Ex3 gradus://3fa1b5afe5e43bf0 hijacked routes originate in DataOne Malaysia (AS 10111) In the Northrop Grumman case, an unused prefix was hijacked. Hijackings of announced, in-use prefixes (and more specifics of those same prefixes) has also occurred many times on the. This is ocassionally a mistake caused by someone misconfiguring a prefix in a router. It may also be malicious, where someone tries to redirect or block legitimate traffic to another s networks. The following image shows the GRADUS graphical depiction of the hijacking, in May, 2004, of Yahoo! s Santa Clara datacenter prefix. The same two prefixes, 66.94.224.0/20 and 66.94.230.0/24, are shown being originated by both Yahoo! (AS26101) and DataOne from Malasia (AS10111). The diversity of peers of the GRADUS system allows accurate detection of such an event anywhere in the world. The graphical interfaces makes it easy to diagnose such an event. Incident Response Thus far we have discussed some best practices that can be applied to the building of a BGP network, some things that can go wrong, and how you can recognize that they have gone wrong. To round out the discussion, we will review briefly what you can do if you discover a problem similar to those described here. The first question that you need to answer is, From where is the problem originating? If the source of the problem is, then correcting it is a matter within your control and standard troubleshooting methods apply. If, however, the problem is external to, then there are no local corrective measures that you can apply to solve the problem. In this case, after using a tool such as GRADUS to identify the source AS of the problem, the best course of action is to contact the NOC of the AS to alert them to the situation. It may seem self-evident, but it is worth mentioning nonetheless, that most network operators provide a contact email and phone number. However, getting through to a knowledgeable operator may not always be easy. One way to get directly to the operations centers of many network operators is the INOC-DBA Hotline Phone system (10). INOC-DBA (Inter-NOC Dial-By-ASN) is a closed VoIP network running on the using SIP phones. Dialing an ASN with an optional 3-digit extension will ring a phone in the NOC at the AS if the network in question is part of the system. Accounts on INOC-DBA are free. 10
Conclusion This paper has suggested some of the guidelines that a network operator should follow in the design of a BGP network to simplify the implementation of routing policies. We then discussed the types of failures a network is subject to which are visible on the but not readily apparent within the AS. We then reviewed how the GRADUS tool can help to detect and respond to these kinds of problems. Laris Benkis is an internetworking consultant based in Montreal, Canada. He can be reached at laris@tpn.cc. References (1) http://www.completewhois.com/hijacked/ (2) Spamhaus SBL9228. http://www.spamhaus.org/sbl/ (3) http://www.spews.org/html/s2624.html (4) http://www.cymru.com (5) http://www.radb.net (6) http://www.ripe.net (7) http://www.isc.org (8) http://irrpt.sourceforge.net (9) http://www.renesys.com/news/renesys-nanog34.pdf (10) http://www.pch.net/inoc-dba 11
renesys CORPORATE HEADQUARTERS 1750 Elm Street Suite 101 Manchester, NH 03104 T +1.603.643.9300 F +1.603.623.1623 www.renesys.com sales@renesys.com 2005 Renesys Corporation. All rights reserved. Renesys and GRADUS are trademarks of Renesys Corporation.