BGP Tutorial John S. Graham
What s In Store? Routing Architectures BGP Basics Path Selection (many examples) BGP State Machine Multiprotocol Extensions Multicast Routing Covered Layer-III VPN Omitted
Cisco Architecture IP Routing Table 192.168.0.0/16 Metric: Next-Hop: 172.16.0.0/12 Metric: Next-Hop: 10.0.0.0/8 Metric: Next-Hop: BGP EIGRP Static OSPF RIP Export Import
Juniper Architecture BGP IS-IS OSPF 172.16.0.0/12 Metric 1: Metric 2: Next Hop: Area: AS-Path: Community: Level: Static Export Import RIP
BGP Schematic (IPv4 Unicast) 192.168.0.0/16 inet.0 < > BGP Table <> Routing Table < > < 172.16.0.0/12 BGP Path Selection Routing Policy
The Autonomous System Collection of routers under one administrative control Single internal routing protocol Identified using an AS Number AS100 AS200
AS Numbers An ASN is a 16-bit number 1 through 64511 are assigned by RIRs 64512 through 65534 are for private use and should never appear on the Internet Numbers 0 and 65535 are reserved AS 23456 used to represent 4-byte ASN to routers unable to handle the new standard All major routing platforms now support 32-bit ASN: http://www.get4byteasn.info/ Interesting contrasts between European and USbased R&E networks
Test 4-Byte ASN 39322 Operated by Force10 Networks Advertises 64.127.137.0/24 Accessible via Internet2 CPS Check the AS-Path on your network! 4-Byte ASN can be expressed as: AS-Plane = 39322 AS-Dot = 153.154 Thanks to Brent R. Sweeny for advice
ISP Routing Components Interior Gateway Protocol (IGP) Internal BGP (ibgp) Routes customer prefixes around internal infrastructure Is NOT congruent with physical connectivity External BGP (ebgp) Prefix interchange with customers Most routing policy located here
Routing Components Depicted Circuit A.4 Router A A.0 A.1 IS-IS Adjacency ibgp Peering ebgp Peering D.4 Router D D.0 B.0 B.1 Router B D.3 B.2 C.3 Router C C.0 C.2 Router E E.0
Attribute Classes Well-Known Optional Mandatory Discretionary Transitive Non-Transitive AS_PATH LOCAL_PREF COMMUNITY MED NEXT_HOP ATOMIC_ AGGREGATE AGGREGATOR CLUSTER_LIST ORIGIN ORIGINATOR_ID MP_REACH_NLRI MP_UNREACH_NLRI
The AS_PATH Attribute 172.16.0.0/12 200 100 172.16.0.0/12 100 172.16.0.0/12 300 200 100 AS100 detects its own AS in the path from AS500 and ignores the prefix 172.16.0.0/12 500 400 300 200 100 172.16.0.0/12 400 300 200 100
Which is the Better Path?
The NEXT_HOP Attribute B.2 C.2 C.3 D.3 B.1 A.1 Router SELF = NO NEXT_HOP SELF = YES A B A.1 A.1 C A.1 B.0 D C.3 C.3
The LOCAL_PREF Attribute Graphic used with kind permission of Philip Smith, Cisco Systems
Multi-Exit Discriminator Graphic used with kind permission of Philip Smith, Cisco Systems
MED: Metric Confusion MED is non-transitive and optional Some implementations send learned MED to ibgp peers by default and others do not Some implementations send MEDs to ebgp peers by default while others do not Default metric varies by vendor No explicit metric implies 2 32-1 or 2 32-2 No explicit metric implies zero
ebgp vs ibgp 1. Uses AS_PATH for loop avoidance 2. NEXT_HOP is modified 3. Peers must be directly connected 4. MED is reset 5. LOCAL_PREF is never advertised 1. Chinese whispers prohibited 2. NEXT_HOP is unchanged 3. Peers not necessarily directly connected 4. MED is propagated 5. LOCAL_PREF always advertised
Internal Peering Topologies Daisy Chain (Wrong) Full Mesh (Allowed) Route Reflector (Allowed)
How Are Prefixes Passed Around? On any given router only the best path for a prefix is passed to other peers Best path learned via ebgp Advertised to all other ebgp peers Advertised to ibgp peers Best path learned via ibgp Advertised onto ebgp peers Not advertised to other ibgp peers
The Golden Rule Never redistribute routes from the IGP into BGP Never redistribute routes from BGP into the IGP
Best Route Selection Longest prefix always wins regardless of routing protocol Source of routing information Connected > Static > ebgp > {IGP} > ibgp BGP ignores received prefixes if There is no route to the NEXT_HOP The AS_PATH contains the local AS number Not synchronized (Cisco IOS only)
BGP Path Vector Algorithm 1. {Highest WEIGHT} 2. Highest LOCAL_PREF is Preferred 3. Shortest AS_PATH 4. Lowest ORIGIN Code 5. Lowest MED 6. Prefer prefix received from ebgp peer over ibgp peer 7. Path with lowest metric to NEXT_HOP (aka Metric2) 8. Lowest ROUTER_ID 9. Shortest CLUSTER_LIST 10. Lowest neighbor IP address
Interior Gateway Protocol (IGP) Can be OSPF or IS-IS Prefer IS-IS Doesn t require IP or ANY Layer-III (e.g. CLNS) protocol to work Routes IPv6 with no more effort than IPv4 Routes ISP infrastructure addresses: Router /32 loopback Point-to-point backbone /30 or /31 subnets (Customer assigned link networks) (IXP assigned address for public peerings) Other infrastructure addresses such as management networks Simple and invariant configuration compared with an enterprise network Adjacencies follow physical backbone connectivity Non-backbone interfaces run IGP passively Metric Based on route miles Can be adjusted for traffic engineering purposes Complex sub-divisions (areas, levels) unnecessary Principal job of IGP is determining least-cost path between any two routers
Static ibgp; Dynamic IGP Within Internet2: 1. Next Hop for customer prefixes in RIB provided by ibgp and does not change if backbone circuit fails. 2. Entry in FIB provided by IS-IS and depends on backbone connectivity.
Route Reflection KANS CHIC ATLA HOUS LOSA NEWY SALT SEAT WASH
Route Reflection Rules Prefix received from a client Reflect to all client peers apart from the sender Reflect to all non-client peers Prefix received from a non-client peer Reflect to all client peers
Loop Prevention for RR ibgp speaker receives a prefix with the ORIGINATOR_ID attribute equal to its own ROUTER_ID Route reflector receives a prefix with the CLUSTER_LIST attribute containing its own CLUSTER_ID
Progress of Route Reflection 1. External Prefix advertised to both RR 2. Prefix reflected to all client peers 3. Prefix passed between RR but ignored due to CLUSTER_LIST attribute
ibgp Tracks the IGP Metric 278 978 2363 2363 3019 3932 4068 WASH ATLA HOUS KANS SALT SEAT LOSA 1000 128.122.0.0/16 128.122.0.0/16 BUFF CHIC NEWY NEWY 1000 LOSA SEAT SALT KANS HOUS ATLA WASH 3212 2932 2019 689 1507 1045 905
NREN and Internet2 Metric = 2932 Metric = 2019
Possible Layer-II Connectivity Internet2 and NREN routers directly connected Connection through a VLAN on a single peering switch Connection via VLAN that traverses multiple Layer-II and optical devices
Internet2 Connection at NGIX-W GE OC192 VLAN 166 10GE 10GE OC192 10GE VLAN 166 OC192
Prefer the 10G-Connected Path Internet2 router in Seattle Associate a high LOCAL_PREF with prefixes received on external peering with NREN Internet2 router in Salt Lake City Receives NREN prefixes over ibgp peering with Seattle with high LOCAL_PREF Now prefers ibgp to SEAT over ebgp via NGIX-W
Boomerang Prefixes CHIC NEWY GLBX GPN KANS 161.46.0.0/16 RRMA RRMA
Boomerang Prefixes 2/2 Loop is prevented by AS_PATH attribute Suppressing boomerang prefixes: Advertise the prefix to the echoing peer Attach NO_EXPORT community to transiting peer Request echoing peer to filter their outbound (sub-optimal)
Redundant Connections Multiple Peerings with Internet2 Set the MED on prefixes sent to I2 Use communities to change the LOCAL_PREF on I2 Allow both peerings with I2 to float Advertise prefixes both directly to I2 and through another RON
Use of MED: Internet2 & ESNet 1000 128.175/16 [0] 128.175/16 [1001] 278 128.175/16 [1000] 128.175/16 [278] 905
ESNet Send MEDs 1000 131.225/16 [1] 131.225/16 [1000] 278 131.225/16 [0] 131.225/16 [905] 905
Bidirectional MED Causing Asymmetric Routing Fermi UPenn UPenn Fermi
What If I2 Doesn t Send MEDs? ESNet use their IGP Metric Traffic remains on ESNet backbone only until it reaches a router with an I2 peering. ESNet use LOCAL_PREF on One Peering A single (congested?) egress from ESNet for all traffic to Internet2-connected destinations Different I2 Prefixes Receive High LOCAL_PREF at Different Locations Same effect as MEDs, but implementation far more manual and complex
Introducing Prefixes Into BGP 1. Use network statement 1. With auto-summary disabled 2. With auto-summary configured 2. Configure aggregate routing 3. Use route maps to redistribute 1. Prefixes learned from an IGP 2. Static routes
1.1 The network Statement (Cisco) router bgp 87 no auto-summary network 129.79.0.0! ip route 129.79.0.0 255.255.0.0 Null0 200 1. There is no mask following the prefix in the network statement as we are advertising a classful network 2. The static route serves two important purposes: 1. The prefix will not be advertised by BGP unless there is an exact match in the IP routing table 2. Traffic sent to non-existent IP addresses in the range will be silently dropped. This avoids wasting b/w sending ICMP Unreachables and is a valuable defense against scanners and DDoS attacks.
1.2 Using auto-summary router bgp 87 auto-summary network 129.79.0.0 (Cisco) 1. The prefix will be advertised by BGP providing there is at least one contained prefix in the IP routing table 2. This IGP-learned prefix can be any length; it does not have to match the classful network that BGP is being asked to advertise 3. Use this command: show ip route 129.79.0.0 255.255.0.0 longer-prefixes to check whether there is an IGP-learned route. If there are none, then BGP will not advertise the 129.79/16 parent
3.2 Redistribute Static (Juniper) policy-options { policy-statement ORIGINATE { term Seed { from { protocol static; route-filter 129.79.0.0/16 exact; } then accept; } } } routing-options { static { route 129.79.0.0/16 discard; } }
Aggregation Scenario Internet2 or NLR 10/8 Regional Network 10.100/16 10.200/16 10.300/16 Universities
Prefix Leaking Problem router bgp 400 network 10.0.0.0 neighbor 192.168.1.1 remote-as 100 neighbor 192.168.1.5 remote-as 200 neighbor 192.168.1.9 remote-as 300 neighbor 192.168.1.13 remote-as 500! ip route 10.0.0.0 255.0.0.0 Null0 200 Router D
(Faking) Aggregation Use the NO_EXPORT Community Not the best choice as the problem and its solution reside in different domains Filter outbound prefixes to NLR/Internet2 A straightforward robust solution Deploy route aggregation Downstream problems could cause routing flaps on peering with Internet2
Configuring Aggregation (Cisco) router bgp 400 neighbor 192.168.1.1 remote-as 100 neighbor 192.168.1.5 remote-as 200 neighbor 192.168.1.9 remote-as 300 neighbor 192.168.1.13 remote-as 500 aggregate-address 10.0.0.0 255.0.0.0 summary-only Router D
Configuring Aggregation (JunOS) routing-options { aggregate { route 10.0.0.0/8 discard passive community 65535:65281; } } policy-options policy-statement ORIGINATE { term AGGREGATE_to_BGP { from protocol aggregate; then accept; } } protocols bgp group ISP { export ORIGINATE; peer-as 400; neighbor 192.168.1.13 { } }
Global NOC Recommendation Use a network statement to originate ARIN allocations to Internet2 Configure a supporting static route Disable the auto-summary capability Filter more specific contained prefixes using an outbound route-map applied to the peering with Internet2 or NLR
Steering Inbound Traffic (1/2) [210 11537 19782 87 I] [19401 19782 19782 19782 87 I] 129.79/16 [19782 87 I] 129.79/16 [19782 19782 19782 87 I]
Steering Inbound Traffic (2/2) LOCAL_PREF = 100 LOCAL_PREF = 200 129.79/16 129.79.9/24 129.79/16
Global Routing Flap Analysis 1/3 jsg@rtr.ll-re1> show route aspath-regex ".* 47868.* " table commodity.inet.0 commodity.inet.0: 284903 destinations, 486789 routes (284902 active, 0 holddown, 5 hidden) + = Active Route, - = Last Active, * = Both 94.125.216.0/21 *[BGP/170] 05:29:58, localpref 200, from 149.165.255.64 AS path: 4323 3257 25512 47868 I > to 149.165.254.25 via ge-0/1/0.1, label-switched-path LL640Lo0.0- >CTC640lo0.0 to 149.165.254.29 via xe-2/0/0.111, label-switched-path LL640Lo0.0- >CTC640lo0.0 [BGP/170] 16:42:14, MED 208, localpref 200 AS path: 1239 3257 3257 25512 47868 I > to 144.228.154.165 via so-1/0/0.0 47868 25512 3257 4323
Global Routing Flap Analysis 2/3 3356 47868 29113 3257 255*[47868] 1299
Global Routing Flap Analysis 3/3 Graphics used with kind permission of Renesys and Earl Zmijewski
Functions Served by Communities 1. Assign prefixes to pre-defined groups (local significance only) 2. Control how prefix is advertised by peer 3. Control your peer s LOCAL_PREF for the specific prefix 4. Signal peer to prepend multiple AS numbers to AS_PATH 5. Blackhole all traffic to specific prefix
Expressing A Community JunOS & Cisco New Format 11537:260 Cisco Old Format 756089092 1. A community is just a 32-bit number 2. By convention, the most-significant 16 bits represent an AS number 3. To convert from new to old formats 1. Multiply the high 16-bits by 2 16 2. Add the low 16-bits to the result
BGP Communities on Internet2 Classify prefixes Directly connected participants Sponsored SEGP Adjust Internet2 LOCAL_PREF Request Internet2 to black-hole a prefix (Prevent ISP from advertising prefix to specified upstream peers)
Well-Known Communities NO-EXPORT 65535:65281 Do not advertise to any ebgp peer NO-ADVERTISE 65535:65282 Do not advertise to any peer NO-EXPORT- SUBCONFED 65535:65283 Do not advertise beyond local AS (confederations only) NO-PEER 65535:65284 Do not advertise to bi-lateral peer
Using Communities (1/5) 128.169.0.0/16 141.225.0.0/16 192.55.208.0/24 +14048:10 +14048:10 96.4.0.0/15 192.55.208.0/24 128.169.0.0/16
NO-EXPORT Community 172.16.0.0/12 172.16.1.0/24 +NO-EXPORT 172.16.0.0/16 172.16.0.0/12 172.16.2.0/24 +NO-EXPORT
Security Diversion Routing Table Interface Inbound Packet A C L Null 0 Routers are optimized for packet forwarding; not packet filtering Routing to Null saves valuable CPU cycles
Customer-Triggered Blackhole Inbound Prefix Static Route 11537:911 No 11537:911 Yes Prefix > /24 Prefix > /24 Yes Discard No Forward Redistribute Into BGP ISP Customer
Customer-Triggered Blackhole (ISP interface Null0 no ip unreachables Perspective; Cisco IOS) ip policy-list BLACKHOLE permit match ip address prefix-list 24_TO_32 match community 10! ip community-list 10 permit 756089743!! ip prefix-list 24_TO_32 seq 5 permit 0.0.0.0/0 ge 24!! route-map CUSTOMER_IN permit 10 match policy-list BLACKHOLE set community no-export set interface Null0
Customer-Triggered Blackhole (Customer Perspective; JunOS) routing-options { static { route 192.168.17.15/32 { discard; community 11537:911; } } } policy-options { policy-statement ORIGINATE { term BLACKHOLE { from { protocol static; route-filter 0.0.0.0/0 prefix-length-range /24-/32; community BLACKHOLE; } then accept; } term { } } }
Customer-Side Policy: IOS vs JunOS Juniper Blackhole prefix statically routed to Discard Attach a community tag to the static route Cisco IOS Blackhole prefix statically routed to Null0 Add the new prefix to the blackhole prefix list Existing route-map Redistributes blackhole prefix list into BGP Attaches the correct community
Recommended Routing Policy Should be implemented Reject any prefix with a private AS in the AS_PATH Reject bogon prefixes (following slide) Consider implementing Assign higher LOCAL_PREF to Internet2 or NLR prefixes than to commodity. Max prefixes limit on some peers
LOCAL_PREF Gotcha LOCAL_PREF = 200 149.165.128.0/17 149.165.128.0/17 Indiana Gigapop 149.165.240.64/26
Bogon Prefixes Prefix Reason RFC 0.0.0.0/0 Default 10.0.0.0/8 172.16.0.0/12 Private 1918 192.168.0.0/16 127.0.0.0/8 Loopback 169.254.0.0/8 Link Local 3330 192.0.2.0/24 IANA Reserved 192.88.99.1/32 6 to 4 relay 198.18.0.0/15 Network device benchmarking 2544 224.0.0.0/4 Multicast group addresses 3171 240.0.0.0/4 Class E addresses
BGP Messages Type Description References 1 Open RFC 4271 2 Update RFC 4271 3 Notification RFC 4271 4 Keepalive RFC 4271 5 Route-Refresh RFC 2918
The BGP Update Message Unfeasible Routes Length Withdrawn Routes Length Prefix P BGP Header Total Path Attributes Length Type Data Path Attributes Length M Value NLRI Length Prefix N N >= M
BGP State Machine Established Open-Sent Open-Confirm Connect Active Idle
BGP Convergence Scenarios Both BGP processes immediately transition from Established to Active The router connected via the unaffected circuit blackholes traffic for up to 90 seconds
Anatomy of Brief Outage 1. Link between ESNet and MANLAN Goes Down 2. ESNet router sends NOTIFICATION which is not received 3. Peering on Internet2 remains Established even though ESNet side is Down 4. Link between ESNet and MANLAN is restored before KEEPALIVE timer expires on Internet2 5. ESNet router negotiates new TCP virtual circuit with Internet2 router 6. The peering on Internet2 resets
Source-Specific Multicast (S,G) Join (S,G) Join S G IGMPv3 Join G R
Multicast Routing Unicast traffic AS400 <> AS200 Multicast traffic AS400 <> AS200 Unicast and Multicast IPv4 Prefixes Unicast IPv4 Prefixes