Anycast Rou,ng: Local Delivery. Tom Daly, CTO h<p://dyn.com Up,me is the Bo<om Line

Transcription

1 Anycast Rou,ng: Local Delivery Tom Daly, CTO h<p://dyn.com Up,me is the Bo<om Line

2

3

4 Agenda What? Why? SeHng up Anycast Rou,ng Applica,ons EngOps Tips and Tricks Caveats

5 What s Anycast? Anycast is a packet rou,ng and delivery technique which primarily relies on Border Gateway Protocol (BGP) to provide homogenous TCP or UDP services to a distributed user base using a geographically distributed installa,on of systems sharing a common IP address.

6 Unicast The Internets

7 Anycast The Internets

8 How Can This Help You CDN- ize your dynamic applica,ons, reducing latency, more throughput Perform simple, low- latency, in- datacenter load balancing Automated datacenter failover and maintenance Contain DDoS a<acks to por,ons of your network, not the en,re network

9 Agenda SeHng up Anycast Rou,ng Applica,ons EngOps Tips and Tricks Caveats

10 Basic Concept Copy applica,ons and data across many datacenters using standard IP networking techniques Assign duplicate service IP (VIP) addresses to frontends in the DCs Inject the same route prefixes consistently to one or more ISPs

11 Basic Concept Anycast Traffic to VIP The Internets Anycast Traffic to VIP Unicast App and Data Replica,on

12 What does it take? Rou,ng need to be able to have stable rou,ng across transit ISPs for anycast services Applica,on Design need to run applica,on generally stateless, or sync state between datacenters Opera,ons EngOps needs to know about the network and applica,on work together!

13 Rou,ng Components Next- Hop / Sta,c Rou,ng BGP Border Gateway Protocol IGP Interior Gateway Protocol OSPF Open Shortest Path First ibgp interior BGP EMCP Equal Cost Mul, Path BFD Bidirec,onal Forwarding Detec,on Peering vs. Transit

14 Next Hop Rou,ng /24 *[BGP/170] 6d 01:59:13, MED 100, localpref 100 AS path: I > to via ge-0/1/0.0 [BGP/170] 6d 01:58:56, MED 100, localpref 100 Prefix Best Next Hop Path AS path: I > to via ge-0/3/0.0 [BGP/170] 3d 13:47:37, MED 100, localpref 100 AS path: I > to via ge-0/2/ /24 *[BGP/170] 2w4d 18:46:15, MED 100, localpref 100 AS path: I > to via ge-0/1/0.0 [BGP/170] 3d 13:47:53, MED 100, localpref 100 AS path: I > to via ge-0/2/0.0 Each network prefix or des,na,on is given a known next hop to reach the des,na,on.

15 GeHng Routed ISPs u,lize BGP to exchange rou,ng informa,on about their network with other ISPs ISPs are represented by an Autonomous System Number (ASN) ISPs use an IGP and sta,c rou,ng inside the network

16 Border Gateway Protocol Border Gateway Protocol (BGP) is used to exchange rou,ng informa,on between networks ibgp is another IGP protocol ignore for now ebgp only exports the best path to peered routers BGP doesn t know anything about link state

17 BGP Rou,ng TRANSIT ISP #1 AS ISP #2 AS PEERING TRANSIT

18 Interior Gateway Protocols An Interior Gateway Protocol (IGP), such as OSPF, ISIS, or RIP is used inside an ISPs network The IGP knows the cost or distance to use a next- hop / link ISPs use an IGP for TE in the backbone Generally, the shortest path, or least cost next- hop wins ISPs hot- potato traffic off of their networks to keep costs low

19 ISP Backbone and IGP Rou,ng R1 50 R2 10 R R6 10 R5 20 R4

20 IGP in a Backbone R1 50 R2 10 R R6 10 R5 20 R

21 Two ISPs: BGP and IGP R1 R2 R3 RA RB R6 R5 R4 RC RD

22 Two ISPs: BGP and IGP R1 R2 R3 RA RB R6 R5 R4 RC RD

23 Anycast to the Datacenter Use anycast rou,ng to a<ract traffic to local datacenters for a given IP prefix containing a service VIP Relies on your TRANSIT ISPs internal rou,ng policy to sanely send traffic to you Trying to use PEERING for this is harder, and not recommended for beginers

24 IGP in a Backbone w/ Anycast R1 50 R2 10 R R6 10 R5 20 R

25 Two ISPs: Anycast R1 R2 R3 RA RB R6 R5 R4 RC RD

26 Applica,ons Components UDP DNS and NTP Work very well, one packet in = one packet out Consistent packet handling TCP HTTP and HTTPS Also works well for shorter- ish lived connec,ons Backbone topologies don t change (ex. irc.dal.net) Long Live TCP Downloads, etc Think about it

27 Designing the Applica,on Data replica,on needs to happen on tradi,onal unicast IP addresses VIPs/Frontends are anycasted TCP state is ignored and possibly reset on re- routes Applica,on session data is reset on re- routes or completely shared (CAP)

28 Opera,ons Components Applica,on availability monitoring to mul,ple loca,ons Applica,on trend / load monitoring to mul,ple loca,ons Every needs to know a lot about networking (BGP, OSPF, ECMP, etc)

29 Availability Monitoring Tradi,onal availability monitoring doesn t work One single monitoring instance cannot monitor all anycast sites Monitor unicast VIPs and use distributed monitoring, anycast VIPs within the datacenters Nagios + NRPE2 for this

30 Trend Monitoring Data collectors running at each site, then stacked graphs back at the NOC.

31 Agenda Tips and Tricks Caveats

32 Tip #1: Selec,ng an ISP for Anycast ISP must have an internally coherent rou,ng policy Generally speaking, ISP must run a single global ASN for regions they cover ISP must have a consistent, reasonably designed interconnect policy <- Inves,gate deeply this can cause problems

33 Tip #2: IP Address Requirements Your network must be able to speak BGP You will want your own ASN You need IP address blocks, generally a /24 will work talk to ARIN You will probably want to setup a hold- down route from one datacenter i.e. /22 in case of flappage

34 Tip #3: No Backbone! You do not need a backbone to run anycast! One or two ISPs is fine You will need to permit your own ASN in your routers Juniper: autonomous-system loops 2; Cisco: allowas-in

35 Trick #1: Outages and Maintenance Anycast routes around datacenter outages and maintenance events AUTOMATICALLY If a site breaks, withdraw your BGP routes ISP BGP reconverges around your outage and passes traffic to new des,na,on automa,cally Beware of flapping your routes Before recovery resync applica,on and data!

36 R1 R2 R3 RA RB R6 R5 R4 RC RD

37 R1 R2 R3 RA RB R6 R5 R4 RC RD

38 Trick #2: Anycast within the Datacenter Anycast can be used within the datacenter for low latency, low cost load balancing To accomplish this, we use an IGP + ECMP IGP running on applica,on servers signals next hop availability to router Router sees mul,ple paths of same cost and load balances to them use L4 hashing

39 /24 OSPF Area 0 Passive Interface Loopbacks in OSPF

40 OSPF + ECMP Router speaking OSPF to internal network Quagga + OSPFD running on every box Service VIP is on loopback, announced into OSPF Router sees equal cost paths to next- hops, and load shares Routers can be configured to hash on 2- tuple (src / dst IP) or 4- tuple (IPs + Ports) If something goes wrong on a box, shut down quagga

41 show route extensive inet.0: destinations, routes ( active, 0 holddow Restart Complete /26 (1 entry, 1 announced) TSI: KRT in-kernel /26 -> { , } forwarding-table { export [ dynect-loadbalance ]; } policy-statement dynect-loadbalance { from { route-filter /26 exact; } then { load-balance per-packet; } } hash-key { family inet { layer-3; } }

42 Trick #3: Par,,oning DDoS Par,,on DDoS to the source area Anycast maps users in a common area to a common des,na,on if all a<acks in same area, only one area impacted If a<ackers well distributed, compute and memory resources are well distributed

43 The Internets

44 The Internets

45 Caveats Agenda

46 #1: Poor Network Planning in Tokyo Level(3) in US and EU No Level(3) in APAC No problem advanced local rou,ng in Asia via PCCW PCCW is a global network Routes leaked from Asia to US and EU via PCCW All users went to APAC

47 APAC US EU

48

49 Solu,on: Test Prefix We burn some IP prefixes for test rou,ng before site turnup We can verify rou,ng, firewall configura,on, and machine configura,on with this Useful for DDoS and load tes,ng Great for EngOps tabletop exercises

50 #2: Scaling OSPF Quagga got too heavy 16- way ECMP Service daemon crashes or machine failures result in OSPF topology changes Topology changes cause undue strain on router CPU Needed a new way to enumerate available next hops for ECMP

51 Enter OpenBFDD Open Source implementa,on of the BFD protocol in FreeBSD BFD is generally used for Ethernet Opera,ons and Administra,on (OAM) in metro ethernet networks But can certainly be used to enumerate available next- hops

52 /

53 /26 *[Static/5] 08:40:38 to via ge-1/3/0.20 > to via ge-1/3/0.20 route /26 { qualified-next-hop { bfd-liveness-detection { minimum-interval 500; multiplier 3; neighbor ; } } qualified-next-hop { bfd-liveness-detection { minimum-interval 500; multiplier 3; neighbor ; } } }

54 Using OpenBFDD daemontools runs bfd- beacon BFD adjacency router comes up, next- hop available, box gets traffic Separate monitoring process monitorings dbind, dunbound, Apache if error svc d bfd- beacon BFD adjacency down in < 1 second

55 h<p://github.com/dyninc/openbfdd

56 Ques,ons? Up,me is the Bo<om Line