Design, Implementation and Evolution of a DNS anycast resolving service in a country-wide ISP network

Design, Implementation and Evolution of a DNS anycast resolving service in a country-wide ISP network Kostas Zorbadelos OTE SA Senior Systems & Network Engineer GRNOG 1 June 5 2015

Presentation Outline OTE DNS Services Overview DNS Resolving Legacy Setup Network Topology Initial Anycast Setup (OSPF, IPv4) Current Anycast Setup (BGP, IPv4/IPv6) Improvements Future work Crazy ideas DNS Anycast Resolving - GRNOG 1 2

DNS services in OTE's network Authoritative DNS for company domains, customers and reverse IP space (~40K zones) Domain registrar for.gr DNS resolving for all customers in OTE's IP network (even customers with their own AS numbers and IP resources) DNS resolving for the infrastructure DNS Anycast Resolving - GRNOG 1 3

Presentation Outline OTE DNS Services Overview DNS Resolving Legacy Setup Network Topology Initial Anycast Setup (OSPF, IPv4) Current Anycast Setup (BGP, IPv4/IPv6) Improvements Future work Crazy ideas DNS Anycast Resolving - GRNOG 1 4

Legacy resolving service setup Will focus on the resolving services for the rest of the presentation Farms of machines behind load balancers in two physical DCs Clients were served two resolver IPs and directed to the primary and secondary locations Primary location basically a Single Point of Failure (SPOF) DNS Anycast Resolving - GRNOG 1 5

Legacy Setup DNS Anycast Resolving - GRNOG 1 6

Why service redesign? The services worked fine for many years Skepticism even inside the company for the service redesign There have been problems however in rare extreme cases Load balancer components EOL from the vendor DNS Anycast Resolving - GRNOG 1 7

Benefits from the service redesign No reliance on a single DC location for such a critical service geographic distribution DNS has all the characteristics for a service to be anycasted (connectionless over UDP or short-lived TCP connections, stateless query/responses) Many operators already operate anycast DNS services (with root operators the most prominent) No need for load balancer components, load balancing handled by the network routing layer Simplification of network setup DNS Anycast Resolving - GRNOG 1 8

Presentation Outline OTE DNS Services Overview DNS Resolving Legacy Setup Network Topology Initial Anycast Setup (OSPF, IPv4) Current Anycast Setup (BGP, IPv4/IPv6) Improvements Future work Crazy ideas DNS Anycast Resolving - GRNOG 1 9

Network topology We consider the typical core, distribution and access model The core has POPs in major cities (2 locations in Athens, Thessaloniki, Patra, Herakleio, Tripoli) with multi 10G interconnections The distribution layer performs layer 3 aggregation and has presence in many more locations (~40 POPs) The access layer (considering L3 access) is even more distributed Cisco gear in core, distribution, Juniper & Cisco in access DNS Anycast Resolving - GRNOG 1 10

The Core Main and only function to quickly forward packets Multi 10Gbps connectivity Contains the exit points to the global Internet and GRIX Not recommended (also by the vendor) to contain IP service nodes DNS Anycast Resolving - GRNOG 1 11

The Core (schema) DNS Anycast Resolving - GRNOG 1 12

Distribution / Access Distribution nodes aggregate access nodes Access nodes mostly terminate PPP sessions of users, but also contain leased line business customers Each access node has high availability backhaul connections to at least 2 distribution nodes Major project underway to simplify / consolidate access - distribution layers (BNG instead of BRAS nodes) Currently the user ratio is 40%/60% (BNG/BRAS) DNS Anycast Resolving - GRNOG 1 13

Distribution / Access example DNS Anycast Resolving - GRNOG 1 14

Anycast node placement With the above information on the topology the DNS anycast nodes were placed in few selected locations (POPs) at the distribution layer Too many locations at the access layer Not all distribution layer POPs suitable to host server equipment Easier access to nodes for the relevant operation teams an extra criterion DNS Anycast Resolving - GRNOG 1 15

Node placement (schema) DNS Anycast Resolving - GRNOG 1 16

Presentation Outline OTE DNS Services Overview DNS Resolving Legacy Setup Network Topology Initial Anycast Setup (OSPF, IPv4) Current Anycast Setup (BGP, IPv4/IPv6) Improvements Future work Crazy ideas DNS Anycast Resolving - GRNOG 1 17

Anycast node specs/choices Commodity Intel based machines (DELL R630/1 multicore CPU, 64GB RAM, 2x300GB disks, 4x1gbps NICs, DRAC remote management modules) Linux CentOS 6.x operating system Anycast IPs configured as loopbacks on the machines /30 point-to-point interfaces on each node IPtables on each node (host based security) BIND 9.x as nameserver No big departure from the legacy setup to also accommodate the operation teams and existing tools DNS Anycast Resolving - GRNOG 1 18

Anycast based on OSPF We are talking about anycast service inside a single AS (OTE's IP network) The initial approach was to use OSPF Each anycast node would participate in the network's IGP OSPF is the chosen IGP for IPv4 The belief was that the convergence time would be faster than other alternatives DNS Anycast Resolving - GRNOG 1 19

OSPF Topology DNS Anycast Resolving - GRNOG 1 20

Quagga config sample! Zebra configuration saved from vty! 2012/10/17 13:50:59! hostname <nodename>.otenet.gr password <vtypwd> enable password <enpwd> log syslog! interface em4 ip ospf network point-to-point ip ospf authentication messagedigest ip ospf message-digest-key 1 md5 <pwd> ip ospf dead-interval minimal hello-multiplier 3 ip ospf priority 0!! (continued next...)! (...continued) interface lo ip ospf cost 10 ip ospf priority 0! router ospf ospf router-id <server IP> log-adjacency-changes detail passive-interface lo network <anycast IP>/32 area X.X.X.X network <server IP>/30 area X.X.X.X! access-list MATCHALL permit any route-map IMPORT deny 10 match ip address MATCHALL! line vty DNS Anycast Resolving - GRNOG 1 21

Presentation Outline OTE DNS Services Overview DNS Resolving Legacy Setup Network Topology Initial Anycast Setup (OSPF, IPv4) Current Anycast Setup (BGP, IPv4/IPv6) Improvements Future work Crazy ideas DNS Anycast Resolving - GRNOG 1 22

DNS resolving dual stack There was a need to support IPv6 in the resolvers IPv6 connectivity / addressing was in place in OTE's IP network However, the IGP choice for IPv6 was ISIS No mature open source implementation of ISIS yet The initial plan was to proceed with static routes and tracking options (IOS IP SLAs) Input from RIPE 67 suggested we take a look at BGP for anycast IP injection DNS Anycast Resolving - GRNOG 1 23

BGP deployment We tested the idea to use BGP for anycast (/32, /128) route injection The anycast node makes an ebgp peering with the neighbouring distribution layer router Anycast nodes use a private AS number (65xxx) The anycast route is announced with no-advertise community The neighbouring router redistributes the anycast routes to OSPF/ISIS respectively DNS Anycast Resolving - GRNOG 1 24

BGP setup/choices There is a common approach for both IPv4/IPv6 Debugging is easier, setup is simpler There are various mature open source BGP implementations that can be used No convergence delay (convergence is faster in some cases!) We chose BIRD as the BGP daemon on the anycast server node DNS Anycast Resolving - GRNOG 1 25

BIRD config sample (1/2) /* BIRD DNS Anycast config bird.conf (bird6.conf is similar) */ log syslog name "bird" all; router id <routerid>; debug commands 9; protocol direct direct_lo { interface "lo*"; # Restrict network interfaces it works with } protocol device { scan time 60; } # Scan interfaces every 60 seconds (continued next...) DNS Anycast Resolving - GRNOG 1 26

BIRD config sample (2/2) (...continued) protocol bgp bgp_65xxx { description "DNS anycast BGP"; debug { states, routes, filters, interfaces, events }; local as 65xxx; neighbor <nei ip> as 6799; password "XXXXX"; # Password used for MD5 authentication } export filter { if proto = "direct_lo" then { # add no-advertise bgp_community = -empty-; bgp_community = add(bgp_community,(65535, 65282)); bgp_origin = 0; accept; } reject; }; import none; DNS Anycast Resolving - GRNOG 1 27

Presentation Outline OTE DNS Services Overview DNS Resolving Legacy Setup Network Topology Initial Anycast Setup (OSPF, IPv4) Current Anycast Setup (BGP, IPv4/IPv6) Improvements Future work Crazy ideas DNS Anycast Resolving - GRNOG 1 28

Operations Current setup contains 17 machines in 3 geo-locations for the customer resolvers (yes, we have overprovisioned!) Infrastructure is served by a different anycast cluster (2 machines, 2 geo-locations) All config is handled by ansible / mercurial VC (also blackholing of domains/eeep etc) The automation tools help us to test different resolving software Introduce software diversity DNS Anycast Resolving - GRNOG 1 29

Monitoring / measurements Each server has IPMI module accessible through the management LANs Monitoring / fault detection / alerting provided via a Nagios infrastructure already in place DNS measurements from BIND using cacti plugin Cacti / rrdtool for graph representations, also pnp4nagios Current set of tools adequate for operation and basic understanding of what is going on DNS Anycast Resolving - GRNOG 1 30

Next generation tools We can do much better in terms of measurements and visualization Real or near-real time visualization and analysis of DNS queries Anomaly detection and trend analysis Graphite/grafana seem very promising Use a big data approach for storage and analysis of the data using pcap captures DNS Anycast Resolving - GRNOG 1 31

Application load balancing There is a need for application load balancing using the DNS The most prominent application is IMS (SIP/VoIP) The clients whould be distributed in different SBC nodes in SIP signalling Currently we address this using BIND views Utilizing edns-client-subnet and gdnsd can produce a match better config for all applications with similar needs DNS Anycast Resolving - GRNOG 1 32

Authoritative setup We keep using a legacy setup for the authoritative service The relevant tools show their age (written about 10 years ago) IPv6 is not properly addressed DNS/IPAM integration for the reverse DNS (an issue for a seperate talk ;-) ) We should use the anycast approach there too We will start with the company/infrastructure domains DNS Anycast Resolving - GRNOG 1 33

New nodes' deployment The deployment of new nodes in the current setup is not an easy task Some nodes are under-utilized. Not all locations receive the same load We need to over-provision Would be great to introduce on-demand new nodes possibly from a few central cloud locations with VM setups We could also choose to deploy specific nodes to access routers Our BGP setup makes this possible, with a bit more complicated network setup (MPLS/VPN?) Rough initial idea in my head. If materializes another good presentation DNS Anycast Resolving - GRNOG 1 34

Questions? DNS Anycast Resolving - GRNOG 1 35

References http://en.wikipedia.org/wiki/anycast https://www.ripe.net/publications/docs/ripe-268 https://labs.ripe.net/members/romeo_zwart/new-archi tecture-for-k-root-local-nodes http://bird.network.cz/ http://www.afasterinternet.com/howitworks.htm CCIE material on IP routing (OSPF, BGP) DNS Anycast Resolving - GRNOG 1 36