How NOC manages and controls inter-domain traffic? 5 th tf-noc meeting, Dubrovnik nino.ciurleo@garr.it
Agenda Inter-domain traffic: o how does NOC monitor and control it? Common case as example: new BGP peer activation -> new uncontrolled traffic balance Tools: o Control plane bgpviz (Ripe RIS) -> partial or limited information o Traffic port counters -> indistinct traffic Class usage counters -> AS peer stats only Netflow data -> AS origin per port stats How to collect AS origin data o Implementation Example: GARR AsTracker
Inter-domain traffic border traffic BGP protocol
Inter-domain traffic peers differ in: policy cost type of traffic
Inter-domain traffic: asymmetries
Inter-domain traffic: balancing
Common case: new peering
New traffic balance Due to unpredictable reason (often commercial policy) some traffic moves from some peer to new one
New traffic balance Interface counters show how much traffic swapped on new peer, but what traffic is moved from old peers to new one?
Available helpful tools: Control plane obgpviz (Ripe RIS) Traffic oport counters ofirewall filter counters onetflow data deployment effort ascending order
Ripe RIS / BGPViz Worldwide distributed route servers collect bgp routes Historical world bgp data. bgp update graphical visualization
Ripe RIS / BGPViz It help to understand inter-domain traffic reroutes Limits: few collection points (RIS route servers) = some ASes only no traffic amount information
Ripe RIS / BGPViz Make a request about a worldwide announced network timeslot selection
Ripe RIS / BGPViz
Ripe RIS / BGPViz update LOG example: changing path from 12350 174 137 to 12350 174 137 137 137 changing path from 6067 174 137 to 6067 3356 137 137 137 changing path from 30844 174 137 to 30844 174 137 137 137 changing path from 39202 174 137 to 39202 174 137 137 137 changing path from 8607 174 137 to 8607 3356 137 137 137 changing path from 22691 2914 3549 137 137 137 to 22691 174 137 137 137 changing path from 19151 3549 137 137 137 to 19151 174 137 137 137 changing path from 22691 174 137 137 137 to 22691 19624 174 137 137 137 changing path from 28917 174 137 to 28917 174 137 137 137 changing path from 39821 28917 174 137 to 39821 9002 3549 137 137 137 changing path from 8359 3356 137 137 137 to 8359 174 137 137 137 changing path from 31323 20764 174 137 to 31323 20764 3549 137 137 137 changing path from 8331 29076 29076 29076 174 137 to 8331 9002 9002 9002 3549 137 137 137
Interface counters got by snmp protocol interface aggregated traffic o no details about moved traffic
Class usage counters Source and Destination Class Usage as-path based counters useful for IXP peering o peer aggregate traffic number of class usage limited
Netflow data IP flow data: got by Netflow protocol IP flow (unidirectional) data: o protocol o IP addresses, o TCP/UDP ports, o AS numbers, o input/output interfaces, o TCP flags, o counters(bytes, pkts, flows) two choices: AS peer or AS origin It is possible to get worldwide AS stats ~ 60000 AS stats historical data (RRD files) per interface AS stats, good for analysis on: o balancing o asymmetries o re-routing
How to collect AS data implementation example = GARR AsTracker AS ranks single flow deep analysis simple AS stats multi AS (stacked) stats per user-as couple analysis
GARR AsTracker Real-time views Historical views data grouped by type: research commodity peer national IXPs direct peering Aggregates: by group stacked
GARR AsTracker backend: o make RRD o fill a database with AS stats for ranking pourpose o written in C language frontend o GUI: AS live ranking graph generation aggregations deep flow inspection o written in php (nfsen plugin)
GARR AsTracker homepage (live) all group aggregate "stacked" (some ASes) peer view
GARR AsTracker AS Traffic ranks: by peer by group general one week one month three months
GARR AsTracker deep flow inspection: o by site lookup function o by flow
GARR AsTracker AsTracker is used for: load balancing and billing policies control inter-domain routing troubleshooting Network planning
Example of use: new BGP peer = new traffic balance Telia + GlobalCrossing + Level3 New peering: Cogent
Tiscali AS example Telia (dismissed in september) Level3 GlobalCrossing Cogent (activated in november)
Tiscali AS example All TISCALI incoming traffic flows through GlobalCrossing All upcoming traffic is balanced flows through all commodity peers (GlobalCrossing, Level3 and Cogent)
Tiscali AS example
Tiscali AS example Traffic "close" to Rome goes through Cogent: RT.RM2-RE0>show route 82.84.217.24 inet.0: 394935 destinations, 899965 routes (394687 active, 5 holddown, 614 hidden) + = Active Route, - = Last Active, * = Both 82.84.0.0/15 *[BGP/170] 2w6d 14:15:08, MED 11010, localpref 100 AS path: 174 3257 8612 I > to 149.6.22.73 via ge-4/1/0.44 [BGP/170] 2w3d 13:36:53, MED 0, localpref 100, from 193.206.129.4 AS path: 3356 3257 8612 I > via so-4/0/0.0 [BGP/170] 1w1d 05:16:08, MED 2503, localpref 100, from 193.206.129.3 AS path: 3549 3257 8612 I > via so-4/0/0.0 HOT POTATO!
Tiscali AS example Traffic "close" to Milan goes through Level3: RT.MI2-RE0> show route 82.84.217.24 inet.0: 394797 destinations, 845650 routes (394609 active, 10 holddown, 249 hidden) + = Active Route, - = Last Active, * = Both 82.84.0.0/15 *[BGP/170] 2w3d 13:43:03, MED 0, localpref 100, from 4.68.3.212 AS path: 3356 3257 8612 I > to 213.242.65.81 via so-0/0/0.0 to 213.242.65.85 via so-5/2/0.0 [BGP/170] 1w1d 05:22:18, MED 2503, localpref 100, from 193.206.129.3 AS path: 3549 3257 8612 I > via so-3/0/0.0 [BGP/170] 2w6d 14:21:18, MED 11010, localpref 100, from 193.206.131.249 AS path: 174 3257 8612 I > via so-4/0/0.0 HOT POTATO!
Thanks for listening Questions?
Netflow data In case of IXP peerings, it is possible to understand what peer send our AS traffic with mac layer accounting data. This feature is supported by Netflow version 9 and IPFIX protocols only.