pmacct: introducing BGP na2vely into a NetFlow/sFlow collector Paolo Lucente pmacct <paolo at pmacct dot net> http://www.pmacct.net/ Netnod 2012 spring meeting, Stockholm, 17 th Feb 2012
Square 0 NetFlow is not the only exis2ng export protocol Current trend is to build extensible protocol formats by means of templates (NetFlow v9, IPFIX) or modules (sflow) Export protocols, versions and templates (or modules) available are solely mandated by the underlying network hardware, ie.: Cisco? NetFlow (and NetFlow- Lite on switches?) Juniper? NetFlow and IPFIX (so baaaad as we speak!) on M/T/MX; sflow on EX Brocade? sflow!
pmacct is open- source, free, GPL ed sovware
pmacct: where is it siwng?
pmacct: typical usage scenarios
pmacct: arising usage scenarios Build traffic matrices Peering & content distribu2on Support Business Intelligence Support Traffic Engineering & CP
The BGP peer who came from NetFlow (and sflow) pmacct introduces a Quagga- based BGP daemon Implemented as a parallel thread within the collector Doesn t send UPDATEs and WITHDRAWs whatsoever Behaves as a passive BGP neighbor Maintains per- peer BGP RIBs Supports 32- bit ASNs Supports both IPv4 and IPv6 Joins NetFlow (or sflow) and BGP data basing on: NetFlow source address == BGP source address
The BGP peer who came from NetFlow (and sflow) (cont.d) Relevant implementa2on details: Bases on trust: peers are not defined but a max number of peers who can connect is defined instead Ensures ibgp peering by presen2ng itself as part of the AS of the neighbor, as read in the OPEN message Enables the following traffic aggrega2on primi2ves: AS path, Local Preference, MED, Peer ASNs (freely mixed with origin ASNs), Communi2es, Peer IPs Caveats: BGP mul2- path is not supported
GeWng BGP to the collector Let the collector BGP peer with all PE devices: facing peers, transit and customers. Determine memory footprint (below in MB/peer) 60 500K IPv4 routes, 50K IPv6 routes, 64- bit executable 50 50 MB/peer 40 30 44.03 ~ 9GB total memory @ 500 peers MB/peer >= 0.12.4 20 22.73 19.97 18.59 18.12 17.89 17.76 17.57 17.48 17.39 MB/peer < 0.12.4 10 0 0 200 400 600 800 1000 1200 1400 Number of BGP peers
GeWng BGP to the collector (cont.d) Set the collector as ibgp peer at the PE devices: Configure it as a RR client for best results Collector acts as ibgp peer across (sub- )ASes BGP next- hop has to represent the remote edge of the network model: Typical scenario for MPLS networks Can be followed up to cover specific scenarios like: BGP confedera2ons Op2onally polish the AS- Path up from sub- ASNs default gateway defined due to par2al or default- only rou2ng tables
GeWng telemetry to the collector Export ingress- only measurements at all PE devices: facing peers, transit and customers. Traffic is routed to des2na2on, so plenty of informa2on on where it s going to It s crucial instead to get as much as possible about where traffic is coming from Leverage data reduc2on techniques at the PE: Sampling Aggrega2on (but be sure to carry IP prefixes!)
The BGP peer who came from NetFlow (and sflow) illustrated
Theory applied, SQL example (1/3) BGP Fields Counters Time shell> cat pmacct-create-db_bgp_v1.mysql [ ] create table acct_bgp ( Tag ); agent_id INT(4) UNSIGNED NOT NULL, as_src INT(4) UNSIGNED NOT NULL, as_dst INT(4) UNSIGNED NOT NULL, peer_as_src INT(4) UNSIGNED NOT NULL, peer_as_dst INT(4) UNSIGNED NOT NULL, peer_ip_src CHAR(15) NOT NULL, peer_ip_dst CHAR(15) NOT NULL, id=100 peer_src_as=<customer> comms CHAR(24) NOT NULL, as_path CHAR(21) NOT NULL, id=80 id=50 peer_src_as=<peer> peer_src_as=<ip transit> local_pref INT(4) UNSIGNED NOT NULL, [ ] med INT(4) UNSIGNED NOT NULL, packets INT UNSIGNED NOT NULL, bytes BIGINT UNSIGNED NOT NULL, stamp_inserted DATETIME NOT NULL, stamp_updated DATETIME, PRIMARY KEY ( ) shell> cat pretag.map shell> cat peers.map id=65534 ip=x in=a id=65533 ip=y in=b src_mac=j id=65532 ip=z in=c bgp_nexthop=w [ ] shell> mysql -u root -p < pmacct-create-db_bgp_v1.mysql
Theory applied, SQL example (2/3) See traffic delivered to a specific BGP peer mysql> SELECT peer_as_dst, bytes, stamp_inserted \ FROM acct_bgp \ WHERE peer_as_dst = <peer customer IP transit> \ GROUP BY peer_as_dst Same as above but also per loca2on if peering at mul2ple places mysql> SELECT peer_as_dst, peer_ip_dst, bytes, stamp_inserted \ FROM acct_bgp \ WHERE peer_as_dst = <peer customer IP transit> \ GROUP BY peer_as_dst, peer_ip_dst Simply replace dst with src in the above examples to see incoming traffic
Theory applied, SQL example (3/3) See outgoing traffic breakdown to all BGP peers of the same kind (ie. peers, customers, transit) mysql> SELECT peer_as_dst, bytes, stamp_inserted \ FROM acct_bgp \ WHERE local_pref = <<peer customer IP transit> pref> \ GROUP BY peer_as_dst
Miscellaneous features implemented AS Path radius: AS PATHs can get long. This can easily get counter produc2ve (ie. waste space) Cuts AS Path down to the specified number of AS hops. Assump&on: people might be more interested into what happens around them. Communi2es paqern filter IP prefixes can have many communi2es aqached Only a small subset might be relevant to the accoun2ng infrastructure Hence filtering capabili2es are made available
The source peer AS issue Traffic is tradi2onally routed to des2na2on Problem #1: limited informa2on about where traffic enters the AS domain. Applying this to tradi2onal NetFlow it means: either origin ASNs OR peer ASNs Problem #2: asymmetric rou2ng. Applying this to tradi2onal NetFlow it means: FIB lookup on the source prefix; result: source peer AS is where traffic should have entered the AS domain, if it was symmetrical.
The source peer AS issue mi2gated Having per- peer BGP RIBs on- board paves the way to capture both origin AND peer ASNs Source peer AS is sta2cally mapped against: A whole port, ie. input port field. Source MAC address. Should be the way to go and depends on NetFlow v9 or sflow. BGP next- hop lookup against the source prefix: Only choice if running NetFlow v5 in scenarios where mul2ple peers are off a single port (ie. IXP) Accuracy is not really predictable Combine with input port to limit false posi2ves
Briefly on scalability A single collector might not fit it all: Memory: can t store all BGP full rou2ng tables CPU: can t cope with the pace of telemetry export Divide- et- impera approach is valid: Assign PEs (both telemetry and BGP) to collectors Assign collectors to RDBMSs; or cluster the RDBMS. The matrix can get big, but can be reduced: Keep smaller routers out of the equa2on Filter out specific services/customers on dense routers Focus on relevant traffic direc2on (ie. upstream if CDN, downstream if ISP) Sample or put thresholds on traffic relevance
Further informa2on hqp://www.pmacct.net/lucente_pmacct_uknof14.pdf En22es on the provider IP address space Auto- discovery and automa2on hqp://wiki.pmacct.net/officialexamples Quick- start guide to setup a NetFlow/sFlow+BGP collector instance hqp://wiki.pmacct.net/implementa2onnotes Implementa2on notes (RDBMS, maintenance, etc.)
pmacct: introducing BGP na2vely into a NetFlow/sFlow collector Thanks for your attention! Questions? Paolo Lucente pmacct <paolo at pmacct dot net> http://www.pmacct.net/ Netnod 2012 spring meeting, Stockholm, 17 th Feb 2012