Defending the Network: Mitigating Attacks

Defending the Network: Mitigating Attacks Roland Dobbins <rdobbins@arbor.net> Solutions Architect +66-83-266-6344 BKK mobile +65-8396-3230 SIN mobile Arbor Public

Agenda Six Phases of Incident Response Reacting with the Data Plane Reacting with the Control Plane Using Dedicated Security Devices Page 2 - Arbor Public

Agenda Six Phases of Incident Response Reacting with the Data Plane Reacting with the Control Plane Using Dedicated Security Devices Page 3 - Arbor Public

Six Phases of Incident Response Page 4 - Arbor Public

Six Phases of Incident Response Post Mortem What was done? Can anything be done to prevent it? How can it be less painful in the future? Preparation Prep the Network Create Tools Test Tools Prep Procedures Train Team Practice Identification How do you know about the attack? What tools can you use? What s your process for communication? Reaction What options do you have to remedy? Which option is the best under the circumstances? Traceback Where is the attack coming from? Where and how is it affecting the network? What other current network problems are related? Classification What kind of attack is it? Page 5 - Arbor Public

Preparation Preparation Develop and Deploy a Solid Security Foundation Includes technical and non-technical components Encompasses best practices The hardest yet most important phase Without adequate preparation, you are destined to fail The midst of a large attack is not the time to be implementing foundational best practices and processes Page 6 - Arbor Public

Preparation Know the enemy Understand what drives the miscreants Understand their techniques Create the security team and plan Who handles security during an event; is it the security folks; the networking folks A good operational security professional needs to be a cross between the two: silos are useless Harden the devices Prepare the tools Network telemetry Reaction tools Page 7 - Arbor Public

Preparation Establish upstream/downstream contacts Understand their capabilities Establish a relationship and contact procedures An attack is no time to figure out how to contact an upstream or understand how they could potentially assist you Infrastructure security All of the techniques talked about today also assume that the infrastructure is available to route and forward packets! Page 8 - Arbor Public

Are You Pushing the Envelope? Know Your Equipment and Infrastructure: Know the performance envelope of all your equipment (routers, switches, workstation, etc.). You need to know what your equipment is really capable of doing. Know the capabilities of your network. If possible, test it. Surprises are not kind during a security incident. PPS vs. BPS and how enabling features impacts them Page 9 - Arbor Public

Are You Pushing the Envelope? Get Real! Operator, I tried to push my aircraft to 70,000 ft and it stalled. Vendor, But the aircraft was only designed for a 50,000 ft ceiling. Operator, I need it to go to 70,000 ft, so you should make that happen. Vendor, But that is not going to happen; 50,000 ft is the only thing it can do. You knew that when you bought it. Operator, Your equipment sucks if you cannot exceed you design specs. Page 10 - Arbor Public

Identification, Classification, and Traceback All of this assumes you can detect and understand the attack Reacting to attacks depends, in a lot of ways, on how you detect the attacks Time of reaction is often a critical factor Once stateful devices fail, the restoration path is usually a hard reboot Page 11 - Arbor Public

Reaction Many varying reaction mechanisms No one tool or technique is applicable in all circumstances Think toolkit Automate where possible Don t forget about the operational costs It is critical to identify and classify an attack so you can choose the most appropriate mitigation tool Every problem does not call for a hammer solution, simplicity is key Some solutions actually make the problem worse! Page 12 - Arbor Public

Attack Vectors Infrastructure attacks Attacks against vulnerability or protocol weakness (D)DoS attacks directed at infrastructure Downstream customer attacks Attack that only impacts target IP Attack that impacts downstream customer Attack where collateral damage impacts multiple customers Downstream customer sourced attacks Attacks that originate from your customers Page 13 - Arbor Public

Attack Vectors Infrastructure attacks Attacks against vulnerability or protocol weakness (D)DoS attacks directed at infrastructure Downstream customer attacks Attack that only impacts target IP Attack that impacts downstream customer Attack where collateral damage impacts multiple customers Downstream customer sourced attacks Attacks that originate from your customers Page 14 - Arbor Public

Post Mortem Post Mortem Analyzing What Just Happened. What Can Be Done to Build Resistance to the Attack Happening Again The step everyone forgets or doesn t make time to conduct What can you do to make it faster, easier, less painful in the future? Complete the loop Page 15 - Arbor Public

Agenda Six Phases of Incident Response Reacting with the Data Plane Reacting with the Control Plane Using Dedicated Security Devices Page 16 - Arbor Public

Reacting with the Data Plane Page 17 - Arbor Public

RFC3704/BCP84 Ingress Packet Filtering Packets should be sourced from valid, allocated address space, consistent with the topology and space allocation Our goal here is to bind the problem and reduce the requirements for implementing security No BCP84 means that: Devices can (wittingly or unwittingly) send traffic with spoofed and/or randomly changing source addresses out to the network Complicates trace back immensely Sending bogus traffic is not free! Attacks can be much more devious with spoofing Page 18 - Arbor Public

BCP 84 Packet Filtering Principles Filter as close to the edge as possible Filter as precisely as possible Filter both source and destination where possible Can be implemented in various ways Infrastructure ACLs (iacls) Unicast reverse path forwarding (urpf) Cable source verify DHCP IP source TMS/DHCP snooping Page 19 - Arbor Public

Where to React? Upstream A IXP-W Sinkhole Network Peer A Peer B Upstream A IXP-E Upstream B Upstream B Target POP NOC Page 20 - Arbor Public

Where to React? Upstream B IXP-W Sinkhole Network Peer A Peer B Upstream A IXP-E The Proper Reaction is the Easiest, Fastest, and Simplest One That Can Minimize the Collateral Damage Upstream B Target POP NOC Page 21 - Arbor Public

Reacting with the Data Plane: Access Control List (ACL) Page 22 - Arbor Public

Reacting to an Attack with ACLs Traditional method for stopping attacks Scaling issues encountered: Operational difficulties Changes on the fly Multiple ACLs per interface Performance concerns Page 23 - Arbor Public

ACLs: Deployment Considerations How does the ACL load into the router? Does it interrupt packet flow? How many ACEs can be supported in hardware? In software? How does ACL depth impact performance? How do multiple concurrent features affect performance? Page 24 - Arbor Public

Working the PPS Engineering 10,000,000 PPS Performance Envelope Packets per Second (pps) 1,000,000 100,000 10,000 1,000 1 Gbps 100 Mbps 10 Mbps CPU Limit 100 Performance Envelope w/features 40 240 400 800 1400 Packet Size (bytes) Page 25 - Arbor Public Basic Performance Envelope

ACL Update and Reaction Speed Internet speeds above the OC-12/48 range require specialized forwarding/feature ASICs to provide services at line rate ACLs loaded into these ASICs require special processing: Example: 100Kb file for 5,000-Line ACL 1. Load ACL into router from mgmt app or ftp server (transfer time for big ACLs) 2. Commit ACL to active 3. Pre-process (compile) ACL 4. Push to Line Card(s) (if distributed architecture) 5. Process for loading into Line Card ASIC 6. Load into Line Card ASIC and activate access speed- and memory carddependent, can be slow e.g., minutes small Can be lengthy: 10 s of seconds to min s msecs small Platform-dependent: usecs to mins Modular design, simplicity, and amplification effect principles apply to ACL design Page 26 - Arbor Public

Filtering Fragments Fragments can be explicitly denied Fragment handling is enabled via fragments keyword Default permit behavior permit fragments that match ACE L3 entries Denies fragments and classifies fragment by protocol: access-list 110 deny tcp any any fragments access-list 110 deny udp any any fragments access-list 110 deny icmp any any fragments Page 27 - Arbor Public

Packet Filtering Viewed Horizontally Spoofed Source Addresses Targeting the Infrastructure Packet Shield # 1 Application Filters Policy Enforcement Targeting the Customer Packet Shield # 2 Packet Shield # 3 Packet Shield # 4 Customer Traffic The best ACL may actually be multiple ACLs at possibly different locations Page 28 - Arbor Public

ACL Construction Most common problems: Poorly-constructed ACLs Ordering matters! Understand platform specifics (i.e., 6500/7600 LOUs, masks)! Scaling and maintainability issues with ACLs are commonplace Make your ACLs as modular and simple as possible Page 29 - Arbor Public

ACL Categories: Hybrid Philosophy Hybrid Permit/Deny Anti-spoofing Anti-bogon (source) Infrastructure Explicit deny specific L3 Explicit deny specific L4 Incident reaction Explicit permit L3 (good traffic) Explicit permit L4 (good traffic) Explicit deny everything else Page 30 - Arbor Public

Layered/Modular ACLs Hybrid Permit/Deny Anti-spoofing Anti-bogon (source) Infrastructure Explicit deny specific L3 Explicit deny specific L4 Incident reaction Explicit permit L3 (good traffic) Explicit permit L4 (good traffic) Explicit deny everything else Rarely Changes Rarely Changes Rarely Changes Sometimes Changes Sometimes Changes Changes Every Day Sometimes Changes Sometimes Changes Rarely Changes Page 31 - Arbor Public

Operational Cost Impact Hybrid Permit/Deny Anti-spoofing Anti-bogon (source) Infrastructure Explicit deny specific L3 Explicit deny specific L4 Incident reaction Explicit permit L3 (good traffic) Explicit permit L4 (good traffic) Explicit deny everything else More Static $ Rarely Changes Rarely Changes $$ Rarely Changes Sometimes Very Changes $$$$$$ Dynamic Sometimes Changes Changes Every Day $$ Sometimes Changes Sometimes Changes More Static $ Rarely Changes Page 32 - Arbor Public

ACL Summary ACLs are widely deployed as a primary containment tool Prerequisites: identification and classification need to know what to filter Apply as specific an ACL as possible ACLs are good for static attacks, not as effective for rapidly changing attack profiles Understand ACL performance limitations before an attack occurs Operational efficiencies are important scripted Page 33 - Arbor Public

The Pros and Cons of ACLs ACLs - key strengths: Detailed packet filtering (ports, protocols, ranges, fragments, etc.) Relatively static filtering environment Clear filtering policy ACLs can have issues when faced with: Dynamic attack profiles (different sources, different entry points, etc.) Frequent changes Quick, simultaneous deployment on a multitude of devices Operationally hard to remove Page 34 - Arbor Public

Reacting with the Data Plane: Committed Access Rate (CAR) Page 35 - Arbor Public

Reacting to an Attack with CAR The Internet Customers Layer-3 CAR Filter Layer-3 input and output rate limits specifically input rate limits Security filters use the input rate limit to drop packets before they are forwarded through the network Aggregate and granular limits Port, MAC address, IP address, application, precedence, QOS_ID Excess burst policies Page 36 - Arbor Public

A CAR Example: SMURF access-list 199 permit icmp any <target> echo-reply access-list 199 deny ip any any interface POS2/0 rate-limit output access-group 199 256000 64000 64000 conform-action transmit exceed-action drop Page 37 - Arbor Public

A Bad CAR Example: Syn-Flood access-list 199 permit tcp any <target> eq www syn access-list 199 deny ip any any interface POS2/0 rate-limit output access-group 199 256000 64000 64000 conform-action transmit exceed-action drop Syn-floods are generally high volume If attack is 99% of traffic, legitimate traffic has a small chance of making it through the rate-limit Are we really achieving anything? Page 38 - Arbor Public

Agenda Six Phases of Incident Response Reacting with the Data Plane Reacting with the Control Plane Using Dedicated Security Devices Page 39 - Arbor Public

Reacting with the Control Plane Page 40 - Arbor Public

Routers Drop Data, Often Scans, Backscatter, Other Garbage AS 100 AS 65530 C D G A 10.1/16 F 10.1.32.0/19 AS 65531 B 10.1.0.0/19 E H 10.1.64.0/19 An AS collects all the garbage (backscatter, scans, etc.) destined for 10.1.0.0/19, 10.1.96.0/19, and 10.1.128.0/17 addresses Routers that source those aggregates drop the data to unreachable parts of the networks, and are required to process data, send ICMP unreachables, etc. Page 41 - Arbor Public

Reacting with the Control Plane: Destination-Based Black Hole Filtering Page 42 - Arbor Public

Black Hole Filtering Black hole filtering or black hole routing forwards a packet to a router s bit-bucket Also known as route to Null0 Works only on destination addresses, since it is really part of the forwarding logic Forwarding ASICs are designed to work with routes to Null0 dropping the packet with minimal to no performance impact Used for years as a means to blackhole unwanted packets Page 43 - Arbor Public

Remotely Triggered Black Hole Filtering We will use BGP to trigger a network-wide response to an attack A simple static route and BGP will enable a network-wide destination address black hole as fast as ibgp can update the network (msecs) This provides a tool that can be used to respond to security-related events and forms a foundation for other remotely triggered uses Often referred to as RTBH Page 44 - Arbor Public

Remotely Triggered Black Hole (RTBH) Configure all edge routers with static route to Null0 (must use some reserved network) Configure trigger router Part of ibgp mesh Dedicated router recommended Activate black hole Redistribute host route for victim into BGP with next-hop set to 192.0.2.1 Route is propagated using BGP to all BGP speakers and installed on routers with 192.0.2.1 route All traffic to victim now sent to Null0 Page 45 - Arbor Public

Step 1: Prepare All the Routers with Trigger Select a small block that will not be used for anything other than black hole filtering; Test-Net (192.0.2.0/24) is optimal since it should not be in use Put a static route with Test-Net 192.0.2.0/24 to Null0 on every edge router on the network ip route 192.0.2.1 255.255.255.255 Null0 Page 46 - Arbor Public

Step 1: Prepare All the Routers with Trigger Edge Router with Test-Net to Null0 IXP-W Peer A Edge Router with Test-Net to Null0 Peer B IXP-E Upstream A Sinkhole Network Upstream A Upstream B Upstream B 171.16.61.0/24 Target 172.16.61.1 POP Edge Router with Test-Net to Null0 G NOC Page 47 - Arbor Public

Step 2: Prepare the Trigger Router The Trigger Router Is the Device that Will Inject the ibgp Announcement into the ISP s Network Should be part of the ibgp mesh but does not have to accept routes Can be a separate router (recommended) Can be a production router Can be a workstation with Zebra/Quagga (interface with Perl scripts and other tools) Can be Arbor Peakflow SP GUI interface, integrated with detection, cleanup timer, etc. Page 48 - Arbor Public

Trigger Router s Config Redistribute Static with a Route-Map Match Static Route Tag router bgp 65535 redistribute static route-map static-to-bgp! route-map static-to-bgp permit 10 match tag 66 set ip next-hop 192.0.2.1 set local-preference 200 set community no-export set origin igp! route-map static-to-bgp permit 20 Set Next- Hop to the Trigger Set Local-Pref Page 49 - Arbor Public

Step 3: Activate the Black Hole Add a static route to the destination to be black holed; the static is added with the tag 66 to keep it separate from other statics on the router ip route 172.16.61.1 255.255.255.255 null0 tag 66 BGP advertisement goes out to all BGP-speaking routers Routers receive BGP update and glue it to the existing static route; due to recursion, the next-hop is now Null0 Page 50 - Arbor Public

Step 3: Activate the Black Hole FIB Glues 172.16.61.1 s Next-Hop to Null0, Triggering the Black Hole Filtering FIB FIB Best Path Selection (Unless Multipath) 172.16.61.1nexthop = 192.0.2.1 BGP Best Path Selection BGP 65560 RIB AS 65000 s Routes AS 65535 s Routes AS 65536 s Routes OSPF/ISIS RIB 172.16.61.1 nexthop = 192.0.2.1 with no-export 192.0.2.1/32 = Null0 Static and Connected Routes 192.0.2.1/32 = Null0 Page 51 - Arbor Public

Step 3: Activate the Black Hole BGP Sent 172.16.61.1 Next-Hop = 192.0.2.1 Static Route in Edge Router 192.0.2.1 = Null0 172.16.61.1 = 192.0.2.1 = Null0 Next-Hop of 172.16.61.1 Is Now Equal to Null0 Page 52 - Arbor Public

Step 3: Activate the Black Hole IXP-W Peer A A Peer B IXP-E Upstream A B Upstream A C D Upstream B Target E Upstream B ibgp Advertises List of Black Holed Prefixes F POP G NOC Page 53 - Arbor Public

Customer Is DOSed (After) Packet Drops Pushed to the Edge IXP-W Peer A A Peer B IXP-E Upstream A B Upstream A C D Upstream B E Upstream B Target ibgp Advertises List of Blackholed Prefixes F POP G NOC Page 54 - Arbor Public

Trigger Router Config Can use multiple tags One tag to redirect attack to sinkhole Another tag to redirect attack to anycast sinkhole Multiple tags to black hole for different reasons Tag #1 is for ongoing (d)dos attack Tag #2 is for black holing botnet command and control Tag #3 is for phishing site Tag #4 is for SPAM Makes tracking easier. Can usually figure out which group to contact about black hole (i.e. NOC, abuse, security, etc.) just by looking at trigger router configuration. Page 55 - Arbor Public

An Alternative: Community-Based Trigger BGP community-based triggering allows for more fine-tuned control over where you drop the packets Three parts to the trigger: Static routes to Null0 on all the routers Trigger router sets the community Reaction router (on the edge) matches community and sets the next-hop to the static route to Null0 Page 56 - Arbor Public

Why Community-Based Triggering? Allows for More Control on the Attack Reaction Trigger community #1 can be for all routers in the network Trigger community #2 can be for all peering routers; no customer routers allows for customers to talk to the DOSed customer within your AS Trigger community #3 can be for all customers; used to push a inter- AS traceback to the edge of your network Trigger communities per ISP peer can be used to only black hole on one ISP peer s connection; allows for the DOSed customer to have partial service Trigger communities per geographic region can be used Page 57 - Arbor Public

Trigger Router Config (Community-Based Approach) Redistribute Static with a Route-Map Match Static Route Tag router bgp 65535 redistribute static route-map static-to-bgp! route-map static-to-bgp permit 10 match tag 123 set community 65535:123 set local-preference 200 set community no-export set origin igp! route-map static-to-bgp permit 20 match tag 124 set community 65535:124 set local-preference 200 set community no-export set origin igp Set Community Set Local-Pref Page 58 - Arbor Public

Drop Router Config (Community-Based Approach) This Router Drops Community 123 router bgp 65535 neighbor <ibgp peer> route-map ibgp-peers in!my Region ip community-list 1 permit 65535:123!Other region ip community-list 2 permit 65535:124! route-map ibgp-peers permit 10 match community 1 set ip next-hop 192.0.2.1 set local-preference 200 set community no-export set origin igp! route-map static-to-bgp deny 20 match community 2! route-map static-to-bgp permit 30 Set Next-Hop to trigger This router does not drop community 124 Page 59 - Arbor Public

Two RTBH Approaches Tag-based approach: Concentrates configuration complexity on one trigger router Edge devices require simple static route to Null0 Monitoring (OpEx) Prefixes which are being dropped (and why) best viewed on trigger router (e.g., show run include tag ) Community-based approach: Configuration complexity spread equally to all devices Allows greater flexibility for drop control (e.g., regional) Monitoring (OpEx) Prefixes which are being dropped on a particular device (and why) can be determined by reviewing the output of sh ip bgp community on that device Page 60 - Arbor Public

Reacting with the Control Plane: Service Provider Support for Customer Initiated Destination- Based Black Hole Filtering Page 61 - Arbor Public

Customer-Initiated RTBH Many service providers offer their customers a customer triggered version of RTBH We ll accept /32s with community <AS>:666 and we ll black hole them in our network for you It s critical to understand which of your upstream/ peers support this How many prefixes will they accept? What community triggers it? Are you going to support it for your customers? Page 62 - Arbor Public

Customer-Initiated RTBH Config router bgp 65535 neighbor <customer> route-map customer-rtbh in neighbor <customer> prefix-list customera-filter in! ip community-list 1 permit 65535:666! route-map customer-rtbh permit 10 match community 1 set ip next-hop 192.0.2.1 set local-preference 200 set community no-export set origin igp! route-map customer-rtbh deny 20!Deny BOGONs! route-map customer-rtbh permit 20 Page 63 - Arbor Public

Customer-Initiated RTBH Issues: Must ensure prefix-list-based filtering We wouldn t want your customers to be able to black hole some important website, now would we? Restrict the black hole to the PE router only with no-advertise, restrict to network only with noexport, or pass along to peers and upstreams? Use the same ebgp session to customers or build a dedicated ebgp session If using the same session, be careful with maxprefix Page 64 - Arbor Public

Reacting with the Control Plane: Source- and Destination-Based Black Hole Filtering Page 65 - Arbor Public

S/RTBH: Triggered Source Drops Dropping on destination is very important Dropping on source is often what we really need Reacting using source address provides some interesting options: Stop the attack without taking the destination offline Filter command and control servers Filter (contain) infected end stations Must be rapid and scalable Leverage pervasive BGP signaling again! Page 66 - Arbor Public

Strict urpf Check (Unicast Reverse Path Forwarding) router(config-if)# ip verify unicast source reachable-via rx i/f 1 S D Data i/f 2 i/f 3 i/f 1 S D Data i/f 2 i/f 3 FIB:... S -> i/f x... FIB:... S -> i/f 2... Same i/f: Forward Other i/f: Drop Page 67 - Arbor Public

Loose urpf Check (Unicast Reverse Path Forwarding) router(config-if)# ip verify unicast source reachable-via any i/f 1 S D Data i/f 2 i/f 3 i/f 1 S D Data i/f 2 i/f 3 Any i/f: Forward FIB:... S -> i/f x... FIB:....?..... Not in FIB or Route Null0: Drop Page 68 - Arbor Public

Source-Based Remotely-Triggered Black Hole Filtering (S/RTBH) Uses the same architecture as destination-based filtering and Unicast RPF Edge routers must have static in place They also require Unicast RPF BGP trigger sets next-hop in this case the victim is the source we want to drop Page 69 - Arbor Public

Source-Based Remotely Triggered Black Hole Filtering What do we have? Black Hole Filtering if the destination address equals Null0, we drop the packet Remotely Triggered trigger a prefix to equal Null0 on routers across the network at ibgp speeds urpf Loose Check if the source address equals Null0, we drop the packet Put them together and we have a tool to trigger a drop for any packet coming into the network whose source or destination equals Null0 Page 70 - Arbor Public

Customer Is DOSed (After) Packet Drops Pushed to the Edge Edge Routers Drop Incoming Packets Based A on Their Source Address IXP-W Peer A Peer B Edge Routers Drop Incoming Packets Based on Their Source Address IXP-E Upstream A Upstream A Upstream B Upstream B Target F POP NOC ibgp Advertises List of Black Holed Prefixes Based on Source Addresses Page 71 - Arbor Public

Source-Dropping Caution Caution: you will drop all packets with that source and/or destination Remember spoofing! Don t let the attacker spoof the true target and trick you into black holing it for them Whitelist important sites which should never be blocked (i.e., root & TLD nameservers, etc.) via prefix-lists Page 72 - Arbor Public

Source-Based RTBH - S/RTBH Advantages: No ACL update No change to the router s configuration Drops happen in the forwarding path Frequent changes when attacks are dynamic (for multiple attacks on multiple customers) Limitations: Source detection and enumeration Attack termination detection (reporting) Resource utilization: finite resources Effects all traffic, on all triggered interfaces, regardless of actual intent Peakflow SP can serve as the GUI-based trigger for S/RTBH supports communities, timers, etc. Makes triggering/blocking easy! Page 73 - Arbor Public

Agenda Six Phases of Incident Response Reacting with the Data Plane Reacting with the Control Plane Using Dedicated Security Devices Page 74 - Arbor Public

Using Dedicated Security Devices Page 75 - Arbor Public

Given Everything Said, What Remains? We have discussed techniques that are very effective at limiting the collateral damage Raise the bar; stop only bad traffic In asymmetric environments, especially across peers, packet spoofing is still problematic Detection of exactly who is attacking is problematic Doing all this in the core requires specialized hardware, which has scaling and availability problems Page 76 - Arbor Public

Network IDS/ IPS Terminology False positives: system mistakenly reports certain benign activity as malicious; also called false alarms False negatives: system does not detect and report actual malicious activity False positives, performance, need for traffic symmetry, and increased risk of DDoS due to capacity/state are the banes of IDS technology! Additionally, you require a signature in order to stop the attack what if this is a new attack? Page 77 - Arbor Public

Modern Stateful Firewalls: The Inadequate Security Default What It Is Sometimes called a hybrid Combines features of other firewall approaches such as: Access control lists Application-specific proxies/inspections Stateful inspection Plus features of other devices: Web (HTTP) cache Specialized servers SSH, SOCKS, NTP Most include VPN, some include IDS/ IPS Pros and Cons Pro: Maintains most of the speed advantage of a simple stateful firewall Pro: Application layer gateway services provide application security while resolving the NAT issue Con: Does not provide complete session termination, as would a full proxy Con: Actively tracks the state of incoming connections a DoS issue Con: Performance a DoS issue Con: Inspectors are an attack surface Page 78 - Arbor Public

Formal Requirements for a Core Security Device Need to avoid state Constant state tracking leaves us vulnerable to DDoS attacks Doesn t rely on signatures If I get an attack with no signature, I cannot block it Possibly can use signature-like filters, however, after the fact Doesn t have to be in-line when it isn t needed Scales easily Doesn t require traffic symmetry The Internet is a very asymmetrical place! Page 79 - Arbor Public

Firewalls and IDS/ IPS don t help! It s time to put the firewall and IDS/ IPS myth to rest! Firewalls are policy-enforcement devices they can t help with DDoS, and in most cases, the policies applied to the firewalls have been devised with no visibility into network traffic, so the firewall rules bear little relation to what should actually be permitted and denied. IDS/ IPS are by definition always behind the attackers in order to have a signature for something, you must have seen it before. IDS/ IPS have proven to be totally ineffective at dealing with application-layer compromises, which is how most hosts are botted and used for DDoS, spam, corporate espionage, identity theft, theft of intellectual property, etc. Firewalls & IDS/ IPS output reams of syslog which lacks context, and which nobody analyzes. It is almost impossible to relate this syslog output to network behaviors. End-customers subscribe to traditional managed security services based on firewalls and IDS/ IPS, and still get compromised! Firewall & IDS/ IPS deployments cause performance & usability problems, and don t scale, shouldn t be deployed in front of servers! Page 80 - Arbor Public

Core Design Philosophies Scale by using traffic shunting Core packet cleaning requirements 1) Validate incoming traffic to make sure it comes from the source IPs that are in the SRC IP field of the packet 2) Evaluate these validated sources against a baseline and then recommend either further processing or dropping for sources that misbehave Don t need to stop every bad packet instead, focus on not stopping any good packets Pad thresholds to reduce likelihood of false positives Have very high defaults so in a non-profiled environment you won t block good traffic Page 81 - Arbor Public

Packet Cleaning Issues Shunting the Packets Cleaning the Packets Page 82 - Arbor Public

Traffic Shunts Intercept and shunt traffic to the mitigation device the scrubber Return good traffic back to the customer Need to avoid forwarding loops means some sort of tunneling Page 83 - Arbor Public

Arbor DDoS Solution: Diversion/Offramping Arbor TMS NetFlow to Arbor Peakflow SP Protected Zone 1: Web Protected Zone 2: Name Servers Protected Zone 3: E-Commerce Application Page 84 - Arbor Public

Arbor DDoS Solution: Diversion/Offramping Arbor TMS NetFlow to Arbor Peakflow SP 1. Detect Protected Zone 1: Web Target Protected Zone 2: Name Servers Protected Zone 3: E-Commerce Application Page 85 - Arbor Public

Arbor DDoS Solution: Diversion/Offramping Arbor TMS 2. Activate: Auto/Manual NetFlow to Arbor Peakflow SP 1. Detect Protected Zone 1: Web Target Protected Zone 2: Name Servers Protected Zone 3: E-Commerce Application Page 86 - Arbor Public

Arbor DDoS Solution: Diversion/Offramping BGP Announcement 3. Divert Only Target s Traffic Arbor TMS 2. Activate: Auto/Manual NetFlow to Arbor Peakflow SP 1. Detect Protected Zone 1: Web Target Protected Zone 2: Name Servers Protected Zone 3: E-Commerce Application Page 87 - Arbor Public

Arbor DDoS Solution: Diversion/Offramping BGP Announcement Traffic Destined to the Target 4. Identify and Filter the Malicious 3. Divert Only Target s Traffic Arbor TMS 2. Activate: Auto/Manual NetFlow to Arbor Peakflow SP 1. Detect Protected Zone 1: Web Target Protected Zone 2: Name Servers Protected Zone 3: E-Commerce Application Page 88 - Arbor Public

Arbor DDoS Solution: Diversion/Offramping BGP Announcement Traffic Destined to the Target Legitimate Traffic to Target 4. Identify and Filter the Malicious 2. Activate: Auto/Manual 1. Detect 3. Divert Only Target s Traffic Arbor TMS NetFlow to Arbor Peakflow SP Protected Zone 1: Web Target Protected Zone 2: Name Servers Protected Zone 3: E-Commerce Application Page 89 - Arbor Public 5. Forward the Legitimate

Arbor DDoS Solution: Diversion/Offramping 6. Non- Targeted Traffic Flows Freely BGP Announcement Traffic Destined to the Target Legitimate Traffic to Target 4. Identify and Filter the Malicious 3. Divert Only Target s Traffic Arbor TMS 2. Activate: Auto/Manual NetFlow to Arbor Peakflow SP 1. Detect Protected Zone 1: Web Target Protected Zone 2: Name Servers Protected Zone 3: E-Commerce Application Page 90 - Arbor Public 5. Forward the Legitimate

Design Considerations Network chokepoints Back haul attack traffic across potential costly or congested links. SLA s being offered Availability Guaranteed mitigation capacity Provisioning and Operation Simpler is better Existing Networking technology Often limited by what capabilities exist in the network today Number of mitigation devices and capacity Redundancy What happens in a failure scenario? Page 91 - Arbor Public

Deployment strategies Distributed mitigation Regional deployment strategy Per PoP or per Peering center location Internet PEERING PEERING P1 P2 P1 P2 Peakflow SP TMS Peakflow SP TMS L3 Switch S1 S2 S1 S2 POP A POP B CORE CORE C1 C2 C1 C2 Core Customer CPE Page 92 - Arbor Public

Distributed Mitigation Benefits Keeps attack traffic at edge Limited backhauling of attack traffic Limits exposure of Internal infrastructure Easier Capacity planning Not as worried about how much attack traffic would have to be backhauled Good shared mitigation capability Geographical redundancy Drawbacks Power and Space Requirements Scalability - How much mitigation capacity can you add at each location Harder to dedicate mitigation capacity per-customer Potentially more equipment to purchase upfront Potential to backhaul Customer to Customer attack traffic Multiple scrubbers involved in every single mitigation Page 93 - Arbor Public

Distributed Mitigation Bottom line This can be a workable strategy for Tier-Two, -Three, and MSOs Strategically-consolidated Internet access points. Backbone capacity is already focused on these locations. Customer-to-customer attacks are likely to be small in size. Not many large-bandwidth customers. Backbone capacity and infrastructure likely to be vulnerable to large scale attacks. Better ROI to keep attack traffic off of backbone. Fewer business customers likely to pay for dedicated mitigation capacity. Page 94 - Arbor Public

Deployment Strategies Centralized mitigation AKA Cleaning Center Peers Transit P1 P2 IP Core Data D1 D2 C2 C2 S1 S2 Cleaning Center POP B Customer Aggregation A1 S1 A2 S2 Peakflow SP TMS Customer CPE Page 95 - Arbor Public

Centralized Mitigation Benefits Can start small 2 TMS devices for redundancy Can be located where power and space allow easy growth Possibly fits in more with other hosted service offerings Easier to troubleshoot You know where traffic should go to and from Drawbacks Back haul attack traffic Potential for backbone infrastructure to be impacted by attack traffic Must plan for capacity How much attack traffic potentially could customers pull to Cleaning Center Limited topological/ geographical diversity regional Cleaning Centers are the answer Page 96 - Arbor Public

Centralized Mitigation Bottom Line This is a good strategy for most deployments Very distributed Peering locations and limited or no purchased Transit. Customer-to-customer attacks can be quite large. Think of a MSO-based zombie attacking large Bank. Many large-bandwidth customers. Backbone and Internet Data Center capacity readily available. More business customers likely to pay for dedicated mitigation capacity because they have higher-bandwidth demands. Easy to scale capacity more TMSes per cluster, multiple clusters Page 97 - Arbor Public

Offramping/Diversion Goal Get attack traffic to correct TMS and Port BGP Next-hop Anycast Get attack traffic to closest TMS Load balance distributed attacks geographically Built-in redundancy by leveraging routing protocols ECMP (Equal Cost Multi-Path) Multiple equal routes pointing to same advertised TMS BGP next-hop IP to achieve multi-gigabit performance CEF-based load-leveling is Cisco terminology SLA-based Next-hops or Communities Provide dedicated mitigation ports and capacity to meet customer SLA Page 98 - Arbor Public

BGP Next-hop Anycast 1.TMS advertises off ramp (victim) prefix w/ next hop of TMS L0 (virtual IP) Internet 2. P1 and P2 do a recursive lookup on TMS L0 which matches static route to directly connected interface/s PEERING S1 P1 P2 P1 P2 S2 POP A 3. Since victim prefix and nexthop is learned in both PoP-A and PoP-B Attack traffic is sent to closest TMS POP B S1 S2 PEERING Peakflow SP TMS CORE CORE C1 C2 C1 C2 Other option: Use same Off-ramp Pt2Pt for all TMS off-ramp ports announce bgp next-hop to be TMS Pt2Pt. Eliminates static routes IP Core Customer CPE Note: Allow Victim Prefix to be propagated and redistribute Static or Pt2Pts in IGP for failover Page 99 - Arbor Public

ECMP (Equal Cost Multi-Path) PEERING 1.TMS advertises off ramp (victim) prefix w/ next hop of 1.1.1.1 2. Equal static routes to 1.1.1.1(BGP Nexthop) load balances off ramp across two or more ports Peakflow SP TMS POP B CORE Note: You can also use port bonding (logical ports) to treat multiple ports as one. Customer CPE BGP announce Direct Off-Ramp Direct On-Ramp Page 100 - Arbor Public

SLA-based Dedicated Mitigation Capacity 2. Router s BGP policy takes customer specific community and changes nexthop to dedicated PEERING pt 2 pt interface or recursive lookup match ECMP routes to multiple ports POP B CORE Note: You can just different next-hops but communities allow you to set up shared mitigation, dedicated mitigation, and overflow of dedicated to shared Customer CPE Peakflow SP TMS 1.TMS advertises offramp prefix w/ next hop of 1.1.1.1 and customer specific off-ramp community BGP announce Direct Off-Ramp Direct On-Ramp Page 101 - Arbor Public

Off-ramping/Diversion Bottom Line Use lessons learned from Sinkhole and Blackhole usage. Keep it simple with next-hop changes or add granularity by utilizing BGP communities, multiple next-hops and ECMP routes to next-hops. To achieve multi-gigabit mitigation capability, traffic must transit multiple TMS ports. Think about where routes are being heard and how. Then run through failure scenarios. Is attack traffic still going to make it to a TMS or will it go to customer dirty? Page 102 - Arbor Public

On-Ramping/Re-injection Goal Avoid Routing loops GRE/mGRE Tunnels Routing loop avoided by forwarding decision being performed on tunnel endpoint VRF VPN Avoid routing loop by utilizing non-global route table L2 Forwarding Routing loops avoided by selective route advertisement and distribution control. Requires hierarchical logical network design. Page 103 - Arbor Public

Tunnel GRE (Generic Route Encapsulation) TMS advertises victim prefix which is propagated throughout network. One to one mapping of off-ramp port to on-ramp port Peakflow SP TMS POP B L3 Switch CORE Customer CPE processes GRE packet on Tunnel interface and forwards original packet to victim IP Customer CPE To avoid routing loop, TMS encapsulates packet in GRE packet with tunnel Source and Destination IP BGP announce Direct Off-Ramp Direct On-Ramp Clean Traffic Page 104 - Arbor Public

On-Ramping/Re-injection via GRE/mGRE Benefits Easy to avoid routing loops Redundant tunnels can be configured Proven On-Ramping method Drawbacks per-customer mitigation prefix configuration (not dynamic) Provisioning Engineers / Systems must touch Peakflow system Per-TMS tunnel configuration (more work for distributed model) Page 105 - Arbor Public

VRF - VPN TMS advertises victim prefix which is propagated throughout network. VRF VRF One to one mapping of off-ramp port to on-ramp port Peakflow SP TMS MPLS/IP Core Customer Aggregation VRF VRF Routing loop is avoided because traffic is being forwarded inside VPN/ Label switched BGP announce Static route to customer s protected CIDR is preconfigured and redistributed into VPN Customer CPE Direct Off-Ramp Direct On-Ramp Clean Traffic Page 106 - Arbor Public

On-Ramping/Re-injection via VRF-VPN Benefits Easy to avoid routing loops Leverages built in network redundancy Simple static route required for each protected customer prefix. Most provisioners know how to do this. Drawbacks Per-customer mitigation prefix configuration MPLS must be used throughout network Multiple technologies in use could cause operational issues. Page 107 - Arbor Public

Direct L2 Onramping/Re-injection TMS advertises victim prefix which is propagated throughout network. Off-ramp and On-ramp routers must be different L3 devices Peakflow SP TMS POP B L3 Switch CORE Customer CPE processes GRE packet on Tunnel interface and forwards original packet to victim IP Customer CPE To avoid routing loop, you restrict the more specific victim prefix to only specific routers BGP announce Direct Off-Ramp Direct On-Ramp Clean Traffic Page 108 - Arbor Public

On-Ramping/Re-Injection via Direct L2 Benefits Leverages built in network redundancy No special router configurations New protected customer prefixes can be dynamically on-ramped. No new configuration required. Drawbacks Harder to avoid routing loops. Special care of BGP announcements and route distribution is required. Difficult to mitigate customer to customer attacks especially if both customers are on same router. Static in nature, not easily re-configured/scaled Page 109 - Arbor Public

Shunts in the Data Center All devices on the same subnet Either TMS-driven or configured in router May use remotely-triggered shunt trick All traffic in core to target goes to the TMS Optionally, you can use VLANs to avoid loops Bypassing the modified router is trivial with VLANS and.1q trunking Page 110 - Arbor Public

I C S T S t y s S S R I t r c s r C S S S Hosting/SP Data Center Backbone Cisco IOS Router Backbone P r p y P w p C a 5 0 Arbor TMS Cisco Catalyst Switch GEnet Arbor TMS Alert Arbor Peakflow SP Firewalls Switches Target Internal Network Web, Chat, Email, etc. DNS Servers Page 111 - Arbor Public

Shunts in the Data Center Arbor Peakflow SP IDC Edge ISP A IDC Edge ISP B Arbor TMSs (Cleaning Server Center) Farms Access Aggregation Core Edge BGP Peering for Diversion Dirty Traffic Clean Traffic NetFlow Page 112 - Arbor Public

Shunts in the IP Core: GRE Injection Core routes target IP to the TMS Either TMS-driven or configured in router May use remotely triggered shunt trick All traffic in core to target goes to the TMS Injection into GRE tunnel Bypassing the modified core routing GRE starts on TMS-attached router, terminates on CPE, which has clean routing to target Page 113 - Arbor Public

Shunts in the IP Core: GRE Injection Attack 1. BGP: I m next-hop for 1.1.1.1 3. Rerouting to 1.1.1.1 2. Redistribution into Core TMS (2.2.2.2) 4. Injection to Target GRE (Preconfigured) Target (1.1.1.1) Page 114 - Arbor Public

Shunts with MPLS VPNs Easy to deploy: Core remains untouched, injection VPN preconfigured VPN invisible to core No performance impact No need to touch CPE But: MPLS VPN required on core Page 115 - Arbor Public

MPLS VPN Shunt Attack 1. BGP: I m next-hop for 1.1.1.1 3. Rerouting to 1.1.1.1 2. Redistribution into Core TMS (2.2.2.2) VPN 4. Injection to VPN MPLS VPN (Preconfigured) VPN Target (1.1.1.1) Page 116 - Arbor Public

Packet Cleaning Issues Shunting the Packets Cleaning the Packets Page 117 - Arbor Public

Packet Cleaning in the Core: The Cleaning Center Arbor TMS (Customer A) Arbor TMS (Customer B) Cleaning Center NOC SP Core Core ASBR Peering SP1 Core Core Arbor Peakflow SP PE Peering SP2 NetFlow PE Dirty Traffic Clean Traffic CE Customer A Out-of-Band WAN Connection CE Customer B Page 118 - Arbor Public

Scaling a Cleaning Center: Clustering Topology with ECMP/CEF Load-leveling Attack Load-Leveling Router up to 16 TMSes, 160gb/sec w/n7k TMS TMS TMS TMS Mitigation Cluster TMS TMS Target host TMS Arbor TMS IDMSes Multiple Cleaning Centers via Anycast Page 119 - Arbor Public

Backbone Option: Cleaning Centers Question is: how many? Most national providers have decided to start with two Geographic redundancy Adequate incoming bandwidth in key locations Limit the backhaul of traffic across expensive links Once you decide on where, then the hard part! Page 120 - Arbor Public

Packet Spoofing What can be spoofed? IP Any field in a packet header (well, almost) Spoofing most often happens in combinations with several fields being spoofed Spoofing is used to: Hide the source so the attacker or resource is not revealed Bypass security masquerading as valid packets Spoofing the real target getting others to take out the target TCP HTTP DST SRC Prtcl CRC Port Port SYN FIN SSL GET URL CGI www.victim.com Page 121 - Arbor Public

TMS Mitigation Processing DDoS attacks consist of undesirable traffic mixed in with some amount of desirable traffic Undesirable traffic may come in large quantities or it could come shaped in a way designed to disrupt normal processing The TMS allows desirable traffic through while lowering the impact of undesirable traffic The TMS uses various countermeasures defense mechanisms to target and remove the most egregious attack traffic to allow the network to continue operating Different countermeasures are designed to stop different types of attack traffic The countermeasures as a whole provide defense in depth mitigation Page 122 - Arbor Public

DDoS Attacks DDoS attacks can consist of just about anything Large quantities of raw traffic designed to overwhelm a resource or infrastructure Application specific traffic designed to overwhelm a particular service sometimes stealthy in nature Traffic formatted in such a way to disrupt a host from normal processing Traffic reflected and/or amplified through legitimate hosts Traffic from compromised sources or from spoofed IP addresses Pulsed attacks start/stop attacks DDoS attacks can be broken out by category Page 123 - Arbor Public

TCP Stack Flood Attacks Description Flood a certain aspect of the TCP connection process to keep the host from being able to respond to legitimate connections May be spoofed or non spoofed Peakflow SP Detection Capabilities Misuse TCP SYN, RST, Total Traffic detection Peakflow TMS Mitigation Countermeasures TCP SYN authentication, zombie army, white list/ black list Common names TCP SYN, TCP FIN, TCP RST, TCP Flags Page 124 - Arbor Public

Generic Flood Attacks Description Flood of traffic for one or more protocols or ports Designed to look like normal traffic Reflection attacks May be spoofed or non spoofed Peakflow SP Detection Capabilities Misuse UDP, ICMP, Total Traffic detection Profiled anomaly detection for managed object Peakflow TMS Mitigation Countermeasures White list/blacklist, zombie army, baseline enforcement, rate limiting, payload filtering Common names Ping Attack, Smurf Attack, reflection attacks, UDP flood, Stream, dc++, blackenergy Page 125 - Arbor Public

Fragmentation Attacks Description A flood of TCP or UDP fragments are sent to a victim overwhelming the victim s ability to re-assemble the streams and severely reducing performance Fragments may also be malformed in some way May be a result of a network mis-configuration Peakflow SP Detection Capabilities Misuse IP Fragment detection Peakflow TMS Mitigation Capabilities White list/blacklist, zombie army Common names Teardrop, Targa3, Jolt2, Nestea Page 126 - Arbor Public

Application Attacks Description Attacks designed to overwhelm components of specific applications Commonly seen against HTTP, DNS and SIP in particular May be stealthy by mixing with a much higher traffic volume on the same protocol/port Peakflow SP Detection Capabilities Requires TMS systems deployed in span mode doing appid and feeding SP systems with application level managed objects defined. Peakflow TMS Mitigation Countermeasures HTTP malformed, HTTP rate limiting, HTTP payload filtering, SIP malformed, SIP request rate limiting, DNS authentication, DNS malformed, Payload regex filtering, Regex filtering Common names HTTP GET floods, SIP Invite floods, DNS amplification attacks, Page 127 - Arbor Public

Connection Attacks Description Attacks that maintain a large number of either ½ open TCP connections or fully open idle connections with the victim impeding new connections from forming Peakflow SP Detection Capabilities Limited Peakflow TMS Mitigation Capabilities TCP syn authentication, TCP Idle Reset Common names TCP Idle attack Page 128 - Arbor Public

Vulnerability Exploit Attacks Description Attacks designed to exploit a vulnerability in the victim s operating system Some are single packet or very low level attacks Many of these are obsolete in modern operating systems Peakflow SP Detection Capabilities Limited: ATF based fingerprint detection Peakflow TMS Mitigation Capabilities Limited: Malformed HTTP, SIP, DNS, White list/blacklist The most effective method of stopping these attacks is to patch hosts on your network Common names Most Worms and Trojans, Land, Xmas Tree, Ping of Death, Targa3, Kiss of Death, Nuke Page 129 - Arbor Public

Putting All This Together to Stop DDoS The Core Functional Components of an Anti-DDoS Packet Scrubber: Destination detection Source verification (via anti-spoofing) Source detection (via anomalies) Source/attack blocking/filtering Attack termination detection Page 130 - Arbor Public

Packet Cleaning via Shunts Advantages: Not on critical path during normal operation Anomaly-based detection with baselining Optimized for high-performance blocking Is resistant to state limitations of most other devices Limitations: Not designed to stop single-packet exploits, but regexp countermeasure can be used if the exploit packet signature is known Inherent assumption of a destination to be protected i.e., a server Resource utilization: finite resources in the scrubber complex (160gb/sec max per cluster with Cisco CEF-based load-leveling of 16 paths w/n7k multiple clusters via IPv4 anycast addressing) Requires up-front network engineering to implement Page 131 - Arbor Public

Q&A Page 132 - Arbor Public

Thank You Roland Dobbins <rdobbins@arbor.net> Solutions Architect +66-83-266-6344 BKK mobile +65-8396-3230 SIN mobile