Practical Advice for Small and Medium Environment DDoS Survival

Practical Advice for Small and Medium Environment DDoS Survival Chris "Mac" McEniry Sony Network Entertainment @macmceniry November 8 13, 2015 Washington, D.C. www.usenix.org/lisa15 #lisa15 1

Practical Advice for Small and Medium Environment DDoS Survival Chris "Mac" McEniry Sony Network Entertainment @macmceniry Branson Matheson Cisco @sandinak November 8 13, 2015 Washington, D.C. www.usenix.org/lisa15 #lisa15 2

Who are we? Chris McEniry Network Engineer Sony 15+ Year IT Pro Mgr Defcon network Trainer and Speaker Branson Matheson Tech Lead - Cisco 15+ year IT Pro Mgr ShmooCon Network Trainer and Speaker Geeks who like to do fun stuff with cool networks

Who is this for? Online Services 1-3 points of presence 1Gbps-20Gbps per POP 4

Topics Defining a DDoS Knowing Attack Categories Defense Strategy 5

What's a Distributed Denial of Service? Attack intended to disrupt access to your service with the following properties: Internet based (multiple sources) Sufficient volume targetting a critical resource Most obvious: Pipes Others: Connection, state tables, CPU, etc 6

DDoS getting larger 7

Renting is easy 8

"Know Thyself" 9

Know Thyself What does your traffic look like Are you using all the protocols and ports that are likely to be open? Using UDP? Only using TCP ports? Only using HTTPS? Does everyone need to be able to access you? (E.g. do you really need traffic from China?) 10

Know Thyself What does recovery look like Does it look different than an attack? What does it mean to your business model? 11

Know Thyself How much downtime can you take How much does it cost? What parts of your site or infrastructure can be down? In what order are you willing to sacrifice functionality to keep some availability? 12

Know Thyself Your responses to the above will dictate your DDoS Mitigation Strategy. 13

Know Thy Adversary? 14

Know Thy Adversary? Knowing the attacker thinking can be irrelevant* Knowing the attacker methodologies is useful * Ok, not completely, but less than you think. 15

Know Thy Adversary? There's probably multiple attackers Multiple motivations Majority are making a name or statement Some are smarter than you * Ok, not completely, but less than you think. 16

Know Thy Adversary? Can you really game theorize all possible reasoning and vectors? * Ok, not completely, but less than you think. 17

Know Thy Adversary? The mitigations are largely the same And largely determined by "Know Thyself" The hows - not the whys 18

Know Thy Adversary? Places to look for attack types and volumes Verisign DDoS Trends Report Neustar DDoS Attacks and Protection Report Arbor Atlas - DDoS Digital Attack Map ISC Storm Center And many other reports 19

Attack Categories 20

Attack Categories Three Dimensions What Network Layer (Vertical)? What Network Point (Horizontal)? Type of Source? 21

Attack Categories: Vertical What layer of the network stack? "Network": Layer3/4 - IP/UDP/ TCP/etc "Application": HTTP 22

Attack Categories: Horizontal How deep are they attacking you? Your border? Your application servers? Your ISP's infrastructure? 23

Attack Categories : Source Direct or indirect? Am I (or my botnet) sending the attack? Am I coercing others into sending the attack? 24

Common Attacks Network Attacks Target Infrastructure - Pipe Capacity, State Tables, Network Device CPU, etc. IP Fragmentation, SYN Flood, Christmas Tree Floods 25

Common Attacks "Application" (and here it's HTTP centric) Target the application servers and consume their CPU, connections, threads, etc HTTP Post, Slow Read 26

Common Attacks Reflection and Amplification Attacks Requires a small amount of input to send a large amount of traffic to target Uses single/multiple third parties to send the actual traffic https://www.internetsociety.org/sites/default/files/01_5.pdf 27

Common Attacks Application Amplification Attacks are starting to happen Being able to change a small amount of HTTP (e.g. javascript) traffic to trigger many requests to the target Great Cannon 28

Common Attacks Intermediaries ISPs Dependent SaaSes (e.g. Image Converters, Code Hosting) Dependent JS libraries 29

Lifecycle Small attackers / Scanners - Firewall is able to Block Attackers get bigger - Move to stateless ACLs to solve Network attacks Attackers use reflections - Move to vendor who can sustain bandwidth Attackers turn to application - IPS/WAF Attackers get/sacrifice a large bonnet - Vendor Application Proxies 30

Defense Strategy 31

How to defend? Its all about reducing the bad traffic which you receive Fundamentally, 2 questions: What effort at each level are you going to use? Are you going to do it yourself? or have someone else do it? 32

Self Defense: Network Level Any packet filtering device defends at the Network level However, that packet filtering device can become a bottleneck itself ACLs, Firewalls, iptables, "not listening" The further upstream you can push these, the better Some ISPs may apply ACLs for you BGP null route advertisements 33

Self Defense: Network/Application Level Intrusion Prevention Systems (IPS) Since can do deep packet inspection, can be considered application level Matches patterns (e.g. regex) and denies But can also do network level protections (SYN Proxy) Likely to be able to handle more bad traffic than similarly classed firewalls Likely to be able to incorporate reputation filters 34

Self Defense: Application Level Web Application Firewalls See above - IPS Inspect flows in an application context and react Deny bad patterns Rate limit flows Incorporate reputation filters 35

Self Defense: Pros You will have a better understanding of your own traffic Always active Lower coordination on mitigation 36

Tipping the scale (Self Defense: Cons) Pipes can only handle so much At some point, can't even handle the incoming traffic on your pipes, regardless of internal protections Consider a vendor solution (beyond ISP cooperation) 37

Stateful vs Stateless Rough comparison: Switch/Router ACLs vs Firewalls Rules of thumb: Use stateless to control destination and IP reputation "Of the 64K tcp ports, only allow in to port 443/tcp" "Don't let in Country X" Use stateful to control source (outside IP reputation) "Only allow my offices to hit my port 443/tcp" 38

Vendor Solutions Two main methods: Network Scrubbers Application Proxies 39

Network Scrubbers Route all incoming traffic, at the IP level, through scrubber They become your BGP announcer Traffic is then sent to you: either direct connection or encapsulated and retransmitted over the Internet Can be always-on or on-demand Even under always-on, have a static stance (e.g. ACLs) and an under-attack stance (actual mitigation gear) 40

Application Proxies Direct traffic to their endpoints by updating DNS Become the endpoint of connection for client connections Inspect, and then proxy those connections back to you 41

Application Proxies Typically, always on Application specific: "Works if they speak your protocol" - Typically DNS, HTTP/HTTPS, even SMTP/IMAP But by nature will front Layer3/4 attacks 42

Vendor Pros 1Tbps+ Internet capacity and scrubbing capacity 24x7 staff focused on DDoS Up to date mitigation software/hardware 43

Vendor Cons Over-subscription: Other customer's attacks can affect you Vendors becomes your ISP to a degree Any of their carrier issues become your issues Added bonus: No longer a direct carrier relationship. More coordination 44

Vendor Gotchas What is mitigation going to do to your normal traffic? Most stances are based around browser based web traffic. Native phone apps, embedded devices (IoT) may not like that 45

Vendor Gotchas Cannot stress how important that connection back to you is How hidden is your origin? Application to either Network Scrubbers or Application Proxies Do your origin endpoints show up in WHOIS? Reverse DNS? Can one scan your actual network and start to determine where those endpoints really are? 46

Items to look for in a Vendor Internet Capacity vs Scrubbing Capacity Points of Presence: Internet facing, and clean traffic to you facing Scrubbing Mechanisms Configuration, Incident, Maintenance Transparency Connection options: GRE or Meet-Me Cross Connects? Pricing Model: Always on? Incoming or just clean traffic? Port fees? 47

Potential Vendors A10 Networks Akamai Amazon AWS Arbor Networks CloudFlare Corero F5 / Silverline Google Compute Engine Imperva / Incapsula Level3 Limelight Neustar Verisign No endorsements and not exhaustive. Strictly a starting point. 48

Which path? Depends on where you are in attacker/defender cat mouse game or your budget (time and money)? Depends on your applications (e.g. are they already on a vendor CDN?) Depends on your whole system (e.g. think about all of your dependencies not just for the running application, but also for your business mechanisms like updating the application) 49

Parting Shot Amplification attacks are the biggest threats right now Don't contribute to the problem Are you running an open NTP server? Do you have any large public DNS records available via UDP? Are you watching your egress traffic? Is it all from you? 50

Chris "Mac" McEniry Sony Network Entertainment @macmceniry Branson Matheson Cisco @sandinak 51