Debugging With Netalyzr

Transcription

1 Debugging With Netalyzr Christian Kreibich (ICSI), Nicholas Weaver (ICSI), Boris Nechaev (HIIT/TKK), and Vern Paxson (ICSI & UC Berkeley) 1

2 What Is Netalyzr?! Netalyzr is a comprehensive network measurement and debugging tool built as a Java Applet located at A suite of custom servers hosted primarily using Amazon EC2! Multiple properties checked! NAT properties: Presence. Port Renumbering, DNS proxies! IP properties: Blacklist membership! Network properties: Latency, Bandwidth, Buffering, IPv6 capability, Path MTU! Service properties: Outbound port filtering and proxies on a large list of services 2

3 What Is Netalyzr?! Further properties:! HTTP properties: Hidden HTTP proxies or caches. Correct operation of proxies and caches! DNS properties: Latency of lookups, glue policy, DNS wildcarding, name validation, DNS MTU, DNSSEC readiness! Host properties: Browser identity, Java version, clock accuracy! Over 120,000 executions to date! A technical report is available! Findings include path MTU holes and MTU-related ICMP failures, chronic overbuffering, DNS wildcarding, DNS MitM attacks from both malcode and ISPs 3

4 Netalyzr Requirements! Requires a Java capable Web Browser! Javascript highly useful! Some personal firewalls block unknown-port UDP by default! As do some corporate firewalls! Prevents latency, bandwidth, MTU, Fragmentation, UDP port filtering, and some DNS tests from operating! On a Mac, Allow incoming connections to the web browser if prompted! Some tests require trusting the applet! But even untrusted, most tests still operate within Java s Same Origin policy 4

5 Results Summary! After running the test suite, a single results page is displayed! Normal results are automatically compressed to reduce visual clutter! Can be expanded with the +/- icon! Anything interesting is highlighted in the top in orange or red depending on severity! Results URL can be cut & pasted for forwarding to others! Very useful for Mom-level debugging as well! An Under the Hood view exists:! Client side debugging transcript link: Pretty much everything we do is exported to the debugging output 5

6 Some Problem Areas Of Interest! Unknown outbound port filtering/proxies! Broken HTTP proxies & caches! Fragmentation! Path MTU issues! Blacklists! DNS Issues! Overbuffering 6

7 Unknown Outbound Port Filtering & Proxies...! To check for outbound filtering or proxies, Netalyzr connects to custom echo servers! Echo server returns IP/Port used for the connection! Response indicates status: Port Filtered Redirected through a proxy Terminated by application-aware proxy/filter Port unfiltered! Protocol aware filtering is acceptable if you know about it!! A lot of SIP-aware proxies running around! Firewalls commonly enforce DNS semantics! Save yourself a lot of grief: run an ssh server on port 443 back at your home institution 7

8 HTTP Proxies and Caches! Do they exist at all?! Either mandatory or configured in the web browser! Detected by sending requests which trip up proxies:! Including IP changes, header changes, illegal requests, repeated fetches of data with different cache-control, etc...! Far more likely to be known if on a corporate network! But can be annoyingly common at random hotspots etc! Do they cache or transcode data?! If they exist, they are quite commonly broken: Caching data which should not be cached! Can cause significant privacy leakages and web-site breakages 8

9 Do they improperly interpret the hostname?! An in-path HTTP proxy must not interpret the HTTP host header to decide where to route the request! Otherwise, this enables Java & Flash to grossly violate the same origin policy! But some in-path proxies are broken:! Test by generating a request to our web server, but the host field is for A broken proxy will reinterpret the request and forward it to Google!! Mildly bad on hotspot networks! Suggests that the administrators aren t keeping things upto-date! Really bad on corporate networks! Can often be used to pillage internal web sites using Java or Flash 9

10 Fragmentation Problems! Applet sends and receives fragmented UDP traffic! A ~2000B packet that has a small echo! A small request which generates a ~2000B response! About 8% can t send or receive fragments! Common cause: broken firewalls/nats! Stateless firewall rules that include a deny all for any unmatched packet! Without a rule that says all but the first fragment are allowed! Stupid NATs! Big problem for DNSSEC! DNSSEC effectively requires fragmented UDP traffic 10

11 Path MTU issues! Can Ethernet MTU - 1B packets be sent over UDP?! If not, you re probably running Linux: Linux does path MTU discovery for UDP, setting DF on UDP packets that fit onto the link s MTU...! Which then passes through a lower than Ethernet MTU link...! This is a Linux bug: " Workaround #1: Disable all path MTU discovery! " Workaround #2: Lower the interface s MTU to ~1400B! What is the path MTU from our server to the client! And are the ICMP too-big messages sent properly?! If not, the tunnel endpoint (eg, the other side of the VPN tunnel) or an intermediary firewall is broken. (unfortunately common) 11

12 Blacklists...! Netalyzr checks the external IP against multiple blacklists! Spamhaus PBL: Has someone designated this IP as not supposed to send ! Could result in SMTP problems! Spamhaus XBL: Has this IP been reported as exploited! If not on a shared hotspot, this should be investigated further! SORBS DUHL: Has the ISP reported this as a should be dynamic IP! Some mail servers use this to guide reception of SMTP! TOR exit node: Are you coming to us through Tor? 12

13 DNS Issues! DNS port randomization! If untrusted, just is a boolean result! If trusted, reveals the request ports used! DNS MTU! What is the advertised DNS MTU! What is the actual DNS MTU! If the actual MTU is less this can cause big problems with DNSSEC: Instead of immediately failing over to TCP when truncation is set, a large reply will take a timeout first! Commonly due to a no fragment or DNS == 512B firewall! DNS truncation! Can the resolver use TCP as well as UDP! Again, this is effectively necessary for DNSSEC 13

14 More DNS checks! Can the host directly query DNS?! If so, does DNS pass through a proxy or firewall?! If so, does the network allow EDNS requests of various sizes to pass?! Failures on this are due to common (old) firewall problems:! No EDNS == Firewall doesn t understand EDNS! EDNS medium & large fails == Firewall mistakenly assumes DNS = 512B! EDNS large fails == Network can t handle fragments 14

15 Overbuffering! Attempts to saturate first the uplink and then the downlink of the network! Measure the latency of packets when the system is quiet vs when the buffer is stressed! Causes the network to fail the Walk and chew gum test:! A full rate TCP flow (or multiple flows) will fill the buffer and cause interactive TCP sessions to experience massive lag! Only a few solutions:! Get the cable-modem/dsl modem vendors to design the buffers right! Get the NAT vendors to add remote active queue management! Pay for a higher tier of service! Just don t expect good websurfing or VoIP when running (BitTorrent, photo uploads, etc) 15

16 This Problem Is Chronic! Inside the Netalyzr 16Mb/s 4Mb/s Upload Bandwidth 1Mb/s 256Kb/s 64Kb/s 16Kb/s Uplink Buffer can introduce >1s delays. This is a big problem for P2P programs like BitTorrent 1KB 4KB 16KB 64KB 256KB 1MB 4MB 16 Inferred Buffer Capacity

17 We do consider further tests...! If a problem is significant/interesting, we ll consider adding new tests! DNS tests driven by DNSSEC deployment issues! Currently building a significant suite of IPv6 tests 17