PlanetSeer: Internet Path Failure Monitoring and Characterization in Wide-Area Services

Transcription

1 PlanetSeer: Internet Path Failure Monitoring and Characterization in Wide-Area Services Ming Zhang, Chi Zhang Vivek Pai, Larry Peterson, Randy Wang Princeton University

2 Motivation Routing anomalies are common on Internet Maintenance Power outage Fiber cut Misconfiguration Anomalies can affect end-to-end performance Packet losses Packet delays Disconnectivities 2

3 Background Anomaly detection and diagnosis are nontrivial Asymmetric paths Failure information propagation Highly varied durations Limited coverage 3

4 Contributions New techniques for Anomaly detection Anomaly isolation Anomaly classification Large-scale study of anomalies Broad coverage High detection rate, low overhead Characterization of anomalies End-to-end effects Benefits to host service 4

5 Outline State of the Art PlanetSeer Components MonD passive monitoring ProbeD active probing Anomaly Analysis Loop-based anomaly Non-loop anomaly Bypassing Anomalies Summary 5

6 State of the Art Routing messages BGP: AS-level diagnosis IS-IS, OSPF: Within single ISP Router/link traffic statistics SNMP, NetFlow: proprietary End-to-end measurement Ping, traceroute 6

7 End-to-End Probing All-pairs probes among n nodes O(n^2) measurement cost Not scalable as n grows 7

8 Key Observation Combine passive monitoring with active probing Peer-to-Peer (P2P), Content Distribution Network (CDN) Large client population Geographically distributed nodes Large traffic volume Highly diverse paths The traffic generated by the services reveals information about the network. 8

9 Our Approach Host service CDN Components Passive monitoring Active probing Advantages Low overhead Wide coverage Client R1 R2 C A B 9

10 MonD: Anomaly Detection Anomaly indicators Time-to-live (TTL) change Routing change n consecutive timeouts (n = 4 in current system) Idling period of 3 to 16 seconds most congestion periods < 220ms 10

11 ProbeD Operation Baseline probes When a new IP appears From local node Forward probes When a possible anomaly detected From multiple nodes (including local node) Reprobes At 0.5, 1.5, 3.5 and 7.5 hours later From local node 11

12 Number of Groups US (edu) US (nonedu) ProbeD Groups 353 nodes, 145 sites, 30 groups According to geographic location One traceroute per group Canada Europe Asia & MidE Other 12

13 Estimating Scope Local ProbeD Client Remote ProbeD r a r b r c r d Which routers might be affected? Routers which possibly change their next hops Traceroutes from multiple locations can narrow the scope 13

14 Tier Coverage Path Diversity 100% 80% 60% Core Edge 40% 20% 0% Tier 1 Tier 2 Tier 3 Tier 4 Tier 5 22 ASes 215 ASes 1392 ASes 1420 ASes ASes Monitoring Period: 02/ /2004 Unique IPs: 887,521 Traversed ASes: 10,090 14

15 Confirming Anomalies Reported anomalies 2,259,588 Conditions Loops Route change Partial unreachability ICMP unreachable Very conservative confirmation Undecided 22% Anomaly 12% Nonanomaly 66% 15

16 Confirmed Anomaly Breakdown Confirmed anomalies 271,898 2 per minute 100x more Temp loop 1% Temp anomalies Inconsistent probes Persist Loop 7% Other Outage 23% Temp Anomalies 16% Fwd Outage 9% Path Change 44% 16

17 Scope of Loops 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% 1% persist loops cross ASes 15% temp loops cross ASes How many routers or ASes are involved? Persistent Temp Temp loops involve more routers than persistent loops 97% persistent loops and 51% temp loops contain 2 hops 17

18 Distribution of Loops 50% 45% 40% 35% 30% 25% 20% 15% 10% 5% 0% Many persistent loops in tier-3, few in tier-1 Worst 10% of tier-1 ASes implications for largest ISPs 20% traffic Tier 1 Tier 2 Tier 3 Tier 4 Tier 5 35% persistent loops Persistent Temp Traffic 18

19 Duration of Persistent Loops 60% 50% 40% 30% 20% 10% 0% <0.5 hrs <1.5 hrs <3.5 hrs <7.5 hrs >= 7.5 hrs How long do persistent loops last? Either resolve quickly or last for an extended period 19

20 fraction Scope of Forward Anomalies How many routers or ASes are affected? 60% outages within 1 hops 78% outages within 2 ASes 57% changes within 2 ASes hops 75% outages and 68% changes within 4 hops change outage 20

21 fraction Location of Forward Anomalies hops How close are the anomalies to the edges of the network? 44% outages at the last hop 72% outages and 40% changes within 4 hops change outage 21

22 Distribution of Forward Anomalies 50% 45% 40% 35% 30% 25% 20% 15% 10% 5% 0% Tier 1 Tier 2 Tier 3 Tier 4 Tier 5 Change Outage Traffic Which ASes are affected? Tier-1 ASes most stable Tier-3 ASes most likely to be affected 22

23 Overlay Routing Use alternate path when default path fails source destination intermediate 23

24 fraction Bypassing Anomalies bypass ratio How useful is overlay routing for bypassing failures? Effective in 43% of 62,815 failures, lower than previous studies 32% bypass paths inflate RTTs by more than a factor of two 24

25 Summary Confirm 272,000 anomalies in 3 months Persistent and temporary loops Persistent loops narrower scope, either resolve quickly or last for a long time Path outages and changes Outages closer to edge, narrower scope Anomaly distribution Skewed. Tier-1 most stable. Tier-3 most problematic. Overlay routing Bypasses 43% failures, latency inflation 25

26 More Information In the paper More details about anomaly characteristics End-to-end impacts Classification methodology Optimizations to reduce overheads & improve confirmation rate 26