Outline EEC 274 Internet Measurements & Analysis Spring Quarter, 2006 Traffic Measurements Traffic measurements What metrics are we interested in? Measurement and analysis methodologies Traffic characterization Measurement studies & observations 2 Motivations Applications Observe Internet traffic characteristics. Develop reasonable models to understand these characteristics. Failure of traditional mathematical modeling techniques (e.g. Queuing theory). Earlier models deal with issues which are noncritical from the practitioner s point of view. Attempt to close the void between theory and practice. Provisioning network resources (capacity, buffer, etc) How should the network be provisioned to satisfy certain constraints. Constraints may differ with the type of traffic. Obtain characteristic workloads for use in simulations Typical packet sizes Typical flow durations Most commonly used TCP flavors Important for ISPs to formulate policy decisions (Service Level Agreements) Developing techniques to detect network anomalies e.g. Denial of Service attacks. Verify rule of thumb type design guidelines. 3 4 Page 1
Part A. Traffic Measurements Traffic Measurement: Metrics Measurement Methodologies Traffic count Aggregate traffic: # of packets, bytes within a time window (bin) Packet/byte count broken down by protocol, applications, etc. Packet level Packet size distribution Inter arrival time Flow level Packets per flow Number of active flows Traffic Dynamics Temporal variation (time of day, day of week effects) Active vs. Passive monitors Lets start with passive measurements Design Challenges Collection of detailed traffic statistics from heterogeneous network links. Non interference with the measured network (nonintrusiveness). Obtaining a global view of the monitored network from a reasonable number of monitoring points. 5 6 Existing Options NetFlow Packet capturing at the edge tcpdump. http://www.tcpdump.org/ Some routers have built in monitoring capabilities. Netflow Cisco routers. SNMP: 5 minute average Core: IP monitoring infrastructure Optical splitter Developed by Darren Kerr and Barry Bruins at Cisco Systems in 1996 The value of information in the cache was a secondary discovery Initially designed as a switching path NetFlow is now the primary network accounting technology in the industry Sampled NetFlow a Cisco innovation NetFlow version 9 an IETF standard Answers questions regarding IP traffic: who, what, where, when, and how 7 8 Page 2
What is a flow? Creating Export Packets Defined by seven unique keys: Source IP address Destination IP address Source port Destination port Layer 3 protocol type TOS byte (DSCP) Input logical interface (ifindex) Exported Data 9 Enable NetFlow Traffic PE Export Packets Approximately 1500 bytes Typically contain 20-50 flow records Sent more frequently if traffic increases on NetFlow-enabled interfaces UDP NetFlow Export Packets Core Network Collector (Solaris, HP-UX, or Linux) Application GUI 10 Inbound traffic only Unidirectional flow NetFlow Principles Accounts for both transit traffic and traffic destined for the router Works with Cisco Express Forwarding (CEF) or fast switching Not a switching path Supported on all interfaces and Cisco IOS Software platforms Returns the sub interface information in the flow records 11 Network Layer Applications NetFlow Features Access Attack Mitigation User (IP) monitoring Application monitoring Aggregation Schemes (v8) show ip cache flow command Arbor Networks Distribution Billing Chargeback AS Peer Monitoring NetFlow MPLS Egress Accounting BGP Next-hop (v9) Multicast NetFlow (v9) NetFlow Uses Core Traffic Engineering Traffic Analysis MPLS Aware NetFlow (v9) BGP Next-hop (v9) Sampled NetFlow Distribution Billing Chargeback AS Peer Monitoring Access Attack Mitigation User (IP) monitoring Application monitoring NetFlow Aggregation MPLS Egress Schemes (v8) Accounting show ip cache BGP Next- flow command hop (v9) Arbor Networks Multicast NetFlow (v9) 12 Page 3
SNMP IPMON Approach Simple Network Management Protocol (SNMP) Standard operation and maintenance protocol for the Internet (analogous to SS7 for Telephone Network) SNMP management framework Architecture that defines how to move data Defines Data definition language Management information (MIB) Protocol Security and administration Bottom line: Gives average link utilization data, e.g., total traffic volume averaged over 5 minutes [01IPMON] C. Fraleigh, C. Diot, B. Lyles, S. Moon, P. Owerzarski, and K. Papagiannaki, Design and Deployment of a Passive Monitoring Infrastructure, Passive and Active Measurement Workshop, Apr 2001. [03FML] C. Fraleigh, S. Moon, B. Lyles, C. Cotton, M. Khan, D. Moll, R. Rockell, T. Seely, and C. Diot, Packet level Traffic Measurements from the Sprint IP Backbone, IEEE Network, 2003. Insert optical splitter on links in multiple POPs in Sprint s Tier 1 IP backbone network Collect and timestamp all IP headers (44 bytes) Collect routing information (IS IS, BGP) Transfer data to lab for off line analysis 13 14 IPMON architecture Advantages Backbone links Backbone Peering points Router Access Access Access Router Router Router customer customer customer Transparent to network Data from an operational IP backbone Full TCP/IP headers (not http) Timestamps allow correlating packets on different links for traffic dynamics analysis Traces archived for future use Analysis platform (located @ ATL) 15 16 Page 4
Drawbacks Measurement Facilities Requires deployment in operational network Expensive and time consuming to deploy Difficult to deploy in each POP POPs evolve too fast : ) Does not scale Technology limitations (PCs, disks, etc.) Only perform off line analysis 44 bytes is sometimes not enough IPMON System Collects packet traces from fiber in POPs Data Repository Large tape library to archive data Analysis Platform 17 nodes computing cluster for off line analysis SAN for fast access to traces 17 18 IPMON System Clock synchronization Requirements Support OC 3 to OC 48 data rates Global clock synchronization Architecture Linux PC POS/ATM PCI network interface (DAG) Large RAID disk array IPMON Linux PC NTP synchronized DAG cards Use embedded onboard 16MHz clock to generate packet timestamps Initialize with the value of the system clock Synchronize on 1pps signal from GPS Lab tests results DAG cards synchronized within 2usec 19 20 Page 5
Part B. Traffic Characterization Other Projects Traffic Observations OC3MON (MCI) Passive monitor designed for OC3 links (155 Mbps). NetScope (AT&T) A set of tools for traffic engineering in IP backbone networks. Network Analysis Infrastructure (NAI) Performance of vbns (very high speed Backbone Network Service) and Abilene networks. Commercial tools Niksun s NetDetector and NikScout s ATM Probes. Case Study #1: Link utilization, Per hop queuing delays TCP flows only Trace from 10am, August 9th, 2000, 24 hours San Jose POP Web-out Peer-out Web-in Peer-in 21 22 Why these results Link Utilization: bandwidth High level observation (Step 2) Necessary to have a global picture of what an IP network looks like Give directions for further research Early to generalize yet Show how important traffic analysis: often different from the common thinking discuss consequence on the way we engineer networks, or on the future of QoS, Traffic engineering, etc. 23 24 Page 6
Link Utilization: applications Link Utilization: emerging applications 25 26 Link Utilization: protocols Link utilization: packets 27 28 Page 7
Link utilization: flows Packet size cumulative distribution 29 30 Delay vs. time Delay distribution 31 32 Page 8
Traffic dynamic Where does the traffic come from Between any two POPs: Traffic Matrix For each ingress POP : identify traffic to each egress POP further analyze this traffic What is the volume of traffic? What are the traffic patterns? How to design traffic matrices City A City B City C City A City B City C Measure traffic over different timescales Divide traffic per destination prefix, protocol, etc. 33 34 POP to POP Traffic Matrix Why TCP flows analysis? TCP is the most frequent protocol ( 80 %) in charge of fairness, congestion control liveness of the network In deep analysis of TCP behavior (loss, congestion, delays, characterization, mapping with routers mechanisms) Give directions for future research (TCP improvements, resource control and management) 35 36 Page 9
TCP flows TCP flow size (packets) TCP flows are: Identified by usual five tuple Measured between SYN and FIN RTT measured between SYN and ACK RTT SYN SYN-ACK ACK 37 38 Packets sizes distribution (TCP) TCP flow duration distribution Percentage 70 60 50 40 30 Avg : 359 bytes Min : 40 bytes Max : 1500 bytes Percentage 35 30 25 20 15 avg = 12 s min = 0 max = 1621 s 20 10 10 5 0 0 200 500 800 1100 1400 0 size (bytes) 0 5 10 15 20 25 30+ 39 duration (s) 40 Page 10
Percentage 10 9 8 7 6 5 4 3 2 1 TCP flows RTT Avg = 386 ms Min = 5 ms Max = 3.4 s (TCP timeout) 0 0 100 200 300 400 500 600 700 800 900 RTT (msec) 41 TCP loss (retransmission) statistics 11.52 % of TCP flows experience 1 loss or more. 3.52 % of TCP packets are lost. Note (before I forget): more than 80% of TCP connections do not leave slowstart 42 Elephants and Mice Behavior Elephants and Mice Behavior 1st granularity level: prefix mask of 8 bits split heaviest POP to POP stream into substreams equivalent to aggregating all packets with same 8 bit prefix into one stream top 10% make up 82% of traffic 2nd granularity level: prefix mask of 16 bits within mask 8 substreams subdivide an elephant of mask 8 streams top 10% make up 97% of traffic 43 44 Page 11
Measurement Studies MCI Study Daily and weekly effects Wide Area Internet Traffic Patterns and Characteristics Thompson, Miller, Wilder, MCI Telecommunications, 1997. One of the first studies of commercial backbone traffic. Used the OC3MON traffic monitor described earlier, at two locations on MCI s commercial backbone. Characterize traffic on timescales of 24hrs and 7 days in terms of traffic volume, flow volume, flow duration, packet sizes, traffic composition (by protocol, application). Two links monitored. Domestic and International. Traffic volume shows a clear diurnal pattern, with traffic tripling from 06:00 through 12:00 noon EDT. Traffic decreases by about 25% during the weekend. The two directions of the monitored link are not symmetric. 45 46 MCI Study Asymmetry in packet sizes Measurement Studies Flow level Packet sizes are different in the two directions, and are roughly inversely proportional to each other. Understanding Internet Traffic Streams: Dragonflies and Tortoises Brownlee, Claffy CAIDA. Results of flow level measurements from two links: OC3 link (Auckland) and OC12 link (UCSD) Uses an extension of NeTraMet to monitor stream lifetimes. Previous classifications of flows were on basis of size (packets or bytes) Elephants (large transfers) Mice (short transfers) Propose alternate classification of TCP flows on basis of their lifetime. Tortoises (long lasting transfers) Dragonflies (short duration transfers) Here flows are defined as sets of packets traveling in either direction between a pair of end points. 47 48 Page 12
Dragonflies and Tortoises Short Streams Streams lasting less than 15 mins Percentages of streams and bytes. Long Running (LR) streams (>15 mins) account for about 1% of the streams. Very Short streams (<2 sec) account for 40 70 % of streams, showing a diurnal pattern of variation. At UCSD site, 50% of all bytes were in LR streams, while this fraction was 5% for Auckland. Most of these streams are nonweb traffic. Lifetime distributions 45% of streams have lifetimes less than 2 sec. Distributions do not change rapidly over time. 49 50 Short Streams Streams lasting less than 15 mins Tortoises Streams lasting more than 15 mins Byte size distributions Short stream size distributions for UDP, non web TCP and web TCP are considerably different. Distributions are stable over long periods of time Bit rates Longer duration LR streams are low rate (interactive) or high rate (multimedia) with approximately equal frequency. Medium duration LR streams tend to be high rate. (file transfers) UDP streams run at constant bit rates, but these rates may change in response to the application s state (online games). 51 52 Page 13
Tortoises Streams lasting more than 15 mins CAIDA LR stream lifetimes LR stream lifetimes seem to follow a power law distribution. CAIDA: Workload Characterization http://www.caida.org/research/ SD NAP (San Diego Network Access Point) Passive Data Report Collector http://www.caida.org/dynamic/analysis/workload/sdnap/ Example Current applications, sorted by bytes Current source countries, sorted by bytes 53 54 Page 14