Monitoring Cloud Services using Flow-Based Measurements. Idilio Drago

Transcription

1 Monitoring Cloud Services using Flow-Based Measurements Idilio Drago

2 Outline 1 Measurements: Why? What? How? Flow monitoring: Basic concepts Architecture Measurement artifacts Applications: Threat detection Performance monitoring

3 Why measure the network? 2 Traffic engineering e.g., understand, predict and plan the behavior of the network Accounting e.g., how much resources has been used per user? Network security e.g., which machines are compromised in the network? Performance monitoring e.g., why is the application slow? etc.

4 What to measure? 3 Delay (one-way, round-trip) Delay variation (jitter) Throughput (average, peak) Usage per protocol and application Packet loss... Direct measurements vs indirect inference

5 How to measure? 4 Active measurement Inject traffic on the network Examples: Ping, Traceroute Passive measurement Don t generate traffic, but only observe production traffic Packet capture, log files etc

6 Packet measurements: Challenges and limitations 5 Capture: High-speed packet processing without packet loss Recording: 10 Gb/s (duplex) 200 TB/day 100 disks/day Required storage (TB) Gbps campus /03 23/03 06/04 20/04 04/05 18/05 Time (days) Retrieval: Get only some packets from a pile of data Privacy: Capture on the network is privacy-invasive

7 Packet measurements: Challenges and limitations 5 Capture: High-speed packet processing without packet loss Recording: 10 Gb/s (duplex) 200 TB/day 100 disk/day Required storage (TB) Gbps campus Invasive and prohibitive costs. 600 Long-term 450 archival is unfeasible! /03 23/03 06/04 20/04 04/05 18/05 Time (days) Retrieval: Get only some packets from a pile of data Privacy: Capture on the network is privacy-invasive

8 Overcoming packet measurement limitations 6 Make capture and processing faster: Move to kernel space Dedicated hardware Distributed collection & processing Capture less information: Truncate packets collect only headers, not payload Ignore some packets (sampling & filtering) Ignore individual packets (aggregation)

9 Make capture and processing faster 7 User Space Copying OS Kernel Space Copying Driver... packet rings DMA NIC (HW)... RSS queues Standard Linux network stack 1 SW: Reduce the number of copies, parallelism up to user space etc HW: Offload processing to lower layers (HW) 1 J. L. Garcia-Dorado et. al. High-Performance Network Traffic Processing Systems Using Commodity Hardware.

10 Capture less information 8 Lossy compression used by Tstat and the TimeMachine: Trim flows 1 i.e., record first X B per flow 5 10% of volume, 90 95% of full flows e.g., 450 TB/month 45 TB 1 G. Maier et. al. Enriching Network Security Analysis with Time Travel. In: SIGCOMM 08.

11 Aggregation: Flow export 9 Export flow records to summarize a sequence of packets Lots of other proprietary protocols, export tools (e.g., Tstat) etc st paper on flow measurement 2004 NetFlow v NetFlow-Lite 1990 IETF IA WG 1996 IETF RTFM WG 1999 RTFM 1996 NetFlow patented by Cisco 2002 NetFlow v Flexible NetFlow 2013 IPFIX Internet Standard 2004 IETF IPFIX WG st IPFIX specification

12 Typical flow monitoring setup 10 IPFIX reference architecture Packets Flow Export Protocol Packet Observation Flow Metering & Export Data Collection Data Analysis NetFlow is used for both export protocols and flow meters by Cisco

13 Example of a practical deployment 11 Packets Flow export protocol File, DBMS, etc. Flow probe 1 Flow collector 1 Automated analysis (Appliance) Forwarding device Internet Flow collector 2 Manual analysis Production traffic

14 Flow metering & exporting 12 Packets coming into the observation point Packet capturing Time-stamping Metering process Sampling Filtering Flow cache Exporting process Flow records Why time-stamping should be done first?

15 Flow cache functioning 13 Upon packet arrival: Calculate a hash based on the flow key Check whether hash exists in flow cache: Yes Update a flow cache entry No Create a new flow cache entry Flow cache entries are expired based on: Timeouts: active, idle Natural expiration e.g., observation of a TCP FIN or RST packet... Analysis applications receive flow records

16 What information is exported? 14 Example: IPFIX Information Elements (IE) Link Ethernet Network IP Transport TCP, UDP Application HTTP, DNS, etc. Common IEs Other protocols support specific fields and layers (e.g., NetFlow v5) Flow keys and exported fields are implementation-specific

17 Common IPFIX Information Elements 15 ID Name Description 152 flowstartmilliseconds Timestamp of the flow s first packet. 153 flowendmilliseconds Timestamp of the flow s last packet. 8 sourceipv4address IPv4 source address in the packet header. 12 destinationipv4address IPv4 destination address in the packet header. 7 sourcetransportport Source port in the transport header. 11 destinationtransportport Destination port in the transport header. 4 protocolidentifier IP protocol number in the packet header. 2 packetdeltacount Number of packets for the flow. 1 octetdeltacount Number of octets for the flow. Considering metrics such as delay, usage per protocol and packet loss Is flow-based monitoring direct or indirect?

18 Storage volume 16 Original traffic: 2.1 TB Sampling rate Protocol Exported Storage Reduction 1:1 NetFlow v5 2.1 GB MB 2,301 x 1:1 2.5 GB 1.0 GB 2,100 x 1: GB MB 4,169 x NetFlow v9 1: MB MB 20,212 x 1: MB 20.4 MB 102,941 x 1:1 IPFIX 3.0 GB MB 2,560 x Exported volumes on this table: NetFlow v5 usual fields NetFlow v9 NetFlow v5 + templates IPFIX NetFlow v5 + templates + 64 bit time resolution Compressed formats implemented by nfdump

19 Storage volume 16 Original traffic: 2.1 TB Sampling rate Protocol Exported Storage Reduction 1:1 NetFlow v5 2.1 GB MB 2,301 x 1:1 2.5 GB 1.0 GB 2,100 x 1: GB MB 4,169 x NetFlow v9 1: MB MB 20,212 x 1: MB 20.4 MB 102,941 x 1:1 IPFIX 3.0 GB MB 2,560 x Does it solve the scalability problems of packet recording? Campus example: 450 TB/month 180 GB/month of flow CESNET: 125 SURFnet: 16 GB/day using 1:100 packet sampling

20 How reliable are flow data? 17 Do the exported data reflect the network traffic? We evaluated a set of export devices from popular models: No. Model Modules Software version 1. Cisco Catalyst 6500 WS-SUP720-3B (PFC3B, MSFC3) IOS 12.2(33)SXI5 2. Cisco Catalyst 6500 WS-SUP720-3B (PFC3B, MSFC3) IOS 12.2(33)SXI2a 3. Cisco Catalyst 6500 VS-SUP2T-10G-XL (PFC4XL, MSFC5) + WS-X G IOS 15.0(1)SY1 4. Cisco Catalyst 7600 RSP720-3C-GE (PFC3C, MSFC4) IOS 15.2(1)S 5. Juniper T1600 MultiServices PIC 500 JUNOS 10.4R INVEA-TECH FlowMon

21 Measurement methodology 18 Cisco Catalyst 6500 Control server Internet NetFlow v9 SNMP NetFlow v9 SNMP pcap pcap Sub-network INVEA-TECH FlowMon Probe Legend: Production traffic Mirrored traffic Measurements Test computer All devices deployed in production networks

22 Artifact 1: Overloaded device 19 Records / 100 ms (k) Flow records Flow learn failures Packets / s (k) 0 07:20 07:30 07:40 07:50 0 Create a new flow cache entry What if there is hash collisions? Cisco Catalyst 6500: Flow learn failures Informed via SNMP only

23 Artifact 1: Overloaded device (p) Day Night Amplitude Frequency (Hz) Periods of times when no new flow is measured Flows are expired every 4 s, cleaning some table cells It artificially creates periodicity on the measurements!

24 Artifact 2: Flow record loss 21 Packets Flow export protocol File, DBMS, etc. Flow probe 1 Flow collector 1 Automated analysis (Appliance) Forwarding device Internet Flow collector 2 Manual analysis Production traffic Packet loss if monitored traffic is: > monitoring link bandwidth > storage/processing rate at the measurement host Critical when monitoring lots of small flows (when? why?)

25 Artifact 2: Flow record loss 22 Records / 100 ms (k) Flow records 0 09:00 09:10 09:20 09:30 Some protocols (e.g., NetFlow) and collectors trace loss of flow records Collector is overloaded (compressing and saving to disk) Non-periodic drops in flow record time series

26 Artifact 2: Flow record loss 22 Records / 100 ms (k) 1.2 Flow records Artifacts might render flow data unusable! 0.3 e.g., security, traffic profiling etc 0 09:00 09:10 09:20 09:30 Some protocols (e.g., NetFlow) and collectors trace loss of flow records Collector is overloaded (compressing and saving to disk) Non-periodic drops in flow record time series

27 Artifact 3: Can we rely on flow timestamps? 23 Goal: estimate delay using flow data Inject traffic using the test computer, compare the recorded times Possible sources of errors: Clocks not synchronized, link delay etc. Expectation: Those errors should be small and more of less constant

28 Artifact 3: Can we rely on flow timestamps? 24 1 Cisco Catalyst Dedicated probe CDF Difference (s) Difference (s) Reality: Difference of up to 1 s when exporting using NetFlow v9 Independent of the used flow exporter Can we use such data to measure delay? Why?

29 NetFlow v9 design flaw 25 NetFlow packet: Version Number Count sysuptime UNIX Secs sysuptime Time in milliseconds since this device was first booted. UNIX Secs Time in seconds since 0000 UTC 1970, at which the Export Packet leaves the Exporter.... sysuptime in msec at which FIRST_SWITCHED 22 4 the first packet of this Flow was switched f start =UNIX Secs+FIRST SWITCHED sysuptime What is wrong?

30 NetFlow v9 design flaw 26 Example: 2 Router boot time 01/01/ :00: UNIX Secs: Flow 1 starts at 01/01/ :00: Flow 2 starts at 01/01/ :00: ms later Flow 1 is exported at 01/01/ :00: Flow 2 is exported at 01/01/ :00: ms later f start =UNIX Secs+FIRST SWITCHED sysuptime f1 start = = OK! f2 start = = Oops! Collectors could fix it (see how in 1 ), but usually they don t! 2 B. Trammell et. al. Peeling away Timing Error in NetFlow Data. In: PAM 11.

31 Artifact 3: Dedicated probe 27 Difference (ms) Reference start time (s since experiment start) NetFlow v5, IPFIX etc. don t suffer from this problem The dedicated probe exports stable time, which slowly drifts This might be a problem if data from different exporters are jointly analyzed

32 Artifact 3: Catalyst NetFlow v5 28 Difference (ms) Reference start time (s since experiment start) All tested Catalyst 6500 present other time-related artifacts There is always a residual error of up to 60 ms!

33 Artifact 3: Catalyst NetFlow v5 28 Difference (ms) Precision is not 0 the number of digits your tools report! Reference start time (s since experiment start) All tested Catalyst 6500 present other time-related artifacts There is always a residual error of up to 60 ms!

34 Artifact 3: Catalyst NetFlow v5 28 Difference (ms) Measurement errors are common 0 and might lead to artifacts Reference start time (s since experiment start) All tested Catalyst 6500 present other time-related artifacts There is always a residual error of up to 60 ms!

35 Artifact 3: Catalyst NetFlow v5 28 Difference (ms) Calibration is essential 0 in any sound measurement study! Reference start time (s since experiment start) All tested Catalyst 6500 present other time-related artifacts There is always a residual error of up to 60 ms!

36 Analysis applications 29 Network security Performance monitoring Analyzes follow a similar pattern: Understand the target application or security threat Define flow information to be exported (i.e., keys and IE) Calibrate the environment Map flow data to meaningful performance metrics Validate the measurements!

37 Example 1: SSH brute-force attacks 30 Lots of machines have an SSH server Users are not careful with passwords Daily logs of any SSH server exposed to the Internet: sshd: pam_unix(sshd:auth): authentication failure; [...] rhost= user=root sshd: Failed password for root from port ssh2 sshd: Failed password for root from port ssh2 sshd: Failed password for root from port ssh2 sshd: Disconnecting: Too many authentication failures for root [preauth] sshd: PAM 2 more authentication failures; [...] rhost= user=root sshd: pam_unix(sshd:auth): authentication failure; [...] rhost= user=root... Can successful attempts be detected from flow data?

38 Example 1: SSH brute-force attacks 31 Attack phases: 3 Start End Scan Brute-force Compromise NetFlow v5 data during brute-force phase: Start Source Destination Flags Pkts :07: : :22...S. 1 03:09: : :22.AP.SF 12 03:09: : :22.AP.SF 12 03:09: : :22.AP.SF How to learn the signature of compromised machines? 3 R. Hofstede et. al. SSH Compromise Detection using NetFlow/IPFIX. In: ACM CCR.

39 Example 1: SSH brute-force attacks Scan Brute-force Compromise Packets per flow Time (s) Once you have signatures, find potential compromises More complex than the figure suggests e.g., active mitigation such asdenyhosts andfail2ban Still, this narrows down the search for compromised machines

40 Example 2: Is Dropbox off-line? 33 Scenario: Lots of users rely on Dropbox Some companies and universities outsource file storage to Dropbox The TI department wants to be informed about outages: To be proactive when there are problems To validate the QoS offered by Dropbox How to monitor Dropbox status from passive measurements?

41 How does Dropbox work? 34 Clear separation between storage and meta-data/client control Sub-domains identifying parts of the service sub-domain Data-center Description client-lb/clientx Dropbox Meta-data notifyx Dropbox Notifications api Dropbox API control www Dropbox Web servers d Dropbox Event logs dl Amazon Direct links dl-clientx Amazon Client storage dl-debugx Amazon Back-traces dl-web Amazon Web storage api-content Amazon API Storage HTTP/HTTPs in all functionalities

42 How does Dropbox work? 35 Notification Kept open Not encrypted Device ID Folder IDs Client control Login File hash Meta-data

43 Example 2: Is Dropbox off-line? 36 Connections (k) / min Abnormal Total Unhealthy 0 00:00 12:00 00:00 12:00 00:00 Time Surges on unidirectional flows (client to server) unhealthy traffic Attempts to reconnect to notification servers Likely only some clients have been affected

44 Example 3: Response times 37 New IE to report responsiveness of HTTP servers Web server latency =...? Flow exporter Client Server SYN Exporter to server latency SYN+ACK Client to exporter latency ACK GET /index.html Application latency Web server latency HTTP/ OK

45 Example 3: Response times Median latency (s) :00 13:00 14:00 15:00 16:00 17:00 18:00 Time (h) Debian mirrors at a campus ground truth for validation Peaks in latency mirrors synchronizing with upstream repositories Can we do similar analysis for external services?

46 Example 4: Unveiling bottlenecks on Dropbox 39 Application layer sequential ACKs Storage Amazon EC2 Retrieve vs. Store

47 Example 4: Unveiling bottlenecks on Dropbox 40 10M θ 1M Throughput (bits/s) 100k 10k Chunks 1k k 4k 16k 64k 256k 1M 4M 16M 64M 400M Upload (bytes) Storage throughput in campuses (42 days of monitoring) Time between first and last packet with payload in storage flows Most flows experience a low throughput

48 Example 4: Unveiling bottlenecks on Dropbox 40 10M θ 1M Throughput (bits/s) 100k 10k 1k Chunks k 4k 16k 64k 256k 1M 4M 16M 64M 400M Upload (bytes) Flows carrying 1 chunk Size 4 MB (Dropbox max chunks), RTT 100 ms Most of them finish in TCP slow-start

49 Example 4: Unveiling bottlenecks on Dropbox 40 10M θ 1M Throughput (bits/s) 100k 10k 1k Chunks k 4k 16k 64k 256k 1M 4M 16M 64M 400M Upload (bytes) Flows carrying several chunks Pause between chunks RTT and client/server reaction

50 Example 4: Unveiling bottlenecks on Dropbox 40 10M θ 1M Throughput (bits/s) 100k 10k Chunks 1k k 4k 16k 64k 256k 1M 4M 16M 64M 400M Upload (bytes) Flows carrying several chunks Transferring 100 chunks takes more than 30 s RTTs 10 s of inactivity

51 Example 4: Unveiling bottlenecks on Dropbox 40 10M θ 1M Throughput (bits/s) 100k 10k Chunks 1k k 4k 16k 64k 256k 1M 4M 16M 64M 400M Upload (bytes) Possible solutions Distributing servers Delaying acknowledgments Bundling chunk deployed sometime after our capture

52 Take away 41 Network-based monitoring of applications/threats has pros and cons: Scalable when compared to packet recording Vantage point view No client/server instrumentation Often requires inference and approximations Hard to keep the peace with protocols... Never trust in measurements blindly! Check for outliers, validate using different methods etc.

53 References 42 Some of these slides have been taken from Ramin Sadre and Rick Hofstede. Reading material: R. Hofstede, P. Celeda, B. Trammell, I. Drago, R. Sadre, A. Sperotto, A. Pras. Flow Monitoring Explained: From Packet Capture to Data Analysis with NetFlow and IPFIX. In: IEEE Communications Surveys & Tutorials. Vol. 16, No. 4, I. Drago, M. Mellia, M.M. Munafo, A. Sperotto, R. Sadre, A. Pras. Inside Dropbox: Understanding Personal Cloud Storage Services. In: IMC 12. R. Hofstede, I. Drago, A. Sperotto, R. Sadre, A. Pras. Measurement Artifacts in NetFlow Data. In: PAM 13. R. Hofstede, L. Hendriks, A. Sperotto, A. Pras. SSH Compromise Detection using NetFlow/IPFIX. In: ACM CCR, Vol. 44, No. 5, G. Maier, R. Sommer, H. Dreger, A. Feldmann, V. Paxson, F. Schneider. Enriching Network Security Analysis with Time Travel. In: SIGCOMM 08. B. Trammell, B. Tellenbach, D. Schatzmann, M. Burkhart. Peeling away Timing Error in NetFlow Data. In: PAM 11. J. L. Garcia-Dorado, F. Mata, J. Ramos, P. M. Santiago del Rio, V. Moreno, J. Aracil. High-Performance Network Traffic Processing Systems Using Commodity Hardware. In: Data Traffic Monitoring and Analysis V. Paxson. Strategies for Sound Internet Measurement. In: IMC 04.

54 Thanks! 43