March 10 th 2011, OSG All Hands Mee6ng, Network Performance Jason Zurawski Internet2 NDT

Transcription

1 March 10 th 2011, OSG All Hands Mee6ng, Network Performance Jason Zurawski Internet2 NDT

2 Agenda Tutorial Agenda: Network Performance Primer Why Should We Care? (15 Mins) GeNng the Tools (10 Mins) Use of the BWCTL Server and Client (30 Mins) Use of the OWAMP Server and Client (30 Mins) Use of the NDT Server and Client (30 Mins) Diagnos6cs vs Regular Monitoring (30 Mins) Network Performance Exercises (1 hr 30 Mins) 2 2/21/11, 2010 Internet2

3 Hands on Tes6ng of NDT MLab (Commodity Networking) hzp://ndt.iupui.donar.measurement lab.org:7123/ Internet2 (R&E Networking) hzp://ndt.atla.net.internet2.edu:7123/ To not overwhelm the server, also try replacing atla with: chic hous kans losa newy salt seat wash 3 2/21/11, 2011 Internet2

4 NDT User Interface Web based JAVA applet allows tes6ng from any browser One Click tes6ng Op6on to dig deep into available results Send report of results to network administrators Command line client allows tes6ng from remote login shell Same op6ons available Client soaware can be build independent of server soaware 4 2/21/11, 2011 Internet2

5 NDT Results 5 2/21/11, 2011 Internet2

6 Mo6va6on for Work Measure performance to users desktop Lots of tools to measure performance to a nearby server Also plugable hardware to measure everything up to the network cable Want something to accurately show what the user is seeing Develop single shot diagnos6c tool that doesn t use historical data Combine numerous Web100 variables to analyze connec6on Develop network signatures for typical network problems Based on heuris6cs and experience Lots of problems have a smoking gun pazern, e.g. duplex mismatch, bad cable, etc. 6 2/21/11, 2011 Internet2

7 How It works Simple bi direc6onal test to gather end to end data Test from client to server, and the reverse Gets the upload and download direc6ons Gather mul6ple data variables from server Via Web100, also some derived metrics (packet inter arrival 6mes) Compare measured performance to analy6cal values How fast should a connec6on be given the observa6ons of the host and network Translate network values into plain text messages Geared toward campus area network 7 2/21/11, 2011 Internet2

8 Web100 Project Joint PSC/NCAR project funded by NSF Develop a system mib, similar to data that is exposed via SNMP First step to gather TCP data Kernel Instrument Set (KIS) Requires patched Linux kernel Geared toward wide area network performance Goal is to automate tuning to improve applica6on performance Patches available for vanilla kernels (e.g. non vendor modified) 8 2/21/11, 2011 Internet2

9 Web Based Performance Tool Operates on Any client with a Java enabled Web browser No addi6onal client soaware needs to be installed No addi6onal configura6on required What it can do: State if Sender, Receiver, or Network is opera6ng properly Provide accurate applica6on tuning info Suggest changes to improve performance What it can t do Tell you where in the network the problem is Tell you how other servers perform Tell you how other clients will perform 9 2/21/11, 2011 Internet2

10 Finding Results of Interest Duplex Mismatch This is a serious error and nothing will work right. Reported on main page, on Sta's'cs page, and mismatch: on More Details page Packet Arrival Order Inferred value based on TCP opera6on. Reported on Sta's'cs page, (with loss sta6s6cs) and order: value on More Details page Packet Loss Rates Calculated value based on TCP opera6on. Reported on Sta's'cs page, (with out of order sta6s6cs) and loss: value on More Details page Path BoZleneck Capacity Measured value based on TCP opera6on. Reported on main page 10 2/21/11, 2011 Internet2

11 NDT Tes6ng Normal Opera6on 11 2/21/11, 2011 Internet2

12 NDT Tes6ng Duplex Mismatch 12 2/21/11, 2011 Internet2

13 NDT Tes6ng Low Throughput 13 2/21/11, 2011 Internet2

14 NDT Tes6ng Increase TCP Buffer Size 14 2/21/11, 2011 Internet2

15 BoZleneck Link Detec6on What is the slowest link in the end to end path? Monitors packet arrival 6mes using libpcap rou6ne Data and ACK packets Is aware of packet sizes used to calculate speed Use TCP dynamics to create packet pairs Quan6ze results into link type bins Broad classifica6on, e.g. FastE No frac6onal or bonded links currently Example: Consider the following setup 1G network card on Host 1G LAN 100M (FastE) Wall Jack NDT will report there is a slow link somewhere in the path. It can t tell you where, but something is limi6ng the test speed 15 2/21/11, 2011 Internet2

16 Duplex Mismatch Detec6on Duplex Mismatch: Opera6on between a host and an interface are at different duplex modes (e.g. one half, one full) Common in networks where auto nego6a6on is disabled, or faulty Classic example of a soa failure, connec6vity is present and speeds are poor Developed analy6cal model to describe how Ethernet responds Expanding model to describe UDP and TCP flows Develop prac6cal detec6on algorithm Test models in LAN, MAN, and WAN environments 16 2/21/11, 2011 Internet2

17 Faulty Hardware or Link Detect non conges6ve loss due to Faulty NIC/switch interface Bad Cat 5 cable Dirty op6cal connector 17 2/21/11, 2011 Internet2

18 Conges6on Detec6on Shared network infrastructures will cause periodic conges6on episodes Detect/report when TCP throughput is limited by cross traffic Detect/report when TCP throughput is limited by own traffic 18 2/21/11, 2011 Internet2

19 Addi6onal Func6ons and Features Provide basic tuning informa6on Features: Basic configura6on file FIFO scheduling of tests, support for tes6ng with simultaneous clients Simple server discovery protocol and ability to federate (e.g. load balance) servers Logging of all test results on the server side Command line client support Other Clients can be developed against open Javascript API: hzp:// Posted on Google Code: hzp://code.google.com/p/ndt/ 19 2/21/11, 2011 Internet2

20 Architecture NDT - Server Web Server Well Known NDT Server Web Page Request Web page response Client Web Browser Testing Engine Spawn child Child Test Engine Test Request Control Channel Specific test channels Java Applet 20 2/21/11, 2011 Internet2

21 Finding a Server The Old Way Sta6c List of servers doesn t scale 21 2/21/11, 2011 Internet2

22 Finding a Server The New Way perfsonar Infrastructure automa6cally search for instances 22 2/21/11, 2011 Internet2

23 Finding a Server MLab Measurement Lab Joint Project between several partners More Info Here: hzp:// Locate a close NDT server using DONAR ( hzp://donardns.org/) 23 2/21/11, 2011 Internet2

24 General Requirements Support Source should compile for all modern *NIX *BSD, Linux, OS X configure/make/make install Web100 Patched Kernel perfsonar PS Project also offers two alterna6ves: ps Performance Toolkit (bootable ISO) Pre packaged kernel with Web100 for CentOS ( hzp://soaware.internet2.edu) Other Soaware Java SDK Libpcap RPMs compiled specifically for CentOS 5.5 May work with other RPM based systems (Fedora, RHEL) 24 2/21/11, 2011 Internet2

25 Recommended SeNngs There are no senngs or op6ons for the Web based java applet. It allows the user to run a fixed set of tests for a limited 6me period Test engine senngs Turn on admin view ( a op6on) If mul6ple network interfaces exist use i op6on to specify correct interface to monitor (ethx) Simple Web server (fakewww) Use l fn op6on to create log file Could also use a real web server like Apache 25 2/21/11, 2011 Internet2

26 Poten6al Risks Non standard kernel required Web100 patching may be difficult to apply to new kernels Hard to keep up with vendor patching GUI tools can be used to monitor other ports Consider using ps Performance Toolkit enhancements if this scares you Public servers generate trouble reports from remote users Respond or ignore s Test streams can trigger IDS alarms Configure IDS to ignore NDT server 26 2/21/11, 2011 Internet2

27 Availability Main Page: hzp:// Mailing lists: ndt ndt 27 2/21/11, 2011 Internet2

28 NDT March 10 th 2011, OSG All Hands Mee6ng Jason Zurawski Internet2 For more informa6on, visit hzp:// 28 2/21/11, 2011 Internet2