Hitachi Protection Platform S-Series

Transcription

1 Hitachi Protection Platform S-Series Network Performance Data Collection Rev 1.1 ZZ-99HPP999-99

2 2015 Hitachi, Ltd. All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopying and recording, or stored in a database or retrieval system for any purpose without the express written permission of Hitachi, Ltd. and Hitachi Data Systems Corporation (hereinafter referred to as Hitachi ). Hitachi, Ltd. and Hitachi Data Systems reserve the right to make changes to this document at any time without notice and assume no responsibility for its use. Hitachi, Ltd. and Hitachi Data Systems products and services can only be ordered under the terms and conditions of Hitachi Data Systems' applicable agreements. All of the features described in this document may not be currently available. Refer to the most recent product announcement or contact your local Hitachi Data Systems sales office for information on feature and product availability. Notice: Hitachi Data Systems products and services can be ordered only under the terms and conditions of Hitachi Data Systems applicable agreements. The use of Hitachi Data Systems products is governed by the terms of your agreements with Hitachi Data Systems. By using this software, you agree that you are responsible for: 1. Acquiring the relevant consents as may be required under local privacy laws or otherwise from employees and other individuals to access relevant data; and 2. Verifying that data continues to be held, retrieved, deleted, or otherwise processed in accordance with relevant laws. Hitachi is a registered trademark of Hitachi, Ltd. in the United States and other countries. Hitachi Data Systems is a registered trademark and service mark of Hitachi in the United States and other countries. All other trademarks, service marks, and company names are properties of their respective owners. Export authorization is required for Data At Rest Encryption. Import/Use regulations may restrict export of the S-Series platform to certain countries: China S-Series is eligible for import but the License Key and encryption may not be sent to China France Import pending completion of registration formalities Hong Kong Import pending completion of registration formalities Israel Import pending completion of registration formalities ii

3 Table of Contents Introduction... 1 Testing/Evaluation prerequisite... 1 Capturing session output for later review... 1 Grading network performance... 2 Data Gathering... 3 Basic data gathering... 3 Using sar and scp... 4 Using iperf... 6 Using Graphical-based tools... 9 Network triage Bonding issues Firewalls Potential duplicate IP address End-to-end connectivity Appendix A: CLI command usage statements ping tracepath iftop iperf ethtool nethogs net_perf_mon Appendix B: Installing iperf Appendix C: Bonding modes Appendix D: Tools iii

4 iv

5 1 1 Introduction This documents provides a list of tests and procedures which should be run on each HPP system at the site to gather networking-related performance statistics. This document is intended for personnel who assist in installing, maintaining, and managing the HPP platform in the backup environment. Testing/Evaluation prerequisite As a pre-cursor to performing any of this testing, a network diagram at both the LAN and WAN level should be provided by the customer. The diagram should list all IP addresses of the HPP s and media servers as well as their DNS names. Capturing session output for later review Most (if not all) of the tests outlined in this document should be run from a Secure Shell (SSH) session, usually via the putty application. To allow a Global Support engineer to review the data for analysis at a later time, it is required that all session output be captured to a file on a local desktop. To effect the capture with the putty SSH application, use the following steps: 1) Bring up the main putty window (not a session window) 2) On the Category: tree, click on Session Logging. 3) Under Session logging:, click on the radio button All session output. 4) Select a log file name. The default is putty.log but it is advisable to use a unique name. Be aware, if an existing putty.log file exists on your desktop it will be overwritten, not appended. 1

6 5) On the Category: tree, click on Session, connect to your target system, and run the procedure. 6) When the session is complete get the putty.log file from your desktop and archive it for analysis. Grading network performance In general terms, network performance can be gauged based on the average percentage of line rate achieved in the tests: Ratings apply to one LAN and no more than two interconnecting switches. % Line Rate MB/sec Grade Min Max 1Gb network 10 Gb network A B C D F <75 <750 Any connections that would receive an F grade should be investigated by the customer s networking team and improved or at least satisfactorily explained. Any connections that would receive a D grade should be reviewed by the customer s networking team for any possible improvements. 2 2

7 2 3 Data Gathering This section specifies a number of methods that can be used to gather network performance data on HPP systems: Via a combination of the Linux sar and scp commands Via the public domain tool iperf Via a graphical-based tool In addition, a set of basic commands should be run to document the current configuration of the networking subsystem on the HPP system. Basic data gathering The following are a set of basic commands to document the current configuration of the networking subsystem on the HPP system. Steps: 1. Retrieve a full listing of the Ethernet devices: ifconfig a 2. Retrieve the kernel interface table for all bonds and interfaces: netstat i 3. Retrieve the state of all IP sockets netstat a more 4. Retrieve per-networking-protocol statistics netstat s more 5. Retrieve all active networking connections netstat o more 3

8 6. Retrieve the address resolution protocol (ARP) table arp a more 7. Retrieve the routing tables route 8. Check for TCP and UDP connection issues netstat tnp 9. Retrieve network interface statics ethtool s eth# Using sar and scp The system activity recorder ( sar ) and secure copy ( scp ) can be used together for simple tests over LAN (and WAN, where applicable) links to media servers. On the source HPP system, use SCP to transfer a file over the desired ethx port and measure performance from the app. For example, run the SCP test from node0 s replication IP on HPP system A to node0 s replication IP on HPP system B. Continue with all slave node s that are configured for I/O. Steps: NOTE: If running SCP to Windows Servers you must install an SCP compliant tool such as WindowSCP from a freeware site Generate a 500MB data file: dd if=/dev/zero of=/shm/dev/scptemp bs=1024 count=500k This will create the 500MB data file in a RAM disk. This will allow us to avoid the read latency of an on-disk data file during the transfer part of the test. 2. On the target, run the following to measure performance throughput on all eth# replication-duplication ports for each node. sar n DEV 1 This will display networking statistics ( -n ) once every second ( 1 ) for each networking DEV ice (i.e., each NIC). 3. Use SCP to copy and measure throughput: scp /dev/shm/scptemp root@remoteipaddress:/dev/null 4

9 Run this test from node0 and all processing nodes configured for I/O. If you notice a stall status from SCP then that may indicate a problem with the customer s network. 4. Watch the sar output on the target system s processing nodes. Pay attention to the rxkb/s (Received KBs) field. For example, if using mahpp01 as the source HPP system and mahpp02 as the remote HPP system, communicating over a 10Gb NIC (eth4 on both systems): MAHPP01 [root@mahpp01 ~]# dd if=/dev/zero of=/dev/shm/scptemp bs=1024 count=500k records in records out bytes (524 MB) copied, seconds, 1.3 GB/s [root@mahpp01 ~]# df k [root@mahpp01 ~]# scp /dev/shm/scptemp root@ :/dev/null scptemp 100% 500MB 62.5MB/s 00:08 [root@mahpp01 ~]# MAHPP02 [root@mahpp02 ~]# sar -n DEV 1 grep eth4 IFACE rxpck/s txpck/s rxkb/s txkb/s rxcmp/s txcmp/s rxmcst/s 15:50:03 eth :50:04 eth :50:05 eth : 15:51:07 eth :51:08 eth :51:09 eth :51:10 eth :51:11 eth :51:12 eth :51:13 eth :51:14 eth :51:15 eth :51:16 eth :51:17 eth :51:18 eth This shows a network speed of 62.5MB/sec (or 500Mb/sec) [ignoring the first and last statistics since they are most likely partial] on a single connection. Multiple, simultaneous runs of scp can be used to achieve higher bandwidth. For example, twenty (20) simultaneous scp s were able to transfer data at an approximate combined transfer rate of 770MB/sec (6160Mb/sec). 5

10 Using iperf iperf always uses port You can use the IP address of the remote HPP system or the media servers you want to test to as a target. The minimum test should be from each HPP node to the corresponding HPP node across the WAN. So from site A node0 run iperf to site B node0 and again in the reverse order so performance can be compared bi-directionally. Continue the same methodology for site A node1 run iperf to site B node1 and so forth. iperf must be installed on each processing node of an HPP system. iperf requires that it be run with the -s option for the server side use and with the -c for the client side use. NOTES: 1) The iperf64 obtained from the FTP site is compiled for to be run on the HPP appliance (ie. a 64-bit RedHat-based Linux OS). Other OS s will have to be compiled separately. (Windows, SUSE, AIX, etc.) 2) An iperf server process can handle only one connection at a time. You can not run parallel iperf tests to the same server process. For reference on iperf see the following: Steps: 1. Install iperf on each node the HPP system. See Appendix B for instructions. 2. On all processing nodes of the HPP system, you must temporarily shut down the firewall: service iptables stop 3. Start the iperf server process on all nodes. First run the command on node0 (the master node). Then ssh into each of the other processing nodes from node0 and run the same command: nohup iperf64 s & 4. On all processing nodes, run iperf in client mode, specifying the IP address of each processing node of the remote HPP system: /root/iperf64 c destinationipaddress i 1 For example: /root/iperf64 c i 1 If you want to monitor for network errors, use: sar -n EDEV 1 6

11 5. Once testing is complete, on each node of both HPP systems: a. Find the iperf server process with the following: ps ef grep iperf b. Kill the process: kill -9 pid 6. On all processing nodes of the HPP system, restart the firewall: service iptables start Examples: -c~]#./iperf64 c Client connecting to , TCP port 5001 TCP window size: 8.34 MByte (default) [ 3] local port connected with port 5001 [ 3] sec 9.90 GBytes 8.50 Gbits/sec [root@dcd01-sept01 ~]# Print stats every seconds continuosly: [root@dcd01-sept02 ~]#./iperf64 -c i Client connecting to , TCP port 5001 TCP window size: 8.34 MByte (default) [ 3] local port connected with port 5001 [ 3] sec 635 MBytes 5.32 Gbits/sec [ 3] sec 663 MBytes 5.56 Gbits/sec [ 3] sec 767 MBytes 6.44 Gbits/sec [ 3] sec 841 MBytes 7.06 Gbits/sec [ 3] sec 867 MBytes 7.27 Gbits/sec [ 3] sec 895 MBytes 7.51 Gbits/sec [ 3] sec 868 MBytes 7.28 Gbits/sec [root@dcd01-sept02 ~]# 7

12 Change TCP window size from OS default to 512KB, print stats every 5 seconds for 60 seconds (1 minute). [root@node1 systest]#./iperf64 -c w i 5 -t Client connecting to , TCP port 5001 TCP window size: 1.00 MByte (WARNING: requested 512 KByte) [ 3] local port connected with port 5001 [ 3] sec 4.31 GBytes 7.40 Gbits/sec [ 3] sec 4.71 GBytes 8.08 Gbits/sec [ 3] sec 5.09 GBytes 8.74 Gbits/sec [ 3] sec 4.46 GBytes 7.66 Gbits/sec [ 3] sec 4.65 GBytes 7.99 Gbits/sec [ 3] sec 4.91 GBytes 8.44 Gbits/sec [ 3] sec 4.83 GBytes 8.29 Gbits/sec [ 3] sec 5.06 GBytes 8.69 Gbits/sec [ 3] sec 4.94 GBytes 8.49 Gbits/sec [ 3] sec 5.16 GBytes 8.87 Gbits/sec [ 3] sec 4.77 GBytes 8.20 Gbits/sec [ 3] sec 4.90 GBytes 8.42 Gbits/sec [ 3] sec 57.8 GBytes 8.27 Gbits/sec [root@node1 systest]# 8

13 Using Graphical-based tools The following lists a number of more advanced routines, some productspecific which can be run to gather additional information to be used as part of the triage process. It should be noted, though, that these tools are realtime tools that will require a screen shot to record properly. Steps: 1. Monitor interface statistics in real time: nethogs m This tool is very useful with the m flag switch that will show you totals received and rate in KB/s. 2. Displays a real time graph showing network performance net_perf_mon 3. Displays display bandwidth usage on an interface by host iftop n N p -P i interface 9

14 10

15 3 Network triage This section details some data-gathering steps for issues involving: Network interface bonding Communications through firewalls Potential duplicate IP addresses Basis end-to-end connectivity Bonding issues The HPP system has 5 predefined potential NIC bonding scenarios: Name Devices Description Mode bond0 eth2/eth3 HPP s private backend network (not used on S1500 systems) 1/ active-backup bond1 eth4/eth5 (legacy) eth8/eth9 (HPP) 10Gb NIC ports 4/LACP bond2 eth4/eth5 ports 1 and 2 of quad-1gb NIC 4/LACP bond3 eth6/eth7 ports 3 and 4 of quad-1gb NIC 4/LACP bond4 eth4/eth5/eth6/eth7 all four ports of quad-1gb NIC 4/LACP If a bonding issue is suspected the following commands should help to determine the configuration. 11

16 Steps: 1. Check the current OS settings for the bond: cat /proc/net/bonding/bond# i. Verify the bonding mode matches what is in the table above. ii. iii. Verify the speed and duplex of the slave interfaces match Verify that the MII status is up. 2. Check the configured settings for the bond: cat /etc/sysconfig/network-scripts/ifcfg-bond# Verify the bonding mode matches what is in the table above. 3. Check the ARP table: arp e v 4. Check the individual link statuses of the bond and its slave NICs: ip link show bond# ; ip link show nic1 ; ip link show nic2 For example, assume we have an issue around the bonded 10Gb NIC ports of a G8 processing node (where the 10Gb ports are eth8 and eth9 ): [root@mahpp01 bonding]# cat /proc/net/bonding/bond1 Ethernet Channel Bonding Driver: v3.7.0 (June 2, 2010) Bonding Mode: fault-tolerance (active-backup) Primary Slave: eth2 (primary reselect always) Currently Active Slave: eth2 MII Status: up MII Polling Interval (ms): 100 Up Delay (ms): Down Delay (ms): 0 Slave Interface: eth8 MII Status: up Speed: 1000 Mbps Duplex: full Link Failure Count: 0 Permanent HW addr: ac:16:2d:78:af:da Slave queue ID: 0 Slave Interface: eth9 MII Status: up Speed: 1000 Mbps Duplex: full Link Failure Count: 0 Permanent HW addr: ac:16:2d:78:af:db Slave queue ID: 0 [root@mahpp01 bonding]# [root@mahpp01 ~]# cat /etc/sysconfig/network-scripts/ifcfg-bond1 DEVICE=bond1 12

17 BOOTPROTO=static IPADDR= NETMASK= ONBOOT=YES BONDING_OPTS="mode=4 ad_select=bandwidth lacp_rate=fast xmit_hash_policy=layer3+4 miimon=100 updelay=45000" ~]# ~]# arp -e -v Address HWtype HWaddress Flags Mask Iface dvmnode ether 00:23:7d:34:87:08 C bond ether 50:57:a8:5a:11:c5 C eth0 Entries: 2 Skipped: 0 Found: 2 [root@mahpp01 ~]# [root@mahpp01 ~]# ip link show bond1 ; ip link show eth8 ; ip link show eth9 15: bond1: <BROADCAST,MULTICAST,MASTER> mtu 1500 qdisc noop link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff 13: eth8: <BROADCAST,MULTICAST> mtu 1500 qdisc noop qlen 1000 link/ether 10:60:4b:94:c3:38 brd ff:ff:ff:ff:ff:ff 14: eth9: <BROADCAST,MULTICAST> mtu 1500 qdisc noop qlen 1000 link/ether 10:60:4b:94:c3:3c brd ff:ff:ff:ff:ff:ff [root@mahpp01 ~]# Firewalls The ability for communications between the HPP system and other HPP systems at the site as well as between the HPP and other customer systems (e.g., Netbackup media servers), must be tested for. The typical ports to test for are: 22 Secure-Shell (ssh) 80 HPP GUI (http) 443 HPP GUI (secure http) 5562 OST master service 5564 OST I/O service Version 8.1 and later Use the nc to test that all required TCP ports are open through any firewalls that may exist between the HPP system(s) and other network hosts: nc v z w 5 destinationipaddress port# For example, to test that the ssh port is open: [root@mahpp01 ~]# nc v z w Connection to port [tcp/ssh] succeeded! [root@mahpp01 ~]# 13

18 Version 8.0 and earlier Use telnet to test that all required TCP ports are open through any firewalls that may exist between the HPP system(s) and other network hosts: telnet destinationipaddress port# When the command is successful, it will appear to hang; it s actually waiting for input. In some cases (e.g., port 80 [ssh]), simply hitting the Enter key should be enough to cause the command to exit (in ssh s case, with an expected protocol mismatch error message). For others, it may be necessary to type Ctrl-C or Ctrl-D to get telnet to terminate. When the command is NOT successful, it should return almost immediately with an error. For example, this shows a successful test that the ssh port is open: [root@mahpp01 ~]# telnet Trying Connected to Escape character is '^]'. SSH-2.0-OpenSSH_5.4 Protocol mismatch. Connection closed by foreign host. [root@mahpp01 ~]# The following shows two failed tests: [root@mahpp02 ~]# telnet Trying telnet: connect to address : No route to host [root@mahpp02 ~]# ping -c PING ( ) 56(84) bytes of data. From icmp_seq=1 Destination Host Unreachable From icmp_seq=2 Destination Host Unreachable From icmp_seq=3 Destination Host Unreachable ping statistics packets transmitted, 0 received, +3 errors, 100% packet loss, time 1999ms, pipe 3 [root@ mahpp02 ~]# telnet Trying telnet: connect to address : Connection refused [root@mahpp02 ~]# ping -c PING ( ) 56(84) bytes of data. 64 bytes from : icmp_seq=1 ttl=64 time=0.134 ms 64 bytes from : icmp_seq=2 ttl=64 time=0.153 ms 64 bytes from : icmp_seq=3 ttl=64 time=0.154 ms ping statistics packets transmitted, 3 received, 0% packet loss, time 1999ms rtt min/avg/max/mdev = 0.134/0.147/0.154/0.009 ms [root@mahpp02 ~]# 14

19 Potential duplicate IP address If there are unexplained network issues using the regular methods listed above there may be a duplicate IP address in the network. To check for that condition do the following: Steps: 1. Run the arping command: arping -D -I eth# -c 2 ipaddress ; echo $? where ipaddress is the IP address you want to check for duplicate IP s on, -c # is the number of probes to be sent (in this case 2 ), and -Iand should be the one assigned to the port 2. Examine the return/exit value of arping (i.e., the shell variable '$?'): 0 = no duplication 1 = duplicated: For example, assume we suspect that the IP address assigned to eth0 of processing node #1 is duplicated someplace else on the customer s network. The following would be run on NODE1: [root@node1 ~]# ifconfig eth0 eth0 Link encap:ethernet HWaddr D4:85:64:78:2B:58 inet addr: Bcast: Mask: inet6 addr: fe80::d685:64ff:fe78:2b58/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets: errors:0 dropped:0 overruns:0 frame:0 TX packets:9076 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes: (30.3 MiB) TX bytes: (666.6 KiB) Interrupt:16 Memory:f f [root@node1 ~]# arping -D -I eth0 -c ; echo $? ARPING from eth0 Sent 2 probes (2 broadcast(s)) Received 0 response(s) 0 [root@node1 ~]# In this case, the address is NOT duplicated. 15

20 End-to-end connectivity Use ping to test basic end-to-end connectivity. Any excessively high values need to be investigated. But note that the definition of excessively high values will depend on the network topology and whether the test is running over LAN links and/or WAN links. If any of the pings fail: Verify with the customer the IP address and/or DNS-resolvable host name used Use the tracepath utility to see where things may be failing For example, ~]# ping -c PING ( ) 56(84) bytes of data. From icmp_seq=3 Destination Net Unreachable ping statistics packets transmitted, 0 received, +1 errors, 100% packet loss, time 2000ms [root@mahpp01 ~]# tracepath : ( ) 0.292ms pmtu : ( ) 0.715ms 2: ( ) 2.222ms 3: no reply 4: ( ) asymm ms!N Resume: pmtu 1500 [root@mahpp01 ~]# Steps: 1. Verify that a default gateway has been configured: route n 2. Ping the default gateway using the IP address. 3. Verify that DNS server(s) are defined: cat /etc/resolv.conf 4. Ping the DNS servers. 5. If the HPP system will be replicating to another HPP system, a. Obtain from the customer the IP address and host name (resolvable by DNS) of that other HPP system b. Ping the IP address of that other system c. Ping the DNS-resolvable host name of the other HPP system d. Repeat step 5 on the other HPP system using this HPP system s IP address and DNS-resolvable host name. 16

21 6. If the Symantec Netbackup (NBU) Open-Storage Technology (OST) protocol will be used to communicate with the HPP system: a. Obtain from the customer the IP address and host name (resolvable by DNS) of all NBU servers that will communicate with the HPP system via OST b. ping the IP address of each NBU server c. ping the DNS-resolvable host name of each NBU server For example: Verify gateway configured ~]# route -n Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface UG eth U bond U eth U eth U eth U eth7 [root@karloff ~]# In this case, a default gateway is configured and its IP address is: Ping gateway [root@karloff ~]# ping -c PING ( ) 56(84) bytes of data. 64 bytes from : icmp_seq=1 ttl=255 time=0.697 ms 64 bytes from : icmp_seq=2 ttl=255 time=0.687 ms 64 bytes from : icmp_seq=3 ttl=255 time=1.11 ms 64 bytes from : icmp_seq=4 ttl=255 time=0.655 ms 64 bytes from : icmp_seq=5 ttl=255 time=0.665 ms ping statistics packets transmitted, 5 received, 0% packet loss, time 4001ms rtt min/avg/max/mdev = 0.655/0.763/1.115/0.179 ms [root@karloff ~]# Verify DNS servers configured [root@karloff ~]# cat /etc/resolv.conf domain sepaton.com search sepaton.com sangate.com nameserver nameserver [root@karloff ~]# In this case, two DNS servers are configured and their IP addresses are: and Ping DNS server [root@karloff ~]# ping -c PING ( ) 56(84) bytes of data. 64 bytes from : icmp_seq=1 ttl=63 time=0.172 ms 64 bytes from : icmp_seq=2 ttl=63 time=0.179 ms 64 bytes from : icmp_seq=3 ttl=63 time=0.184 ms 64 bytes from : icmp_seq=4 ttl=63 time=0.165 ms 64 bytes from : icmp_seq=5 ttl=63 time=0.152 ms ping statistics packets transmitted, 5 received, 0% packet loss, time 4000ms rtt min/avg/max/mdev = 0.152/0.170/0.184/0.016 ms [root@karloff ~]# Ping OST storage server 17

22 ~]# ping -c 5 ssdnaught10gn0 PING ssdnaught10gn0 ( ) 56(84) bytes of data. 64 bytes from ssdnaught10gn0 ( ): icmp_seq=1 ttl=64 time=0.221 ms 64 bytes from ssdnaught10gn0 ( ): icmp_seq=2 ttl=64 time=0.153 ms 64 bytes from ssdnaught10gn0 ( ): icmp_seq=3 ttl=64 time=0.170 ms 64 bytes from ssdnaught10gn0 ( ): icmp_seq=4 ttl=64 time=0.182 ms 64 bytes from ssdnaught10gn0 ( ): icmp_seq=5 ttl=64 time=0.163 ms --- ssdnaught10gn0 ping statistics packets transmitted, 5 received, 0% packet loss, time 4000ms rtt min/avg/max/mdev = 0.153/0.177/0.221/0.028 ms [root@karloff ~]# ping -c PING ( ) 56(84) bytes of data. 64 bytes from : icmp_seq=1 ttl=64 time=0.213 ms 64 bytes from : icmp_seq=2 ttl=64 time=0.153 ms 64 bytes from : icmp_seq=3 ttl=64 time=0.149 ms 64 bytes from : icmp_seq=4 ttl=64 time=0.125 ms 64 bytes from : icmp_seq=5 ttl=64 time=0.156 ms ping statistics packets transmitted, 5 received, 0% packet loss, time 3999ms rtt min/avg/max/mdev = 0.125/0.159/0.213/0.030 ms [root@karloff ~]# Ping NBUE media server [root@karloff ~]# ping -c 5 sylvester PING sylvester ( ) 56(84) bytes of data. 64 bytes from sylvester ( ): icmp_seq=1 ttl=64 time=2.03 ms 64 bytes from sylvester ( ): icmp_seq=2 ttl=64 time=0.419 ms 64 bytes from sylvester ( ): icmp_seq=3 ttl=64 time=0.314 ms 64 bytes from sylvester ( ): icmp_seq=4 ttl=64 time=0.219 ms 64 bytes from sylvester ( ): icmp_seq=5 ttl=64 time=0.381 ms --- sylvester ping statistics packets transmitted, 5 received, 0% packet loss, time 4000ms rtt min/avg/max/mdev = 0.219/0.674/2.039/0.686 ms [root@karloff ~]# 18

23 A Appendix A: CLI command usage statements ping Usage: ping [-LRUbdfnqrvVaA] [-c count] [-i interval] [-w deadline] [-p pattern] [-s packetsize] [-t ttl] [-I interface or address] [-M mtu discovery hint] [-S sndbuf] [ -T timestamp option ] [ -Q tos ] [hop1...] destination tracepath Usage: tracepath [-nc] <destination>[/<port>] iftop iftop: display bandwidth usage on an interface by host Synopsis: iftop -h [-npbbp] [-i interface] [-f filter code] [-N net/mask] -h display this message -n don't do hostname lookups -N don't convert port numbers to services -p run in promiscuous mode (show traffic between other hosts on the same network segment) -b don't display a bar graph of traffic -B Display bandwidth in bytes -i interface listen on named interface -f filter code use filter code to select packets to count (default: none, but only IP packets are counted) -F net/mask show traffic flows in/out of network -P show ports as well as hosts -m limit sets the upper limit for the bandwidth scale -c config file specifies an alternative configuration file iftop, version 0.17 copyright (c) 2002 Paul Warren <pdw@ex-parrot.com> and contributors 19

24 iperf Usage: iperf [-s -c host] [options] iperf [-h --help] [-v --version] Client/Server: -f, --format [kmkm] format to report: Kbits, Mbits, KBytes, MBytes -i, --interval # seconds between periodic bandwidth reports -l, --len #[KM] length of buffer to read or write (default 8 KB) -m, --print_mss print TCP maximum segment size (MTU - TCP/IP header) -o, --output <filename> output the report or error message to this specified file -p, --port # server port to listen on/connect to -u, --udp use UDP rather than TCP -w, --window #[KM] TCP window size (socket buffer size) -B, --bind <host> bind to <host>, an interface or multicast address -C, --compatibility for use with older versions does not sent extra msgs -M, --mss # set TCP maximum segment size (MTU - 40 bytes) -N, --nodelay set TCP no delay, disabling Nagle's Algorithm -V, --IPv6Version Set the domain to IPv6 Server specific: -s, --server -U, --single_udp -D, --daemon run in server mode run in single threaded UDP mode run the server as a daemon Client specific: -b, --bandwidth #[KM] for UDP, bandwidth to send at in bits/sec (default 1 Mbit/sec, implies -u) -c, --client <host> run in client mode, connecting to <host> -d, --dualtest Do a bidirectional test simultaneously -n, --num #[KM] number of bytes to transmit (instead of -t) -r, --tradeoff Do a bidirectional test individually -t, --time # time in seconds to transmit for (default 10 secs) -F, --fileinput <name> input the data to be transmitted from a file -I, --stdin input the data to be transmitted from stdin -L, --listenport # port to recieve bidirectional tests back on -P, --parallel # number of parallel client threads to run -T, --ttl # time-to-live, for multicast (default 1) -Z, --linux-congestion <algo> set TCP congestion control algorithm (Linux only) Miscellaneous: -x, --reportexclude [CDMSV] exclude C(connection) D(data) M(multicast) S(settings) V(server) reports -y, --reportstyle C report as a Comma-Separated Values -h, --help print this message and quit -v, --version print version information and quit [KM] Indicates options that support a K or M suffix for kilo- or mega- The TCP window size option can be set by the environment variable TCP_WINDOW_SIZE. Most other options can be set by an environment variable IPERF_longOptionName, such as IPERF_BANDWIDTH. 20

25 ethtool Usage: ethtool DEVNAME Display standard information about device ethtool -s --change DEVNAME Change generic options [ speed ] [ duplex half full ] [ port tp aui bnc mii fibre ] [ autoneg on off ] [ advertise %%x ] [ phyad %%d ] [ xcvr internal external ] [ wol p u m b a g s d... ] [ sopass %%x:%%x:%%x:%%x:%%x:%%x ] [ msglvl %%d ] ethtool -a --show-pause DEVNAME Show pause options ethtool -A --pause DEVNAME Set pause options [ autoneg on off ] [ rx on off ] [ tx on off ] ethtool -c --show-coalesce DEVNAME Show coalesce options ethtool -C --coalesce DEVNAME Set coalesce options [adaptive-rx on off] [adaptive-tx on off] [rx-usecs N] [rx-frames N] [rx-usecs-irq N] [rx-frames-irq N] [tx-usecs N] [tx-frames N] [tx-usecs-irq N] [tx-frames-irq N] [stats-block-usecs N] [pkt-rate-low N] [rx-usecs-low N] [rx-frames-low N] [tx-usecs-low N] [tx-frames-low N] [pkt-rate-high N] [rx-usecs-high N] [rx-frames-high N] [tx-usecs-high N] [tx-frames-high N] [sample-interval N] ethtool -g --show-ring DEVNAME Query RX/TX ring parameters ethtool -G --set-ring DEVNAME Set RX/TX ring parameters [ rx N ] [ rx-mini N ] [ rx-jumbo N ] [ tx N ] ethtool -k --show-offload DEVNAME Get protocol offload information ethtool -K --offload DEVNAME Set protocol offload [ rx on off ] [ tx on off ] [ sg on off ] [ tso on off ] [ ufo on off ] [ gso on off ] [ gro on off ] ethtool -i --driver DEVNAME Show driver information ethtool -d --register-dump DEVNAME Do a register dump [ raw on off ] [ file FILENAME ] ethtool -e --eeprom-dump DEVNAME Do a EEPROM dump [ raw on off ] 21

26 [ offset N ] [ length N ] ethtool -E --change-eeprom DEVNAME Change bytes in device EEPROM [ magic N ] [ offset N ] [ value N ] ethtool -r --negotiate DEVNAME Restart N-WAY negotation ethtool -p --identify DEVNAME Show visible port identification (e.g. blinking) [ TIME-IN-SECONDS ] ethtool -t --test DEVNAME Execute adapter self test [ online offline ] ethtool -S --statistics DEVNAME Show adapter statistics ethtool -h --help DEVNAME Show this help nethogs usage: nethogs [-V] [-b] [-d seconds] [-t] [-p] [device [device [device...]]] -V : prints version. -d : delay for update refresh rate in seconds. default is 1. -t : tracemode. -b : bughunt mode - implies tracemode. -p : sniff in promiscuous mode (not recommended). device : device(s) to monitor. default is eth0 When nethogs is running, press: q: quit m: switch between total and kb/s mode net_perf_mon Usage: net_perf_mon [options] Options: -?, -h : this help message -W <num> : set history Window to <num> seconds (default 10) -R <num> : set Refresh frequency to <num> seconds (default 3) -N : No Graph. CSV output on each refresh interval. -S <num> : For use with -N. Just take <num> samples, then exit. -X : For use with -N, -S. Use XML output instead of CSV output -M <num> : Mode -o <file> : Output file 22

27 B Appendix B: Installing iperf For a more in depth network performance test use iperf. The file can be obtained from our FTP * server at the following location: ftp.sepaton.com Login: upgrade Password: Upgrade!S3paton Steps: 1. cd /users/upgrade/vtl-tools 2. get iperf.tgz The executable must be loaded on all nodes involved in the testing in /root folder (scp iperf64 root@nodex:/root to all slave nodes. Follow procedure below: 1) SCP copy the iperf.tgz file into the /root directory on each node of the HPP system to be tested. 2) Un-TAR the pst.tgz file: tar zxf pst.tgz 3) Add execute permission to the extracted file chmod a+x /root/iperf64 4) Copy the executable to each processing node in the HPP system: scp iperf64 root@node#:/root For example: scp iperf64 root@node3:/root * At least for as long as the Sepaton FTP is available. Future plans call for all FTP-server-available files to be moved to the HDS TUF server at by the beginning of September

28 24

29 C Appendix C: Bonding modes Modes for the Linux bonding driver (network interface aggregation modes) are supplied as parameters to the kernel bonding module at load time. The behavior of the single logical bonded interface depends upon its specified bonding driver mode. NOTE: HPP systems have been qualified with modes 0, 1, and 4 only. Round-robin (balance-rr) [Linux driver mode 0 ] Transmit network packets in sequential order from the first available network interface (NIC) slave through the last. This mode provides load balancing and fault tolerance. Active-backup (active-backup) [Linux driver mode 1 ] Only one NIC slave in the bond is active. A different slave becomes active if, and only if, the active slave fails. The single logical bonded interface's MAC address is externally visible on only one NIC (port) to avoid distortion in the network switch. This mode provides fault tolerance. XOR (balance-xor) [Linux driver mode 2 ] Transmit network packets based on [(source MAC address XOR'd with destination MAC address) modulo NIC slave count]. This selects the same NIC slave for each destination MAC address. This mode provides load balancing and fault tolerance. Broadcast (broadcast) [Linux driver mode 3 ] Transmit network packets on all slave network interfaces. This mode provides fault tolerance. Taken from Wikapedia page ( 25

30 IEEE 802.3ad Dynamic Link Aggregation (802.3ad) [a.k.a. LACP] (Default mode for HPP system bonds) [Linux driver mode 4 ] Creates aggregation groups that share the same speed and duplex settings. Utilizes all slave network interfaces in the active aggregator group according to the 802.3ad specification. Adaptive transmit load balancing (balance-tlb) [Linux driver mode 5 ] Linux bonding driver mode that does not require any special network-switch support. The outgoing network packet traffic is distributed according to the current load (computed relative to the speed) on each network interface slave. Incoming traffic is received by one currently designated slave network interface. If this receiving slave fails, another slave takes over the MAC address of the failed receiving slave. Adaptive load balancing (balance-alb) [Linux driver mode 6 ] Includes balance-tlb plus receive load balancing (rlb) for IPV4 traffic, and does not require any special network switch support. The receive load balancing is achieved by ARP negotiation. The bonding driver intercepts the ARP Replies sent by the local system on their way out and overwrites the source hardware address with the unique hardware address of one of the NIC slaves in the single logical bonded interface such that different network-peers use different MAC addresses for their network packet traffic. 26

31 D Appendix D: Tools NOTE: The following tools are not preloaded on HPP systems nor available via HDS sites. The links below are provided for references purposes only. Useful freeware can be obtained at the following sources: This tool is great for testing Windows networking performance: This tool will benchmark TCP and UDP performance between 2 hosts. Very useful when trying to determine network performance/latency issues. 27

32 Hitachi Data Systems Corporate Headquarters 2845 Lafayette Street Santa Clara, California U.S.A Regional Contact Information Americas Europe, Middle East, and Africa +44 (0) Asia Pacific A m hds.marketing.apac@hds.com ZZ-99HPP999-99