Int'l Conf. Software Eng. Research and Practice SERP'15 157 cnetmon: Ncurses-based Network Interface Activity Monitor Steve Hutchinson 1, John Wittkamper 1, Jovina Allen 1, Robert F. Erbacher 2 1 ICF International for US Army Research Laboratory, Adelphi, MD 20783 2 US Army Research Laboratory, Adelphi, MD 20783 Abstract - This report illustrates the development and use of a network interface activity monitoring tool named cnetmon. This tool is intended to aid system administrators and developers with network-oriented software projects. The main objective for this project was to develop a capability to monitor network activity for all or selected interfaces on a system simultaneously and continuously. We use a display generated by the Linux ncurses library that is updated using a configurable interval. We show added capabilities including interactive response to window-resizing using SIGWINCH. A novel debug-line display capability is provided to show dynamic debug messages on a dedicated line of the display. Keywords: network traffic monitoring, network interface, systems administration, ncurses 1 Introduction cnetmon 1 is a very lightweight command-line tool to display network traffic (packet activity) on any or all of the network interfaces (NIs) on a Linux-based system It uses a ncurses-library-based display that is compatible with any character-based pseudo terminal, and as such, does not require the use of the system graphical user interface (GUI) or Xserver:DISPLAY. cnetmon is intended for use in the field for remote access into devices such as (network) sensors or other network-attached Linux systems when an administrator with user-level access needs to obtain a dynamic indication of all network traffic entering and leaving that system. Because it does not use the GUI, the complexity and access requirements are very minimal. cnetmon can be invoked by any logged-in user, it does not require sudo access, and it can operate within a typical secure shell (ssh) or telnet session. 1 Throughout this paper, Linux commands are set in an italic font. 2 Motivation Server farms, cloud computing, compute clusters, and grid computing are all examples of a common technique to combine multiple computer systems into a cooperative network of systems. These systems often intercommunicate using two or more NIs (on each system). Clusteredcomputers are often rack-mounted for higher density and, as a result, often lack a keyboard or monitor; therefore, they are frequently managed and configured remotely via ssh or telnet over a network connection. During system configuration, installation, and testing, it is often difficult to determine whether network traffic is being sent and received by each interface. In general, such systems are built and configured in a central location and then shipped to remote locations to be added to other servers in a system rack or as a single distributed sensor. cnetmon allows the installer to observe network traffic from each or all NIs to verify that the system seems properly configured for the installed environment. It also does not require the use of the system GUI or Xserver/client because cnetmon will create tabular displays of all traffic using the LIBCURSES library for display on any attached ASCII terminal emulator. cnetmon can be used from a remote location, accessed and invoked typically from a ssh command-line, and can be invoked by any logged-in user; it does not require root-level access. Many techniques to observe or sample traffic from any NI require super-user privileges, but obtaining elevated privileges is often forbidden, hence a benefit of cnetmon. In this paper, we describe a few use-cases for cnetmon. First, cnetmon can be used on a laptop computer, which often has two NIs: wired (eth1) and wireless (wlan), along with the internal loopback interface. Laptop-users often must transition between networks without rebooting. cnetmon is easily invoked from a command window and will show all NI activities to verify communications to the desired network(s). Second, on a desktop or small server with multiple wired or wireless interfaces, cnetmon can show all network activity for each interface dynamically in this more complex network topology. Third, compute-server administration and configuration tasks are often performed using a separate administrative system and command-line tools. cnetmon facilitates server configuration and testing and was developed for use in these more complex, multi-network
158 Int'l Conf. Software Eng. Research and Practice SERP'15 environments. We frequently use one cnetmon window per server during configuration, development, and testing, to obtain a real-time picture of network inter-communications and to verify proper configuration and operation. 3 Related work: bmon In the search for a user-level, multi-ni monitor, we noticed the bmon tool [1], which provides indications of network bandwidth utilization from multiple interfaces using the /proc/ file-system [2] and a curses-interface. We use this strategy to implement a curses-based multi-interface activity tool, cnetmon, providing various command-line and keypress event-driven parameters to control the display and monitoring update interval. Although bmon was intended to show network bandwidth utilization, we liked its design paradigm using a ncurses display using periodic updates obtained from /proc/net/. Our goal was not to show estimated bandwidth utilization, but to show concurrent network activity measured in terms of packet counts and transfer rates per sampling interval and accumulated for the session. 4 How it works A long-standing problem for understanding network activity between (Linux or *nix) systems has been the requirement to obtain root or super-user privileges to access and configure devices, such as a NI. ifconfig is the Unix or Linux command to display the status of NI devices on a system. Upon executing the ifconfig command, the following information is produced on the console, shown below in Figure 1. The first 6 lines pertain to the hardware and network address parameters for each interface as well as the status of the interface. The remaining lines show counts of transmitted and received packets, error counts, and finally the interrupt number and buffers memory location. user@asc2:~$ ifconfig eth0 Link encap:ethernet HWaddr 00:24:81:1c:fd:7d inet addr:10.0.0.16 Bcast:10.0.0.255 Mask:255.255.255.0 inet6 addr: 2601:a:4680:3e6:5cf:ea3d:eed0:64e0/64 Scope:Global inet6 addr: fe80::224:81ff:fe1c:fd7d/64 Scope:Link inet6 addr: 2601:a:4680:3e6:224:81ff:fe1c:fd7d/64 Scope:Global UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:370 errors:0 dropped:0 overruns:0 frame:0 TX packets:120 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:46300 (46.3 KB) TX bytes:20936 (20.9 KB) Interrupt:19 Memory:f0500000 f0520000 Figure 1. Typical ifconfig output. Although it is true that we could issue ifconfig repeatedly to obtain the configuration and counts for network devices, this function call is not intended for repeated invocation to determine network traffic rates. Modern Linux systems provide a /proc/ file system to allow user-level processes to easily read a wide variety of counts for devices; these values are maintained and updated by the kernel in a virtual file system, /proc/. The /proc/ file system was originally intended as a way to provide information about processes in a system. As such, it also was a convenient means of exposing kernel information to a structured file system requiring only user-access rights to read this information. A corresponding application programming interface (API) is provided for read and write access using sysctl (system control) calls to configure parameters of the running kernel [3]. This capability was gradually introduced into Unix systems starting as early as 1984; the current implementation in Linux is as an extended, virtual file system contained only in memory and has directories for other kernel information categories such as kernel-modules, filesystems, interrupts, and devices including NIs, kernel messages, drivers, and CPUs. The cnetmon executable periodically examines the /proc/net/dev file on the Linux system. These values are sampled on each loop cycle (by default, one second), which is configurable on invocation or by pressing a number-key while running. Linux systems also maintain an uptime value, the number of seconds since last rebooting. cnetmon saves this date-time value at launch (fork) time and displays the session length time in the screen header section Contents of /proc/net/dev: Interface lo: bytes 570671 packets 6267 frame 0 multicast 0 bytes 570671 packets 6267 colls 0 carrier 0 eth0: bytes 14797900909 packets 17797994 frame 0
Int'l Conf. Software Eng. Research and Practice SERP'15 159 multicast 3120 bytes 4116686178 packets 14414011 colls 0 carrier 0 5 Implementation The design goals and requirements for cnetmon are to periodically examine the network device-file in the /proc directory on a Linux system to: Enumerate NIs Collect traffic statistics Convert traffic counts to display quantities and units Allow a variety of command-line arguments We also provide a release make/build capability for most Linux systems (including embedded devices, such as Raspberry Pi, etc.) After initialization during which command-line arguments are parsed, cnetmon enters the main_loop. With each pass through main_loop, it obtains new counts for packets, bytes, errors, drops, collisions, etc., and calculates display values as requested updating the ncurses display at the end of each interval. Display values are calculated from the following: Li update loop interval, in seconds Tu Linux uptime in seconds (since reboot) Tnow current Linux system time, epoch time seconds T0 cnetmon invocation start timestamp in epoch time seconds Ls session time length in seconds: (Tnow T0) P[i] packet count parameter from /proc/net/dev, at time interval = i B[i] byte count parameter from /proc/net/dev, at time interval = i For each interface and at each interval: SessionPKT s P[ Tnow ] P[ T 0] (1) IntervalPK Ts P[ Tnow ] P[ now i] (2) SessionRat e ( B[ Tnow ] B[ T 0]) /1000 * Li] (3) IntervalRa te P[ Tnow ] P[ T 0] (4) Command-line programs used for monitoring often generate display data output in the form of one-line records and then render them into a scrolling console window. Very wide, or multi-line records, when scrolled like this, are difficult to understand. Since network interface data is of this nature, a scrolling display will be difficult to use. Instead, we use a display technique that renders these parameters in strict rows and columns such that the location of each on the screen does not change. This tabular process makes the changing parameters more obvious. Cell contents can change with the fixed regularity of the chosen update loop interval. Although this is a somewhat primitive display technique compared with GUI implementations, such a capability is easily provided by the Linux, Ncurses library. Ncurses allows development of rather sophisticated tabular displays, useful in situations in which a GUI display is unavailable (as would be the case for many headless server or computeclustered environments). 6 Ncurses library Ncurses [4] stands-for new curses a reimplementation of the curses library to use a text-based terminal to emulate a more dynamic interface that has some attributes of a modern GUI. Curses was originally developed at the University of California at Berkeley for a Berkeley Software Division (BSD) release around 1980. Ncurses contains enhancements to curses and was made available starting in the mid-1990s under a Permissive free software license and not the General Public License (GPL) to afford wide redistribution and linking to this library.
160 Int'l Conf. Software Eng. Research and Practice SERP'15 interface row-sets that will fit in the new window and updates the display generation parameters in ncurses without resetting any of the current packet counts and rates. Figure 2 illustrates the various sub functions within main_loop, showing the generation and response to window size changes. Figure 3 below shows the help message with option switches and their meaning. Help message: cnetmon -H cnetmon [ad:ehhi:lm:n:rttu:] -a Show errors, data rate & totals (-ert) -D # Debug level (0-15) -e Show error data -H Help message -i name Ignore interface name -L List interfaces (with some statistics) -m name Show only interface name -n # Show total bytes for system uptime -r Show data rate -t Show data totals -T Show total bytes the Quick display -u # Update frequency, seconds (default 1) Interactive: d/d Scroll down interface list q/q Quit r/r Reset Session time u/u Scroll up interface list 1-9 Load value into interval time Figure 3. Usage help message. Figure 2. Resize of window to reveal additional interface row-sets. cnetmon, like many other ncurses programs, obtains terminal window geometry parameters from the terminal emulator when the program is launched. The combination of command-line switches will determine the number of rows needed to describe each interface; by default, the display will require one row per interface with the addition of three header rows. Use of the -a switch will result in the display of 7 or more rows per interface. cnetmon calculates how much space (height) is needed, and then it only displays as many interface row-sets as can fit in the current window geometry. Use of up/down (U/D keys) allows the user to scroll up or down an interface (row-set) at any time. To avoid requiring the user to quit, resize the terminal, and re-launch in order to see additional interfaces, we support dynamic changes in window size using the SIGWINCH signal (window change), which is supported by most terminal emulators. When the user changes the window geometry, the program receives the SIGWINCH signal and obtains new window geometry. cnetmon recalculates the number of 7 Usage scenario An actual usage scenario is shown below. We have an existing Linux server (Ubuntu 14.04 server) that will be used to provide various services to three separate networks, shown in Figure 4 as Internet, MeshNet_1, and MeshNet_2. This server does not have an attached display. We use a 2 nd system with a terminal emulator and establish ssh session to the server. We copy the cnetmon executable onto our /home/user/ directory using scp (secure copy command). This session is established through the Internet and gateway attached to eth0. Invoking cnetmon, we easily observe network activity on eth0 and no activity on eth1 or eth2. cnetmon does enumerate other interfaces such as the local loopback (lo) and a virtual bridge for use by associated libraries to offer network address translation (NAT). It is normal for local loopback to accumulate and show significant traffic during network traffic sessions as it is used for process-process communications. We then connect a second network (MeshNet_1) gateway to eth1. This interface had been configured already to accept a DHCP-issued
Int'l Conf. Software Eng. Research and Practice SERP'15 161 address. cnetmon clearly showed packets corresponding to DHCP requests and lease responses. After obtaining another user shell (ssh) to the server, we were able to access the web admin service on the gateway to continue configuration of this network. We then connected MeshNet_2 gateway to eth2. cnetmon observed no activity on eth2. This required further investigation. /etc/network/interfaces is the configuration file used by Linux systems to initialize and configure all NIs. eth2 had not yet been configured, and it was activated by adding the following to /etc/network/interfaces (these must be done as admin or root access): auto eth2 iface eth2 inet dhcp This change required restarting the networking services: sudo /etc/init.d/networking restart cnetmon showed no eth2 activity after restarting the network services. Next, we tried a shutdown reboot which did reconfigure the interfaces and driver. After rebooting, cnetmon showed activity on all three physical NIs as well as the virtual loopback interface. Figure 4. Server with connection to 3 networks. If the server had been pre-configured prior to installation, it is likely that cnetmon would have allowed us to observe and verify each of the network gateway additions in real-time at power-on. In this case, additional configuration requiring root-level access was required. We were able to observe resulting network activity in real time using cnetmon in a second session window. 8 Compute-server example To illustrate additional capabilities of cnetmon, we show results from running it on a blade-server with 5 NIs. This type of server is common today and is used to populate the many rack spaces at internet and content hosting facilities. Although this server has 5 NIs as shown in the DEBUG: line, the display window geometry affords
162 Int'l Conf. Software Eng. Research and Practice SERP'15 space for only 2 complete record sets, (lo) and (em1) shown in Figure 5. This server has been up (running) for just over 105 days and cnetmon has been running for 85 seconds, updating at 1-second intervals. Figure 5. cnetmon D 1 r showing counts and rates for the session and for the last main_loop interval. Notice this also shows the D flag, which adds an additional debug-message line to the display. Here, a cd_printf statement has been included to show the first and total number_interfaces available.
Int'l Conf. Software Eng. Research and Practice SERP'15 163 9 Debug print Coding and debugging an ncurses program can be very challenging. To facilitate debugging, we incorporate a debug display activated using a command-line switch. The code snippet below from cnetmon.c :main() illustrates how to print messages to the debug message line using the -D 1 command-line argument. // Step through device list for (j = 0, i = first_interface; i <= number_interfaces; i++) { int display = 1; if (!prog_flags.first_time prog_flags.match_inface ) { display = 0; } } ///////////////////////////////////////// cd_printf("first_interface:%d number_interfaces:%d", first_interface, number_interfaces); ///////////////////////////////////////// refresh (); // Update display 11 References [1] Travis Graf. bmon bandwidth monitor and rate estimator, retrieved from https://github.com/tgraf/, June 15, 2014. [2] Terry Dawson. Exploring the /proc/net/ directory, O Reilly, retrieved from http://www.onlamp.com/pub/a/linux/2000/11/16/linuxadmi n.html, March 26, 2015. [3] M. Tim Jones. Access the Linux kernel using the /proc filesystem, IBM developerworks Technical Library, 2006, retrieved from http://www.ibm.com/developerworks/library/lproc/index.html, April 15, 2015. [4] Free Software Foundation. Announcing ncurses release 5.9. Free Software Foundation, 2011, retrieved from https://www.gnu.org/software/ncurses/, March 26, 2015. 10 Conclusions We have shown how cnetmon can provide easy access to the network activity from multiple interfaces, on multiple systems; however, the executable must first be available on each system. Therefore, we intend to provide cnetmon to be available as opensource code, providing the sources, documentation, a makefile, and a pre-compiled, 32-bit binary. Although most systems today are 64-bit architecture, the precompiled 32-bit binary should run on almost any Linux operating system. A sophisticated developer-user can recompile cnetmon from the sources, possibly adding new features and debugging cd_printf statements to facilitate the application and intended uses. We also will approach major Linux packagers and distribution groups, notably Red Hat, Fedora and Ubuntu, to encourage inclusion of cnetmon in future distribution releases.