U.S. Air Force T&E Days 2010 2-4 February 2010, Nashville, Tennessee AIAA 2010-1744 Network Management Tools for Tactical Network Testing and Monitoring on Test Ranges William Brock 1 and Doug Mace 2 Tactical Communications Group, LLC, Tewksbury Massachusetts Abstract: Network Management Tools for Tactical Network Testing and Monitoring on Test Ranges The emergence of wideband tactical networks has introduced both problems and opportunities for the test community. The ongoing effort to perform increasingly robust mission testing has exposed limitations in the tools available for configuring, loading, and monitoring both the networks under test and the networks used for collecting test data. This paper discusses techniques for network management that will enhance monitoring and control of network stability. Using small and efficient SNMP agents, the test manager is provided with continuous assessment of network health and performance. Nomenclature ASN = Abstract Syntax Notation IP = Internet Protocol JMETC = Joint Mission Environment Test Capability LAN = Local Area Network MIB = Management Information Base NMS = Network Management Station OID = Object Identifier RF = Radio Frequency RFC = Request For Comments SNMP = Simple Network Management Protocol TCG = Tactical Communications Group TDL = Tactical Data Link VMF = Variable Message Format VPN = Virtual private Network 1. Introduction In the testing and deployment of network-centric applications, one of the more elusive objectives is obtaining a consistent and reliable status of network health. This paper describes some basic techniques and technical approaches that are being used to improve the control and monitoring of network operations. 1.1 Why are we doing this? Tactical Communications Group, LLC (TCG) builds tools and applications to assist with testing and training for tactical data links. The tools include simulators, recorders, analyzers, and other applications to assist the test engineer with a managing a complex operating environment. As more and more platforms implement data link operations, the configuration of networks for testing and the tactical networks themselves are becoming more and more complex. We deal multiple tactical networks, RF networks, and terrestrial backbone networks connected via a variety of gateways, routers, relay nodes and cryptographic devices. There may be multiple protocols linked via 1 Senior Technical Consultant, TCG. 2 Senior Software Engineer, TCG 1 Copyright 2010 by Tactical Communications Group, LLC. Published by the, Inc., with permission.
gateways and forwarders. When there are problems with the Tactical Data Link (TDL) network or the supporting test network, it becomes increasingly difficult to isolate, diagnose, and correct problems quickly. Regardless of the type of testing being conducted, operators are under the gun to make optimal use of costly test resources. Recent work with IP based wideband networking waveforms identified a need for a more robust network management scheme. We have embarked upon a project to model the problem domain and evaluate the use of conventional Simple Network Management Protocol (SNMP) agents to collect, analyze, and report network status in real time. 2. Description of Environment The testing community is always looking for better and more efficient methods to operate and monitor a complex configuration of multiple network attached devices. Our current project is funded by internal R&D as a product development and improvement program. If the concepts prove valuable to our customers, we will seek additional funding to help tailor the products for specific customer needs and environments. The desire is to design the capabilities to be flexible enough to host on a variety of different platforms. If we have a way for each component in the test network to report its status to a central monitor, debugging and problem correction will be much quicker and efficient. 2.1 Test Environments TCG currently provides products to support several types of testing. For many test applications, management of the test network is not a significant problem. There are the large systems integrators who use the tools for data link developmental testing and subsequent integration testing for data link equipped platforms. These types of applications generally require minimal management of the test network because the testing is conducted in a closed environment and is well controlled. But today, in the area of tactical data links, we are seeing rapid growth in the need for distributed testing. The concept of network-centric operations puts much more emphasis on an internet/intranet backbone that provides connectivity to the components in the operational environment. One of the primary drivers towards more distributed testing is the availability of the persistent JMETC VPN to many of the major test organizations. This has led to more frequent test events and an increasing complexity in the networks being used. Most of the complexity is introduced with the need to interconnect distributed intranets and LANs at labs that are geographically dispersed. The test participants and their local networks sit behind routers, firewalls, and miscellaneous cryptographic equipment. There are many more opportunities for operational problems. Then there are the live tactical networks themselves. We re working with RF networks like Link 16 and VMF each requiring its own version of network management tools that may or may not be interoperable and may not offer direct visibility to the test director. 3. Test Network Vision What we would like to see is a common and consistent mechanism to control and monitor the test environment. We need a test environment that is resilient, relatively easy to implement, and flexible enough to be deployed in most configurations. We re looking to see if there is a low-cost way to implement a monitoring system that will give us a continuous stream of status data that will help quickly identify and help diagnose any problems that may arise. What kind of things do we need to monitor? A typical test network may consist of any number of the following devices. - Routers - Firewalls - Crypto equipment - Switches/Bridges - Host computers Windows/Unix - Tactical Radios - RF Network (TDL) Basic network device status information needed to monitor the network would include: - Device operational status - Throughput rates 2
- Connectivity matrix (what paths can each device see?) - Error/Alert conditions Because of its availability, simplicity, and relatively low cost of implementation, we plan to implement this capability using Simple Network Management Protocol, SNMP. 4. Simple Network Management Protocol 4.1 History & Background SNMP was originally defined in 1989 as a follow on from SGMP (Simple Gateway Monitoring Protocol). SNMP is defined in a number of Request for Comments (RFCs) dating back to 1989. There is a list of the relevant RFCs at the end of this paper. The different versions of SNMP are the SNMPv1, SNMPv2c, and SNMPv3. Versions v1 and v2c lack any security related features. Version v2 can be used with any version of SNMP (including SNMPv1). Version v3 defines a secure version of SNMP and provides for remote configuration management of SNMP enabled physical devices. SNMP can perform the following basic tasks related to Network Management and Monitoring: - Configuration Management Keeps track of device settings and how they function - Fault Management Deals with problems and emergencies in the network (router stops routing, server loses power, etc.) - Security Management Controls access to information on the data network. Provides audit trails and sound alarms for security breaches. - Performance Management How smoothly is the network running? Can it handle the load it currently has? - Accounting Management Keeps track of individual usage and grouping of devices to ensure users have sufficient resources. 4.2 SNMP Advantages The SNMP framework offers a number of distinct advantages for implementing a network monitoring architecture. Although the word Simple is in the name, SNMP can be used for a number of complex operations using a small set of commands that are relatively easy to implement. SNMP was not designed to handle large amounts of data in a single operation. This makes any processing very efficient, imposing a very small overhead load on the test network. Because of the structure of data, it is extremely flexible and can be adapted to virtually any device. The data structure is discussed below. And finally, it is mature standard that has been widely implemented. There is an abundant availability of commercial & open source products to assist with implementing agents. Many of these provide object-oriented APIs that make it easy to integrate with C++ and Java applications. And finally, most commercial network-attached devices, such as routers, switches, servers, printers, and computers, already have some SNMP capability. So we re rarely starting from scratch when we implement SNMP on a network device. And because of the inherent flexibility of the protocol, it is easy to extend the existing applications to meet any custom requirements of testers. 4.3 Structure SNMP is characterized as an IP layered Request / Response Application protocol. This is a four layer networking model that consists of: - Application layer SNMP 3
- Transport Layer User Datagram Protocol (UDP) - Internet Layer IP - Network Interface Layer - 10/100/1000 Base-T, 802.3x SNMP uses a Manager / Agent model. Each device to be monitored or controlled will host an SNMP Agent, or simply agent, that responds to requests or commands sent by an SNMP Manager. 4.4 SNMP Components The following is a short tutorial that explains the components that make up SNMP and how they apply to the TDL test environment. 4.5 Management Information Base (MIB) The data for each managed node is stored in a Management Information Base, MIB. The MIB is a treestructured database consisting of a collection of objects organized with the help of Object Identifiers, OID. The MIB relates an OID with a text label and its object parameters represented using Abstract Syntax Notation One (ASN-1) to define abstract data types in a machine independent manner. The MIB then serves as a dictionary that can be used to interpret SNMP messages. The SNMP agent is responsible for maintaining the MIB database. 4.5.1 Object Identifier (OID) Each SNMP MIB element or Object identifier is laid out in a hierarchy forming unique addresses into a tree structure. Any node or location on the tree can be referenced through a dotted (. ) string notation. If one used a Network Management Station (NMS) commonly known as a SNMP browser application to scan or walk the MIB tree for a Canon network printer, we would be able to see all of the parameters that the printer supports. Example: MIB for Canon ir3225 Printer/copier Symbolic:.iso.org.dod.internet.private.enterprises.1602. Actual:.1.3.6.1.4.1.1602 Graphically, the tree would look something like the following. root (.) - ccitt (0) - iso (1) org (3) dod (6) o internet (1) directory (1) mgmt (2) experimental (3) private (4) enterprise (1) o canon (1602)... - joint-iso-ccitt (3) Example of other Enterprise IDs 1.3.6.1.4.1.311 - Microsoft, Inc. 1.3.6.1.4.1.674 - Dell Computers, Inc 1.3.6.1.4.1.5771 - Cisco Systems, Inc 4.6 The SNMP Agent As previously stated, the SNMP Agent is a software process within the network device that responds to SNMP protocol requests or commands sent by an SNMP Manager. SNMP typically uses Port 161 for command messages 4
and responses, and Port 162 for receiving notifications (TRAP and INFORM) messages. Now, if the SNMP agent is connected to a device that is not SNMP enabled, it can serve as a proxy agent for that device. It is just a matter of defining our own (user defined) MIB branch for the non-snmp device. This becomes most useful when we want to monitor some of the older data link radios. In the case of Link-16, we could get a remote indication of each radio s net entry status, connectivity matrix, and other relevant information that previously was accessible only through the radio s network management messages. 4.7 Commands The SNMP command set is relatively easy to understand. It includes the following operations: - GET The GET command results in a read operation that returns a single data object. - GET-NEXT When the Manager wants to retrieve a series of data objects that are located on the same branch, it uses one or more GetNext commands to walk along the branch retrieving subsequent objects. - GET-BULK (SNMP V2) The GetBulk command is used to retrieve multiple data items with a single command. It is not supported by V1 devices. - GET-RESPONSE The GET-RESPONSE is the message sent back to the Manager in response to one of the above Get messages. The response will contain either the information requested by the Get or notification of some error condition. - SET The Set command is used by the Manager to modify the value of a data object in the Agent. It is a write operation commonly used to set configuration parameters on the managed device. - WALK The Walk command is used by the Manager to issue a series of GET-NEXT requests to query for a tree of information about a network device. - TRAP and INFORM The Trap and Inform message is initiated by the agent in the managed device. This message is used by the agent to preemptively alert the Manager of some abnormal event or condition that has happened. The advantage of this message is that it eliminates the need to continuously poll the agent to monitor a particular condition. The Inform message is nothing more than an acknowledged Trap event. Since Traps may be lost, the sending entity can keep sending the Trap message until the trap gets through. Of course, consideration must be given for prevention of Trap and Inform flooding during major problem times (power outage, etc). Some useful purposes for the Trap and Inform message would be to Report periodic status keep alive, Fault or Error conditions, Timeouts, or Error condition cleared, LinkDown and LinkUp conditions. 4.8 The SNMP Manager The SNMP Manager or NMS is a host application that retrieves and reports information collected from the agents. For our Test Network Management system we are particularly concerned with establishing the ability to monitor all network paths for our test data. This is going to be the central monitoring node that will alert the test director if there is some significant anomaly in the backbone network. 5
4.9 Operational Scenario During operation of the Test Network Management System, our first task will be to identify each component in the specific test configuration. The first levels of components are those that are directly accessible via the IP network. These are the routers, switches, host computers, etc. that make up the backbone network of the system or systems under test. In most cases these already have some SNMP capability. The operator will prepare a script that contains the following information for each device. 1. Device Name 2. Device Type 3. IP Address 4. SNMP Version 5. Location of device MIB Files when needed by the NMS application. 6. List of event Traps (Event/Status, priority, processing rules) The SNMP Manager will then read the script file and perform a network scan to build the various tables and lists detailing each device to be monitored. The User Interface (UI) will provide a summary display showing all of the monitored devices and their current status. See the example monitor display, Figure 1 Figure 1 - Sample Monitor Display When an alert condition is detected, the operator can select the suspect device and if it is still operational, query for a more detailed status. If a device stops reporting it s keep alive status, the operator will be notified and appropriate action may be taken. Once again, the objective is to identify faulty network components quickly so that action can be taken to resolve the reported condition. 4.10 Implementation Methodology The basic requirements for the Test Network Management System are: 1. Load the MIB files for each network device if needed by the NMS application. 2. Identify each node in the network by its IP address and device type and tag it with a logical name. 3. Identify and configure Trap events for each device. 4. Identify and list data path dependencies. (If there is an error condition, what portions of the test will be affected?) 5. Provide a control panel or dashboard to facilitate operator access to network data objects. 6
6. Provide an interface to configure network devices as necessary. 7. Log and time stamp all data requests and responses collected by the SNMP Manager. As previously stated, a primary reason for using SNMP is the widespread availability of existing applications and support tools. There may be instances where it is not practical or possible to access a device via SNMP. In these cases, the Manager can send periodic ICMP ping (echo) messages to confirm that a component is accessible and active. And, of course, ping may be used as a troubleshooting aid and isolation technique when a problem is detected. Table 1, below shows a notional approach to setting up a monitoring scheme for a typical data link test network. Note that for Windows or UNIX based computers, it is possible to wrap the standard SNMP agent within a custom extended agent that can provide additional application parameters or act as a proxy agent for attached devices. Device/Network Type Access Method Parameters Monitored O = Device OK R = RF Network Status T = Throughput C = Connectivity Matrix S = Special Trap Events T = Timeout/NR E = Alert/Error S1 = Special 1 S2 = Special 2 O R T C S T E S1 S2 Router SNMP X X X X Firewall SNMP/Ping X X X X LAN Switch SNMP X X X KG Equip. Proxy X X X X X Windows Host SNMP X X X X UNIX/Linux Host SNMP X X X SIMPLE Proxy X X X X X DIS / HLA Proxy X X X X X TENA Host SNMP X X X X Link 16 Terminal Proxy X X X X X MIDS/JTRS Proxy X X X X X TTNT Radio SNMP X X X X X JRE SNMP/Proxy X X X X EPLRS SNMP/Proxy X X X X X Link 11 DTS Proxy X X X X X Link 22 SNC SNMP X X X X X WNW SNMP X X X X X Table 1 - Network Device Agent Monitoring Capabilities 5. Conclusion There is an opportunity to develop a Test Network Management System using underlying protocols and techniques that have been successfully deployed in commercial networks for years. SNMP has proven to be a lowcost efficient protocol for many different network environments. The challenge for us is to develop adaptors for many of the non-snmp devices that are deployed in Tactical Data Links. TCG plans to develop monitoring agents and applications that can be configured for virtually any test network. We will continue to work with our customers to help improve the availability and reliability of their test environments. 7
6. References For those wishing to investigate further, following is a list of the RFC references that define SNMP and its operation. RFC 1089 - V1; SNMP over Ethernet. M.L. Schoffstall, C. Davin, M. Fedor, J.D. Case. February 1989. RFC 1157 - V1; Simple Network Management Protocol (SNMP). J.D. Case, M. Fedor, M.L. Schoffstall, C. Davin. May 1990. RFC 1187 - V1; Bulk Table Retrieval with the SNMP. M.T. Rose, K. McCloghrie, J.R. Davin. October 1990. RFC 1215 - V1; Convention for Defining Traps for use with the SNMP. M.T. Rose. March 1991. RFC 1228 - V1; SNMP-DPI: Simple Network Management Protocol Distributed Program Interface. G. Carpenter, B. Wijnen. May 1991. RFC 1270 - V1; SNMP Communications Services. F. Kastenholz. October 1991. RFC 1303 - A Convention for Describing SNMP-based Agents. K. McCloghrie, M. Rose. February 1992. RFC 1351 - SNMP Administrative Model. J. Davin, J. Galvin, K. McCloghrie. July 1992. RFC 1352 - SNMP Security Protocols. J. Galvin, K. McCloghrie, J. Davin. July 1992. RFC 1353 - Definitions of Managed Objects for Administration of SNMP Parties. K. McCloghrie, J. Davin, J. Galvin. July 1992. RFC 1442 - Structure of Management Information for Version 2 of the Simple Network Management Protocol (SNMPv2). J. Case, K. McCloghrie, M. Rose, S. Waldbusser. April 1993. RFC 1443 - Textual Conventions for Version 2 of the Simple Network Management Protocol (SNMPv2). J. Case, K. McCloghrie, M. Rose, S. Waldbusser. April 1993. RFC 1444 - Conformance Statements for Version 2 of the Simple Network Management Protocol (SNMPv2). J. Case, K. McCloghrie, M. Rose, S. Waldbusser. April 1993. RFC 1445 - Administrative Model for Version 2 of the Simple Network Management Protocol (SNMPv2). J. Galvin, K. McCloghrie. April 1993. RFC 1446 - Security Protocols for Version 2 of the Simple Network Management Protocol (SNMPv2). J. Galvin, K. McCloghrie. April 1993. RFC 1448 - Protocol Operations for Version 2 of the Simple Network Management Protocol (SNMPv2). J. Case, K. McCloghrie, M. Rose, S. Waldbusser. April 1993. RFC 1449 - Transport Mappings for Version 2 of the Simple Network Management Protocol (SNMPv2). J. Case, K. McCloghrie, M. Rose, S. Waldbusser. April 1993. RFC 1503 - Algorithms for Automating Administration in SNMPv2 Managers. K. McCloghrie, M. Rose. August 1993. RFC 2271 - An Architecture for Describing SNMP Management Frameworks. D. Harrington, R. Presuhn, B. Wijnen. January 1998. RFC 2272 - Message Processing and Dispatching for the Simple Network Management Protocol (SNMP). J. Case, D. Harrington, R. Presuhn, et al. January 1998. RFC 2273 - SNMPv3 Applications. D. Levi, P. Meyer, B. Stewart. January 1998. RFC 2570 - Introduction to Version 3 of the Internet-Standard Network Management Framework. J. Case, R. Mundy, D. Partain, B. Stewart. April 1999. RFC 2571 - An Architecture for Describing SNMP Management Frameworks. B. Wijnen, D. Harrington, R. Presuhn. April 1999. RFC 2572 - Message Processing and Dispatching for the Simple Network Management Protocol (SNMP). J. Case, D. Harrington, R. Presuhn, B. Wijnen. April 1999. RFC 2573 - SNMP Applications. D. Levi, P. Meyer, B. Stewart. April 1999. RFC 2574 - User-Based Security Model (USM) for Version 3 of the Simple Network Management Protocol (SNMPv3). U. Blumenthal, B. Wijnen. April 1999. RFC 2575 - View-Based Access Control Model (VACM) for the Simple Network Management Protocol (SNMP). B. Wijnen, R. Presuhn, K. McCloghrie. April 1999. RFC 2576 - Coexistence Between Version 1, Version 2, and Version 3 of the Internet-standard Network Management Framework. R. Frye, D. Levi, S. Routhier, B. Wijnen. March 2000. 8