Open Source Tools for Monitoring the MPLS Nodes Project guide: Dr. N.P. Dhavale DGM, INFINET Department, Institute of Development and Research in Banking Technology (IDRBT) Road No. 1, Castle Hills, Masab Tank, Project By: Shrestha Pratik 5 th Year, Mathematics and Computing, Indian Institute of Technology, Kharagpur Kharagpur- 721302 Hyderabad 500 057 1
Date: 6 th July, 2012 CONTENTS Certificate 3 Acknowledgement 4 Introduction 5 IDRBT Network 7 Problems and Solutions 8 Parameters 9 Argus Prerequisites 10 Installation 11 Services 19 Interface 25 Discussion 28 Tool Comparison 29 Conclusion 30 References 31 2
CERTIFICATE This is to certify that project report titled Open Source Networking Tools for Monitoring the MPLS nodes submitted by Shrestha Pratik of M.Sc. 5 th year, Dept. of Mathematics and Computing, IIT Kharagpur, is record of a bonafide work carried out by him under my guidance during the period 8 th May 2012 to 9 th July 2012 at Institute of Development and Research in Banking Technology, Hyderabad. The project work is a research study, which has been successfully completed as per the set objectives. Dr. N.P. Dhavale DGM, INFINET Office IDRBT, Hyderabad 3
ACKNOWLEDGEMENTS I would like to express my sincere gratitude to the Institute for Development and Research in Banking Technology (IDRBT) and particularly Dr.N.P. Dhavale,(DGM, INFINET and Services) who was my guide in this project. This opportunity of learning all the nuances of a banking platform and a SLA (Service level Agreement) system application of the country was a boon to me as one rarely gets such exposure. I would not hesitate to add that this short stint in IDRBT has added a different facet to my life as this is a unique organization being a combination of academics, research, technology, communication services, crucial applications, etc., and at the same time performing roles as an arm of regulation, spread of technology, facilitator for implementing technology in banking and non-banking systems, playing a role of an NGO (without being one) and many more varied activities. I am extremely grateful to Dr.N.P.Dhavale for his advice, innovative suggestions and supervision. I thank him for introducing me to an excellent banking application and giving me the opportunity to approach diverse sections of people starting from bankers to general public. I am thankful to the staff of INFINET department at IDRBT for helping me to get familiar with the application. They gave me a chance to study the application and its impact from different perspectives. I am thankful to my college, IIT Kanpur for giving me this golden opportunity to work in a high-end research institute like IDRBT. I am also thankful to my team which worked tirelessly on this project and made it a success. Finally, I thank one and all who made this project successful either directly or indirectly. I am very thankful to Ms. Anuraddha Madam and Shri Shrihari Sir with whom I worked throughout my stint at IDRBT and the project was possible only with their cooperation. Shrestha Pratik Project Trainee Department of INFINET IDRBT, Hyderabad 4
INTRODUCTION A network can be loosely defined as a collection of two or more computers that have some sort of communication path between them. A network can be loosely classified as either a local area network (LAN) or wide-area network (WAN). The use of the terms LAN and WAN is somewhat misleading because which term you use is relative to the particular network installation you re describing. Generally speaking, a LAN covers a much more geographically restricted area than does a WAN. Whereas a LAN may connect computers within an office building, a WAN may connect computers spread across the country. With the advances in networking hardware and software, many widely dispersed LANs can now be connected to form a much larger homogeneous WAN. Network Architecture Devices known as bridges and routers allow for this connection of disparate LANs. Computer networks aren t new, but they weren t accepted in the personal computer realm until perhaps the late 1980s, when computer firms began offering cost-effective and reliable networking for the desktop PC. At that time, the primary goal of the PC network 5
was to provide a central repository for files and to allow printers to be shared among many users. It hasn t been until relatively recently that businesses have realized the true potential of a PC network. The goals of PC networking have been expanding over the last few years from simple file and printer sharing to access of fax machines, modems, and enterprise-wide electronic mail systems. All the while, the essential goals of networking have always been to share resources and to provide a medium for communications. A network resource is either a device or a capability on the network that s available for use by network users. The device that the network resources are attached to is called the server. Other devices that access those resources over the network are called clients. Typically, these devices include Gateways Routers Network bridges Switches Hubs Repeaters. Proxy Servers Firewalls and network address translators. Also, there are other hybrid network devices such as multilayer switches, protocol converters and bridge routers. 6
IDRBT NETWORK IDRBT (Institute for Development and Research in Banking Technology) started INFINET (Indian Financial Network). INFINET is the communication backbone for the Indian Banking and Financial Sector. It is a Closed User Group Network for the exclusive use of member banks and financial institutions and is the communication backbone for the National Payments System, which caters mainly to inter-bank applications like RTGS, Delivery Vs Payment, Government Transactions, Automatic Clearing House, etc. The network is a hybrid one of terrestrial leased lines and VSATs (Very Small Aperture Terminal) was the main communication backbone for inter-bank requirements. Over the years, with the decline in prices of leased lines, the reliance on VSATs for running applications declined. The VSAT technology also matured over the years with the increase in the size of the market and the number of private VSAT operators. The terrestrial leased line market also underwent significant change with the introduction of MPLS / VPN service being offered by many service providers. Further, as the technology had matured, the need for IDRBT to play a role of intermediary between the banks and the commercial VSAT operators also diminished. With the availability of better and more reliable technology in the form of Multi-Protocol Label Switching (MPLS), IDRBT decided to migrate the INFINET backbone to MPLS. The IP VPN MPLS network is an improvement over the Leased Line Network. The Leased Line Network is less scalable and since it is a partial mesh network, adding a new site to the network is difficult. Up gradation of bandwidth too is a time consuming and cumbersome process. Packet switching is slow compared to MPLS and the quality of service for applications too is not of a high standard. 7
PROBLEMS AND SOLUTIONS IDRBT while hosting the INFINET has to take care of the communication backbone of the inter bank transfer and other activities. As a result they have to constantly monitor the entire network RBI locations all across the country. If any link goes down, it needs to be reported to the network administrator and necessary actions have to be taken. This calls for the use of robust and stable network monitoring tools which can counter over these problems and gives worthwhile result. Our project had been a group project under the common heading of Open source Networking Tools for monitoring the MPLS nodes was to analyze as different network monitoring tools, which saw the broad division of the 13 preselected network monitoring tools both under open source as well as licensed version namely as described next. Open Source Networking Tool Zabbix Argus Nagios Cacti NetDisco Zenoss Spiceworks Open QRM Open NMS Frame Flow Licensed Networking Tool OpManager PRTG NetFlow Analyser On the part of the group project I was assigned the task to have a look over one of the tool and that was Argus. 8
PARAMETERS The parameters that we needed to monitor are mentioned in the table below Report Format for Monitoring MPLS Network using Key Parameters Sno Parameter Description 1 Device Availability ( MPLS CPE and Crypto Server ) Threshold Value Periodicity a Devices that is not reachable for more than 5 minutes Executive Report b CPU Utilization in % >70% c Memory Utilization in % >70% d Mean Time Between Failures One Month Monthly e Mean Time To Repair 4 hours 2 Interface Availability and Performance a Interface not reachable for more than 5 minutes. >5 Minutes Executive Report BGP Protocol Status >5 Minutes HSRP Protocol Status >5 Minutes Physical I/O Status >5 Minutes 3 Performance (Link, NNI Link and Crypto Server Link) a Link Availability <99.9% Executive Report Link Utilization >70% b Packet loss % >0.1% c CRC Errors >1% d Round Trip Time >100 msec e Latency >70 msec (Avg) f Jitter Sensitive g GET VPN Utilization >70% h GET VPN Status Monitoring Down i Site Availability (When Both links are down) Both Link/Boxes Down j Locations where Auto failover not happened 4 Crypto Server Monitoring a Crypto Session Status Down Executive Report b Auto Failover between Crypto Servers Auto Failover Not happened Any parameter deemed to be Key Performance 5 Application Wise (IP) and Port utilization. a Top 30 IPs Application wise utilization. 30 IPs Executive Report b Top 30 IPs Port wise utilization. 30 IPs 6 Report Summary a Calls Received from RBI and Member Banks Executive Report b List of RBI locations Managed by Primary SP and Secondary SP c RFOs Pending RFOs > 2 hrs d TT Raised with SIFY and Reliance Pending > 4hrs e Configuration Changes in MPLS network f Number of connectivity s MPLS network like Monthly addition/deletion/shifting of the connections. g Inventory of MPLS network (Upgrade in Box/Link) Monthly 7 Backup for entire Remote NMC Configuration for IDRBT Monthly 9
ARGUS PREREQUISITES Argus would need other software to run. This software runs on UNIX operating system. It is preferable to run it on Ubuntu. These software or prerequisites that Argus would be requiring are: 1. Fping: This software would help Argus in ping operations. These operations include finding devices. It also helps in monitoring status of a device and the amount of time taken to send a packet data. 2. Perl: the software has been tested with 5.6.1 and should work with most other versions of perl 5 as well. It is important to note that there are issues with some versions of perl on some operating systems 3. Sendmail or Qpage: One can download either one of this software. The main purpose of this software is in sending mails to the concerned administrator. One can skip this software if he or she doesn t need to send e mail alerts. 4. Apache: or any other cgi(common Gateway Interface) capable web server is needed to support the web interface of the tool. It is also handy in installation of other monitoring tools as well. Apache 5. Berkeley DB and Perl DB-File : Any database will also work but these two are preferable 10
INSTALLATION One can download the zipped file from http://argus.tcp4me.com/download.html. The current version hasn t been updated since Oct. 2008. The current version is 3.6. The code can also be found on the above site itself. After downloading one would need to follow the steps given below 1) Unbundle the tarball(the zipped file you just downloaded). It is preferable to unbundle the tarball through the terminal as one would be in the same directory for further 2) run./configure. If one is looking to upgrade you can try run./configure --upgrade 3) Answer any questions it asks. Screenshot 1.1 These general purpose programs are not required to edited later so one can use any give path which is not generally used. You can just press enter to allow the files to be put in the default pathway i.e. /usr/local/bin. 11
Screenshot 1.2 This one is for administrative files. Screenshot 1.3 This path is for the files which are going to be needed by the server. These files contain the icons and programs which are going to be needed by the interface of Argus. 12
Screenshot1.4 This one is for supplemental files. These mainly contain library files which Argus would be needing. Screenshot 1.5 This path is for data directory. This directory will be used further when making the config file and the users file. This path needs to be remembered as will be frequently used. 13
4) Run make, through the terminal. This file will make all the required directories that would be used to put the files. Screenshot 2: after running make command successful 5) As root, run make install. This will install all the required files at the right places. Screenshot 3: After running make install command successful 14
6) Now create two files in the data directory(which was mentioned before to remember): a) config file Screenshot 4 : A Sample config file b) users file: To control access, Argus wants to know who is permitted to access what. Associated with every valid user is: Username: Any username would do. Password: give any for no password. The password field is encrypted using your systems standard passwd crypt() function Home object (like a Unix home directory): The home_object field specifies the name of the object for the user's default page. This is what the user will see when they first log in to Argus. List of access control groups: only three levels are present which are user, staff and root with their respective levels of authority. 15
Screenshot 5: A Sample users file in addition to the documentation, the zipped file which one downloaded contains examples in the 'examples' directory. 7) Configure your web server and be sure that data directory is writable by the www user (or whatever uid your web server runs as). One can do that through root terminal using chmod 777 <filename/path for folder>. This will give all the permissions. 8) Start the Argus server by running argusd. The Argus server can be run only through root. Type in argusd or install the rc.argusd script as appropriate for your system. One can also check the status of the Argus server by typing in argusctl status as shown in the Screenshot 6. For switching off the server one can use argusctl shutdown. For any other assistance one can look for the help by the command argusctl help. 16
Screenshot 6: using argusd and argusctl status 9) Check the Argus log file ($data dir/log) and/or your syslog logs to verify that Argus is operating correctly. If Argus is running correctly it will give successful start - Argus running. Screenshot 7: Log file (sample) 17
10) Load the Argus cgi interface in your web browser, and verify that everything is configured correctly. Username can be found in the users file. Screenshot 8: Argus Interface 11) Perform any optional advanced configuration described in the advanced installation section. 18
SERVICES Services specify actual things to monitor. In the config file, services are specified as: Service NAME { data... } Often, there will be no need to specify any data, and you may shorten the specification to just Service NAME NAME specifies the type of test. There are 4 main types of tests: Ping - Pings a host Prog - runs a program TCP - tests a TCP port UDP - tests a UDP port If you are just starting out with Argus, it is recommended that you stick to the short form when possible and let argus fill in the default values. Both TCP and UDP have a number of application tests built-in. Specifying an application does the same thing as setting the various bits of data to values appropriate for the protocol. But you could just as easily specify them directly. For example, this code is how one can change the time of tests and their timeout periods etc. Service NAME { frequency: 60 retries: 5 timeout: 10 } These do not need to be specified, they will be taken by default, or propagate from an enclosing group Frequency specifies how often the service is tested, in seconds and not how many times in one second. 19
Retries is the number of times the test needs to be performed before declaring it down. Timeout is how long argus needs to wait for something (eg. a response from a mail server or test) before giving up and declaring the test a failure. This example tells how a Prog test works. Service Prog { # make sure no one deleted root from the passwd file command: cat /etc/passwd expect: ^root: } These need to be specified. They will not default, or propagate from anywhere Command the command to run, along with any command line arguments. Specify it just as you would to your shell. You should either specify a complete path to the program, or verify that it is in argus's PATH Expect : is a regular expression that needs to match the output from the command. if not specified, the exit code from the program will be used to determine success or failure Ping test can be performed like this example. Service Ping { hostname: } host1.example.com This needs to be specified, but will propagate from an enclosing group hostname : is the hostname or IP address to ping For TCP and UDP tests one would follow the below format. Service TCP { hostname: www.example.com port: 80 send: HEAD / HTTP/1.0\r\n\r\n expect: HTTP readhow: toeof } Both TCP and UDP have many of the same parameters 20
Hostname is the hostname or IP address to test. This needs to be specified, but will propagate from an enclosing group Port the port to test. This needs to be specified. Send data to send once connected. If nothing is specified, nothing will be sent to the remote server. Expect is the regular expression that needs to match the data received from the remote server. This needs to be specified. If nothing is specified, success or failure is determined by whether or not argus received any data at all Read how is the parameter which tell how much data needs to be read and stop. This needs to be specified. o toeof indicates read until the remote end closes the connection. o banner indicates we only want to read a banner. o once indicates to use whatever data is returned in the first read(). This parameter only applies to TCP not UDP Specifying an application (eg. TCP/SMTP) will fill in port, send, expect, and readhow as appropriate for that protocol See below for details on the special TCP/URL, UDP/Domain, and UDP/SNMP tests SNMP SNMP tests are the most important tests for a network monitoring tool. SNMP test work on the fact that every device has MIBs (Management Information Bases) which are databases which contain values of some specific sensor or object. Each of these sensor or object is given a number called Oids (Object Identifiers). The tool sends requests to these databases which the device returns the value to. These oids are needed for each test which is to be performed which one can get, like us, from the internet. These tests results are needed in order to get the parameters that are required. An SNMP (version 1) test is specified in the config file as shown below. Service UDP/SNMP { hostname: cisco-1.example.com community: qwerty oid:.1.3.6.1.4.1.9.9.13.1.3.1.3.1 maxvalue: 27 } Hostname the hostname or IP address to test. This needs to be specified, but will propagate from an enclosing group Community the SNMP community. This needs to be specified, but will propagate from an enclosing group 21
Oid the OID to query. It should be numeric, and the leading '.' is optional. This needs to be specified. Maxvalue the maximum permitted value, if the query returns something larger, we consider the service down. You can specify any of: o maxvalue o minvalue o eqvalue o nevalue SNMP test version 3 requires further more fields which can be specified in the form as given below. Service UDP/SNMPv3 { hostname: cisco-1.example.com oid:.1.3.6.1.4.1.9.9.13.1.3.1.3.1 snmpuser: joe snmppass: secret snmpauth: MD5 snmpprivpass: supersecret snmppriv: DES contextname: public contextengine: 80000000123456 authengine: 80000000123456 } snmpuser the snmp username This needs to be specified, but will propagate from an enclosing group snmppass the snmp authentication password This needs to be specified if authentication is required, but will propagate from an enclosing group snmpauth the authentication algorithm, MD5, SHA1, or none. This will default to MD5 if a password is provided, otherwise none. snmpprivpass the privacy password This needs to be specified if privacy is required, but will propagate from an enclosing group snmppriv the privacy encryption algorithm, DES or none. This will default to DES if a snmpprivpass is specified, otherwise none. contextname the context name This needs to be specified, but will propagate from an enclosing group contextengine the context engine-id This needs to be specified, but will propagate from an enclosing group, or will attempt to be auto-discovered. authengine the authentication engine-id This needs to be specified, but will propagate from an enclosing group, or will attempt to be auto-discovered. 22
Certain SNMP values are not useful themselves, but may be useful to watch after performing some manipulation. InOctets by itself is not useful, but using it to calculate a rate (or bandwidth use) in Bytes (or Bits) per second and comparing that value to a max or min is useful. Service UDP/SNMP { label: Out hostname: cisco-1.example.com # OutOctets (These are comments) oid:.1.3.6.1.2.1.2.2.1.16.2 calc: ave-rate-bits maxvalue: 20000000 # 20M bits/sec minvalue: 1000000 # 1M bits/sec } This will convert the OutOctets value into a rate (Bytes per second), calculate the moving average, and convert it to Bits per second. If the resulting value falls outside the expected range of 1-20 Mbps, the test will fail. TCP/URL The TCP/HTTP tests are tests that tells that the HTTP service is up and running. The TCP/URL test will test the content of a web page. Service TCP/URL { url: http://www.example.com/cgi-bin/shoppingcart.pl browser: Mozilla/5.0 (compatible; just testing) expect: cart contains 1 item } URL the url to fetch. it must begin with 'http://', and may contain an optional ':port'. It needs to be specified. Expect a regular expression that needs to match the data received from the remote server. This needs to be specified. If nothing is specified, success or failure is determined by whether or not we received any data at all Browser spoof the specified browser (send as the User-Agent). This is optional, and can be used if the server or application is serving different content based on browser. 23
UDP/DNS The UDP/DNS test (and UDP/DNSQ test) is a test that tells that a DNS server is up and running. The UDP/Domain test will test that a server is answering authoritatively for a zone. Service UDP/Domain { hostname: zone: } ns1.example.com example.com hostname the hostname or IP address to query. This needs to be specified, but will propagate from an enclosing group zone the DNS zone to check. as a shortcut, an alternative syntax is supported: Group "nameserver" { hostname: ns1.example.com Service UDP/Domain/example.com Service UDP/Domain/example.net Service UDP/Domain/example.org } 24
INTERFACE After coding the config file one can log into the interface by using the password set in the users file and come to the interface to see the alarms, Up and Down services. As for example is the snap shot below from our use. Screenshot 9: Argus Interface Top One can look into the config file from the config tab which can be towards the right side of the screen. One can look into the error log from the right tabs as well. These are readable versions from the config and log file found in the respective folders but they cannot be edited from the interface. Unacknowledged and otherwise, as well, notifications can also be found through the right hand tabs as well. By using these we can find which service has been working properly and which have been causing problems and needs to be fixed first. One can also get an overview by clicking on the overview tab. 25
Screenshot 10: Notifications As one can see the Ping service has returned to up status but other services are still down. This notification tab can also tell us when the last change occurred to a service. On further clicking on a particular notification, for example, say by clicking on the first notification one would see screenshot 11. Screenshot 11: An Up service 26
A Down service would like Screenshot 12. Screenshot 12: A Down service As one can see these pages tell us in much detail how much time the service has been down for periods of 1 day, 2 days, week, month and even year. It also tells us about what action of notification was taken and when. 27
DISCUSSION As was already decided after everyone had become familiar with their tools we weighed the positives against the negatives of each tool in order to finally find which software is best fit for our project s needs. Strong Points 1. Can monitor anything 2. Can be modified according to the user's needs and is fully customizable 3. Can monitor a wide variety of TCP/UDP applications 4. Can monitor the content of a web page (such as a shopping cart application) and enhance its security from outside threats as well 5. Can also monitor output or exit code of a program Problems 1. Not a good interface 2. Not a lot of graphs 3. Everything needs to be programmed from the config file and is not easy if one wants to change the devices or network configuration 4. The config file cannot be manipulated from the entire network even when the person has proper authority as it is not editable through the interface 5. Is outdated. No newer version seems to be coming up and not a lot of help on the internet forums Thus the decision was taken that Argus doesn t have the features we require. 28
T O O L C O M A P R I S O N The tool comparisons for all tools are shown in brief in the tables presented below. Table 1: Tool Comparison Table 2: Tool Comparison (Contd.) The tools shown in red are the tools that have been selected. Cross on the table means that the tool has that feature while a space means it doesn t have. As one can see the four selected have the highest number of fields. 29
CONCLUSION As one can see that the cons have outweighed the pros. Thus we chose to not use Argus. Argus is very strong software but it does not have a good interface and is not fit for our use. But the experience did not go in vain as I could use the knowledge I gained while studying about Argus and SNMP test and how it uses MIBs and oids in different software named Zabbix. I could use combine my knowledge in SNMP tests along with the expertise my group mate, Piyus Vyas, had gained in Zabbix by then, we could enhance the number of tests Zabbix could perform. At first could only perform tests such as if a device is connected or not. With the help of SNMP tests we could perform tests such as: 1. CPU Utilization 2. Memory Utilization 3. BGP Protocol Status 4. HSRP Protocol Status 5. Link Utilization 6. Packet Loss and more. These tests ensured that Zabbix had the maximum features required for our cause and thus was considered the best among all the tools. It was finally selected with other three tools namely Nagios, OpenNMS and Spiceworks. Even though Argus wasn t selected it gave us the important knowledge which helped in making the project a success in the form of Zabbix. 30
REFERENCES argus.tcp4me.com www.idrbt.ac.in www.oid-info.com www.daniweb.com www.google.com www.zabbix.com 31