Twitter and Email Notifications of Linux Server Events

NOTIFICARME Twitter and Email Notifications of Linux Server Events Chitresh Kakwani Kapil Ratnani Nirankar Singh Ravi Kumar Kothuri Vamshi Krishna Reddy V chitresh.kakwani@iiitb.net kapil.ratnani@iiitb.net nirankar.singh@iiitb.net ravikumar.kothuri@iiitb.net vamshikrishnareddy.v@iiitb.net Technical Report IIITB-OS-2010-10 April 2010

Abstract While dealing servers across the globe, the major challenge faced is maintenance. As they will be crucial in one way or the other, they should be always up and running perfectly. For ensuring this, many resources have to be monitored and regularly maintained for their consistency which needs a lot of human effort. Server administrators need to keep an eye on the status of servers day in and day out. For making the task of monitoring easier, automatic server monitoring systems are employed. The use of automated monitoring systems is inevitable nowadays as it is impossible to manually monitor the entire set of processes running on a server, or multiple servers, and check for anomalies. We have developed an efficient, unified, and scalable monitoring infrastructure that allows system administrators to watch events that are occurring on a server. Our system sends notifications in the form of tweets whenever an event occurs on a server. Events in this context refer to impending shutdown/reboot of the server machine, activation or improper exit of certain types of processes and criticality of certain system parameters. The system watches the system for a scheduled shutdown or reboot and sends out tweets in advance so that the administrator can do certain tasks, if needed, before the system shuts down/reboots. The system monitors the processes running on the server and tweets the details of processes that crash or exit abnormally and new processes that are created. It also watches the server resources, i.e., CPU utilization, memory usage, disk I/O and sends notifications if their usage goes beyond a critical value. The monitored events have to be notified to a specific server administrator. The server administrator can respond to the events by sending a reply to the tweets of the server being monitored. The replies can contain name of scripts that are saved on the server, to be executed. The monitoring system executes those scripts as soon as it receives the reply from the administrator. Thus the administrator can respond to events on the server and can take actions on the move. Project URL: http://sourceforge.net/projects/notificarme/ 2010 Chitresh Kakwani, Kapil Ratnani, Nirankar Singh, Ravi Kumar Kothuri, Vamshi Krishna Reddy V; This material is available under the Creative Commons Attribution-Noncommercial- Share Alike License. See http://creativecommons.org/licenses/by-nc-sa/3.0/ for details.

Contents 1 INTRODUCTION 1 1.1 Problem Statement........................... 1 1.2 Server Monitoring............................. 1 1.3 Challenges Involved........................... 1 2 EXISTING SYSTEM - MONIT 2 2.1 Description................................ 2 2.2 Features and Limitations........................ 2 3 NOTIFICARME 3 3.1 Modules.................................. 3 3.1.1 Kernel Space - Monitoring Module............... 3 3.1.2 User Space - Notification Module................ 4 3.2 Approach................................. 5 3.2.1 Probing System Calls....................... 6 3.2.2 Using Kernel Datastructures.................. 6 4 IMPLEMENTATION 7 4.1 Monitoring Process Creation & Improper Exit............. 7 4.2 Monitoring Impending Shutdown and Reboot............. 7 4.3 Monitoring Criticality of System Parameters.............. 8 4.3.1 CPU Load............................ 8 4.3.2 CPU Utilization......................... 8 4.3.3 Memory Usage.......................... 9 4.3.4 Disk Usage............................ 9 4.4 Interaction with Twitter......................... 11 4.4.1 Sending.............................. 11 4.4.2 Receiving............................. 11 4.4.3 Execution of Tweets....................... 12 5 TESTING & RESULTS 13 5.1 Operating Systems............................ 13 5.2 Proxy Environment.......................... 13 6 CONCLUSION 13

1 INTRODUCTION This project aims to develop an effecient server monitoring system which can perform several monitoring tasks and notify accordingly. The actual requirements are as follows. 1.1 Problem Statement To develop a module that allows a Linux server to send a Twitter update, and/or email, upon certain server events like 1. Impending shutdown/reboot; 2. Activation or improper exit of certain types of processes; 3. Criticality of certain system parameters. The server module should include a configuration feature that allows a server administrator to set various values, and make choices. The user should be able to take a limited action upon receiving such a message. The user should be able to send small scripts or commands as e-mail or a Twitter message back to the server. The server should then authenticate and run the scripts/commands, with the results then being re-transmitted back to the user. 1.2 Server Monitoring Server monitoring serves as one of the most critical components of Server Management. To ensure system s error free productivity, it is important to constantly monitor the status of managed servers. Server monitoring is much wider than website monitoring. Monitoring a server means that the server administrators get notified if one or all of their services go down or they get notified about any abnormal behavior of the processes or any system parameters based on configured threshold values. Web server monitoring may be internal, i.e. Monitoring software sits on the machine on which the web server is running, checks its status and notifies the owner if some services go down, and external, i.e, The monitoring software resides on another machine and pings the server at a certain frequency to check the server s status. These Monitoring services automate the tedious job of monitoring constantly for server administrators and detects server problems as soon as they occur. 1.3 Challenges Involved The server administrators have to monitor the server every moment in order to increase the efficiency of their mission-critical e-business operations, and to 1

reduce their risk of failed internet transactions and loss of revenue. They may need to check the status of important processes and scheduled cron jobs. They also have to keep an eye on the system parameters and their trends. Some of the parameters have to be monitored based on some threshold values beyond which they should not be allowed to continue. In such cases, they cannot see all the processes and system parameters manually. So they needs a monitoring system which does these tasks efficiently. 2 EXISTING SYSTEM - MONIT In order to check out the available approaches, some of the existing systems were examined. One of the effecient and existing open source system, which was rated best among the available monitoring systems, is Monit. The following section gives a brief information about Monit. 2.1 Description Monit is a utility for managing and monitoring, processes, files, directories and devices on a Unix system. Monit conducts automatic maintenance and repair and can execute meaningful causal actions in error situations. Monit can start a process if it does not run, restart a process if it does not respond and stop a process if it uses too many resources. You can use Monit to monitor files, directories and file systems for changes, such as timestamp changes, checksum changes or size changes. You can also monitor remote hosts; Monit logs to syslog or to its own log file and notify you about error conditions and recovery status via customizable alert. 2.2 Features and Limitations Features: View server status in Web Browser. It can be used to monitor daemon processes. It services in active, passive and manual modes. Checks system CPU, memory and load. Alerts reports on service error and on recovery. It can monitor files, directories changes such as size changes, checksum changes. 2

Limitations: If a program does not create its own pid file, a wrapper script has to be used that creates the pid file. Possible Enhancements: Add the ability to send the alerts so that one could get SMS on cell phone. Improvement in logging facility because in some cases the log is not enough and it is impossible to tell reasons for failure. 3 NOTIFICARME The following section gives the detailed explanation of different modules present in the system and the approach used for implementing the system. The project has been divided into two distinct modules which are classified based on execution environment of the system. The modules and approach are described below: 3.1 Modules The entire system comprises of two modules each of which deals with seperate environment, i.e, kernel space and user space. The two modules are: Monitoring Module and Notification Module. The details of these modules are as follows. 3.1.1 Kernel Space - Monitoring Module Monitoring module comprises of multiple kernel modules that probe the different system calls and uses various kernel data structures in order to get the details of several server activities. Certain activities may be of interest, these activities are termed as events.events like improper exit of processes, total usage of memory, disk usage, CPU load and CPU utilization are monitored. Events corresponding to criticality of system parameters will be raised only if they cross a particular threshold value whereas for process related activities, events are raised as configured. These events are being raised in kernel space and they have to be passed to user space in order to send notifications to administrator. The communication between the kernel modules and user space notification module takes place using sysfs, an in-memory file system, in conjunction with signals as shown in Figure 1. This allows for an asynchronous mode of communication that reduces overhead of monitoring as compared to a polling approach from user space. Whenever an event is raised by a kernel module, the event is written to a file in shared memory and a real time signal is passed to the user space daemon. The signal name carries the file name with it so that the user space daemon knows which file to read in the sysfs directory of our system 3

Figure 1: Overall Architecture 3.1.2 User Space - Notification Module Notification module is a part of user space daemon, which takes care of notifying the administrator about the events that have been reported by the monitoring module. These events, as already described, are stored in the sysfs directory of our system and the corresponding signal is received by the user space daemon. Based on the signal, the appropriate file is read and the information from that file is extracted. This information is the notification message that has to be passed to the administrator. The notification module will read the message and will send that as a notification to server administrator in 2 ways. 1. Tweets on Twitter. 2. Email message. Twitter The message is sent as a tweet on twitter. This is accomplished by the usage of twitter API, where we can post the messages from a registered twitter account. So in this case, the server will have a twitter account and these notification messages are sent as tweets from that particular account. Usually the server administrator will be following this account and he will get these notifications. The advantage with tweets instead of logged messages is that tweets will give the administrator a quick rundown of the status of the server. 4

Figure 2: Notification Module Thus the user gets notified about the server event. But now the administrator may want to respond to one of the events by executing some scripts. On getting a notification that the system is about to reboot in 10 minutes or if they may want to terminate a process. They may do so by executing a pre-written script that is stored on the server. This is accomplished by the receiving segment where the administrator replies to the servers tweet in a specified format, so that it can be identified as a command to execute a pre-written script for performing a specific action. Email messages The same notification message is sent as an email to the administrator, where the message is similarly read from the sysfs directory using the corresponding signal and sending an email to the administrator informing about the event that has occurred on the server. 3.2 Approach The system needs an approach in order to be a solution for the problems to be solved. As there are existing systems, we have used a different approach of directly interacting with the kernel in order to avoid the overhead of polling. The details of our approach are discussed below: 5

3.2.1 Probing System Calls Every activity in the system will take place with the help of system calls, where each system call will do a specific operation. Kprobes is way of probing these system calls. Using kprobes we can plant specific pre and post handlers to execute our code before or after the system call is made. So by probing the required system calls we can obtain information about the events taking place on the system. 3.2.2 Using Kernel Datastructures The operating system must keep a lot of information about the current state of the system. This is done with the help of Kernel data structures. As things happen within the system these data structures must be changed to reflect the current reality. For example, a new process might be spawned when a user logged into the system. The kernel must create a data structure representing the new process. Mostly these data structures exist in physical memory and are accessible only by the kernel and its subsystems. Data structures contain data and pointers; addresses of other data structures or the addresses of routines. Features Notification on move via Twitter and traditional notification via email. File based Configuration enabling customized monitoring of server events. Notification for improper exit and creation of processes. Notification of scheduled shutdown and reboot. Notification for criticality of system parameters like total memory usage and overall usage, CPU Load/CPU Utilization, disk usage. Execution of scripts received through twitter. Reboot/shutdown the server through an authenticated tweet/email. 6

4 IMPLEMENTATION This project aims to build a system in order to provide various services by monitoring several activities of the server. Monitoring each event type follows a different procedure. The following section explains about how all these have been implemented. 4.1 Monitoring Process Creation & Improper Exit The creation and exit of processes is monitored in order to inform the administrator whenever any process exits abnormally or a process is created for which the module has been configured to send a notification. Events are raised when a process which is being monitored exits, be it a normal exit or an abnormal one. In case of an abnormal exit, the exit code is not 0. The exit code is contained in event information. Events are also raised on creation of processes. The kernel modules can be configured to monitor processes by writing the parameters in following files under /sys/notificarme/configuration/procmon/ : add_pid The pid of the running process which is to be monitored add_process_name The name of the process whose creation is to be monitored. Monitoring method: The exit of processes is being monitored by placing jprobes at the system call do_group_exit. jprobes are a wrapper for kprobes. The PID of the process exiting is matched with the PIDs which are being monitored. If any of the PIDs match an event is raised. The creation of processes is being monitored by placing jprobes at the system call do_execve. The name of the process as seen in the image name is used to identify the process. In case of processes which create a tree of processes when they start, some additional workarounds, like checking pid file have been used in user space to ensure the creation of those processes. 4.2 Monitoring Impending Shutdown and Reboot The administrator may want to get notified before some specified time when a shutdown or reboot is scheduled. So to accomplish this, cron file is monitored for changes using inotify. inotify is a file change notification system a kernel feature that allows applications to request the monitoring of a set of files against a list of events. When the event occurs, the application is notified. When the file changes, it is parsed for cron jobs for scheduled shutdown s and reboot s cron jobs. The time of scheduled reboot and shutdown is then obtained and the administrator is notified about the impending shutdown and reboot just an hour before the shutdown/reboot.. 7

4.3 Monitoring Criticality of System Parameters Various system parameters include CPU Load/ CPU Utilization, Memory usage, Disk Usage. The approach used for monitoring these parameters is explained as follows. 4.3.1 CPU Load The CPU load Average is calculated on the basis of processes in linux s run queue marked running or uninterruptible. This load may go beyond a specific value for which the administrator wants to get notified. In case CPU load average goes beyond the specified value, the administrator is notified. avenrun is a kernel data structure from which CPU load average can be obtained. The avenrun array contains three load averages i.e. the 1-minute, 5- minute and 15-minute average. The module extracts this information from the data maintained by the kernel in an array avenrun and performs some shift operations to display data in fixed point notation. The same array is used to display the information in /proc/loadavg. 4.3.2 CPU Utilization For Every process Linux kernel creates a kernel data structure known as task_struct. Some fields of this structure is as given belowstruct task_struct {... int prio, static_prio, normal_prio; unsigned int rt_priority; struct list_head run_list; const struct sched_class *sched_class; struct sched_entity se; unsigned int policy; cpumask_t cpus_allowed; unsigned int time_slice; cputime_t utime, stime;... } Utime and stime are two fields in the task_struct that hold the information about how much time Process is in user mode and system mode respectively. Our kernel modules captures CPU utilization per process. Calculations are being done on the basis of CPU utilization in user mode as well as in system mode. It is 8

done so because one can look for the process that is making more system calls. Calculations are being performed using utime and stime which are associated with each process structure. For the calculations two samples of utime and stime are taken at different time intervals and then difference of two samples is divided by the time interval of measurement. In this way the value of cpu utilization for that process is obtained. 4.3.3 Memory Usage The overall memory usage of the system is obtained from a function si_meminfo() which is exported by the kernel. This function gives the total available memory and free memory of the system. Based on samples taken over a period of time, if the memory usage is continuously crossing a threshold value, events are raised. The threshold value is provided as a configuration parameter and can be set in the file /sys/notificarme/configuration/memncpu/mem_threshold. 4.3.4 Disk Usage Monitors specified mount points of a disk. Mount point to be monitored can be specified by writing mount point name to disk_mod_cmd file in sysfs. Each mount point monitoring data appears as a directory in sysfs. For example root mount point / appears as rootdisk and a mount point /media/windows_c appears as media_windows_c Directory contains following files: blocks_percent_used - percentage of disk space used. blocks_used - total no of kilobytes used. diskname - device name of the disk. mntpointname - mount point name. percent_threshold - writable file, threshold percentage of disk space. If the mount point exceeds this value an event will be raised to notify. readpersec - instantaneous rate of disk reads in the current mount point. writepersec - instantaneous rate of disk writes in the current mount point. 9

Figure 3: Configuration Parameters Monitoring method: Disk monitoring is done as per the specified mount point. Each mount point to be monitored is probed for getting the following information: disk space left total read bytes total write bytes Disk space: The disk space information is obtained after the end of every io operation the bio_endio call is made by the block device driver. For getting the information about the I/O access, bio_endio call is probed using kprobes, we can place post handler at each of this bio_endio call. Now by the arguments passed to this function we can find out on which mount point the I/O occured. If the io 10

operation is write rather than a read operation, then it simply means that there is some change in disk space, then the updated disk space information is obtained by calling vfs_statfs function. Total read and Total write: On each call of probed bio_endio the information of bytes written or read is also exported. So if the bytes written or read are from the specified mount point, they are recorded and the rate is calculated. 4.4 Interaction with Twitter Twitter is a social networking and microblogging service that enables its users to send and read messages known as tweets. Tweets are text-based posts of up to 140 characters.the monitoring module uses the twitter to get notifications from the server. The way of interaction of this system with twitter is explained as follows: 4.4.1 Sending As events are raised, the administrator has to be notified about this. Twitter and email message are used to send notifications. In twitter, a message can be sent and it is called a tweet. This tweet is limited to 140 characters. So any notification message has to be within range of 140 characters. Initially when the message from monitoring module, it is loaded into buffer and then send it as a tweet using the Twitter API. Twitter API: We have written our Twitter API in C language using the REST API calls provided by Twitter. These calls will allow us to send tweets from our program. This requires libcurl library. 4.4.2 Receiving Whenever the administrator gets notified about certain events, he may want to take some minimum necessary action in order to prevent the server from going haywire. This can be done by receiving messages from the administrator through twitter in form of tweets. The administrator may send a tweet in a specified format to execute a predefined script. the API retrieves the replies given to the server account and then parses them to get the required message or command that has to be executed. 11

Figure 4: Tweets from the server regarding Process creation and exit, Memory Usage etc., 4.4.3 Execution of Tweets The user space daemon will be polling the twitter account of the server every 30 seconds. If there is a new tweet as a reply to server account, then it is retrieved using the get_tweets function of Twitter API. The tweet can be recognized as a reply to our server account, if it contains a string @serveraccount. This is called as mention in the terminology of twitter. So if there is a mention in the name of server account, then that particular tweet is retrieved and parsed by the parser for meaningful information i.e. the command to be executed. If the received tweet contains the correct format, then the specified script will be executed subsequently otherwise it is discarded. 12

5 TESTING & RESULTS For every system that has been developed, its testing is an important aspect of its development. As this is a critical system which deals with server monitoring, we need to test it rigourously in order to make it fail-safe. This system has been tested with various test cases. The following section gives an overview of the testing done on this system. 5.1 Operating Systems This system is intended to monitor a linux server and notify the administrator for certain special events. Thus this system should be able to work on various distributions of linux of both 32 bit and 64 bit in order to satisfy the robustness and compatibility. The project has been tested for its consistency and reliability over 32 bit and 64 bit Ubuntu(Debian) and 64bit Open Suse Linux. It has worked fine as per the expectations without any exceptions. 5.2 Proxy Environment As the servers mainly used in a large network for specific purposes. we can expect a particular proxy environment for them. At the same time, some of them may not have any proxy settings. The configuration parameters of this system provides an option to specify the proxy settings or disable them. The project has been tested to work with both authenticated proxy environment and without proxy authentication also. It has worked fine with both the cases. 6 CONCLUSION This project NOTIFICARME is developed as a system to monitor certain events on a linux server. This is an internal monitoring system where the monitoring is done from the same server. Though there are many existing systems, this system brings out a difference by the approach it takes for monitoring a server as it uses kprobes and jprobes to probe the system calls for specific events and gets information from kernel datastructures. This is a different approach as compared to the traditional polling log files and proc files which involves substantial overhead. This contains several features and notifications are sent as tweets on Twitter which is currently a popular communication platform. Notificarme will presently satisfy the basic needs of server monitoring. As an advancement of this project several new features like network monitoring and several GUI representations/ graphs of system parameters like Memory Usage, Disk Usage, CPU Utilization can be added. 13

References [1] MONIT, Unix Systems Management [Online] Available: http://mmonit.com/monit/ [Accessed: Feb 05, 2010]. [2] curl and libcurl[online] Available :http://curl.haxx.se/ [Accessed: Feb 10, 2010]. [3] Twitter API [Online] Available: http://apiwiki.twitter.com/ [Accessed: Feb 12, 2010]. [4] Inter Process Communication in Linux [Online] Available: http://people.ee.ethz.ch/~arkeller/linux/kernel_user_space_howto.html [Accessed: Feb 19, 2010]. [5] Ananth Mavinakayanahalli, Prasanna Panchamukhi, Jim Keniston; Probing the Guts of Kprobes, IBM Linux Technology Center [6] Daniel P.Bover and Marco Cesati, Understanding the Linux kernel, 3rd rd. United states of America : O reilly Media, 2006 14