How to Troubleshoot Server Load and Processes

Transcription

1 How to Troubleshoot Server Load and Processes Greetings! This is your introductory guide to understanding and troubleshooting a Linux server. This guide will assist you to ensure server stability and efficiency. Please note that this is a crash course and will only provide adequate information for you to complete your tasks. Only through experience will you gain complete understanding. First off, what is load? Server load is technically the number of processes that are waiting to access the server s CPU. However, within a server environment not all processes are identical. Depending on the priority of the process some will instantly be processed while the lower priority processes will wait. How do we know what load values are acceptable or relatively high? The simplest way to calculate the actual load per core is to divide the total load (as displayed by w or uptime commands) divided by the number of cores available to the server. This presents the amount of processes that is being called by each individual core within the CPU. Here the load is To obtain actual load we proceed with 4.23/8 equaling 0.53 per core. Typically a load value of up to 1.0 per CPU/core is acceptable because that value suggests the CPUs or cores are being fully utilized and no processes are waiting long for the CPU to finish. However, due to the multitude of factors that contribute to load, even a value of.75 per core can still cause a server to be sluggish or unresponsive. As mentioned, there are other factors that affect the overall responsiveness of a server. Other things that can affect server performance are physical memory use, swap file utilization and disk usage. For example, if a server does not have sufficient physical memory for an operation, the request will then use a swap file (virtual memory space accessed from the server s hard drive) which places a large amount of strain on the disk to complete the process thus elevating the total load generated by that single process and slowing completion of other processes. This can lead to a snowball effect, causing additional slowness and load until the server eventually becomes unresponsive. The goal is to either stop the issues causing problems before the server becomes unresponsive or diagnose what happened after a server crash to determine what cause the problem, which is far more difficult. 1 H o s t D i m e. c o m

2 What Contributes to Server Load and How to Discern the Origin When you are logging into server that you want to investigate, where do you start? When in a Linux environment you have 3 major points to consider at the start: port connections, server processes, and hardware. Port Connections How do server processes and port connections differ? Technically, though port connections do in fact normally access processes thus spawning process threads, this will not always be reflected when viewing the process list and you must refer to the firewall or port to confirm. Port connections to the server can be made on any port that may be open within the firewall however the following ports are most common: Port Service Port Service 21 FTP 2082/2083 cpanel/ssl 25/26/465 SMTP 2086/2087 WHM/SSL 80/443 HTTP 2095/2096 Webmail/SSL 110/995 POP Remote MySQL 143/993 IMAP 8443 Plesk via SSL The ports on the right will not be displayed when checking on the active connections however are still good to have for general knowledge and practical application. When investigating port connections, we have developed more efficient means such as BASH one liners to assist in providing more organized and refined output such as below. When investigating port connections, as a general policy we do not permit a single IP address to have 30 or more concurrent connections within our shared environments as this is considered to be abusive. However, do not simply suspend or block the IP address as you must investigate what service it is connecting to, what address or domain, and if they are legitimate traffic or simply an external connection that is malicious. Server Processes Not all services require an external connection to spawn them, but there are many internal processes (especially on cpanel servers) that are present and need to be accounted for. The main services you should familiarize yourself with are Apache/PHP (web service), Exim (MTA), Dovecot (POP3/IMAP), MySQL (Databases) and these additional processes: cpsrvd (cpanel), cpbackup (automated backup script), cpanellogd (stat logging), and cpdavd (webdisk). All of these processes will be contributing to the load regularly and will have to be thoroughly investigated (and reconfigured in some cases) to ensure proper server stability and functionality. 2 H o s t D i m e. c o m

3 Apache and PHP All of our shared servers and most of the VPS/dedicated systems that you work with will be running Apache as the web server. The processes spawned by Apache and PHP can be investigated via ps aufx (lists all active processes) and httpd fullstatus (current Apache requests), but can also be investigated in a more efficient manner with further BASH/Linux knowledge. Normally, a domain experiencing a very high amount of connections can be traced back to the IP addresses connecting to the domain, and abusers can be blocked. Also, note that most Apache connections refer to a specific file within the domain and this is one way you can discover certain PHP scripts that are contributing to the current load: The file path for something such as this would be /home/cfarma2/public_html/forums/ There are multitudes of ways to find the largest contributors to load on a server, but it is often simplest just to start with top c to show the process consuming the most CPU resources at that point in time. Do not allow what could be an easy investigation and solution to evolve into a convoluted project. MySQL Queries Typically when an http request is made it will access a file or script. However, many of these requests require additional information which is stored in a MySQL database which is accessible through the MySQL service. Though it may be obvious, a MySQL query cannot be made if the service is not running. MySQL itself can be a highly memory and CPU intensive service depending on the amount of queries, table size, my.cnf configuration and many other variables that affect the functionality of MySQL. Below is the basic MySQL view that you will most commonly be looking at: The command mysqladmin proc displays all active queries with database information and the status allowing pinpointing abusive users. Server load caused by MySQL use is typically simple to locate by checking mysqladmin proc stat and viewing the MySQL server error log at /var/lib/mysql/server.hostname.com.err. This is not a verbose log of MySQL access but will provide information pertaining to the startup, shutdown and errors of the MySQL server. If there is a crashed table, which can sometimes increase load, it will also be listed here and should be repaired with mysqlcheck r database. 3 H o s t D i m e. c o m

4 cpanel/whm Services The developers of cpanel have worked to ensure that their control panel is stable and efficient as possible, but there may be times that certain cpanel initiated processes overload a server. Thankfully, cpanel also includes tools to help deal with problems when they arise. The current cpanel/whm version information and installers can be found at You can check a server s currently installed cpanel version at /usr/local/cpanel/version such as below: We keep all of our shared servers on the latest STABLE build of cpanel. We find that this offers our clients the best security and stability as this version has been thoroughly tested and verified. There are 3 other cpanel versions available as well which are RELEASE, CURRENT, and EDGE. All new installations of cpanel default to RELEASE which is similar to STABLE however with less testing. The CURRENT version of cpanel would be referred to as the release candidate for cpanel but does not have full functionality as the RELASE or STABLE versions. The final version which is not typically utilized is the EDGE release. This version has little previous testing and is normally subject to modification without official public documentation. One of the most intensive process that cpanel currently utilizes is cpbackup, the cpanel backup program. This will normally be launched via cron at 1AM on most servers but it can also be launched manually at any time by executing /scripts/cpbackup. While cpbackup is running it may be compressing, copying and overwriting files onto a backup drive or sending compressed backups to a remote server via FTP. That makes cpbackup very CPU and disk I/O intensive. cpanel includes a special tool called cpuwatch at /usr/local/cpanel/bin/cpuwatch that is designed to try to limit the amount of load a script or command can place on the server. It does this by monitoring the load on the server and pausing the command or script it is running if the load gets too high. The cpbackup script automatically calls cpuwatch with a load cutoff value of the number of CPUs as listed in /proc/cpuinfo multiplied by the number of extracpus listed in /var/cpanel/cpanel.conf (or Tweak Settings in WHM). So, for example, if /proc/cpuinfo says there are 4 CPUs/cores and the extracpus value is 2, cpbackup will be paused if the load on the server goes over 8.0 (4 cpus x 2). Generally it is best to leave extracpus set to 0 so cpbackup cannot overload the server. The other cpanel-initiated process that can greatly increase the load on the server is cpanellogd, the cpanel domain log processing daemon. It is responsible for turning the raw Apache domain logs into the stats that are used by Awstats, Webalizer and Analog. Most servers will be utilizing the Awstats and Webalizer logging software and all shared servers that are seen running Analog should have it disabled as Analog is deprecated and can be inefficient with CPU resources. Currently Spam Assassin is the spam filter that cpanel is offering which will result in the process of spamd being present on the server. Spam Assassin is constantly running and can be load intensive if it is processing a large amount of mail. This can be constricted by editing the /etc/cpspamd.conf file to limit the permitted children for the process. You will be able to locate if a server is experiencing issues from spamd processes by viewing top -c as it will frequent the top of your monitoring. 4 H o s t D i m e. c o m

5 I mentioned cpsrvd and cpdavd previously. The cpsrvd process is cpanel s custom Apache web server that handles serving both cpanel and WHM on their respective ports. It is not often a source of high load. However, if you think there may be issues, you can try restarting the cpanel service: service cpanel restart. That will shut down and restart cpsrvd and other related services that cpanel directly controls. cpdavd is the cpanel webdisk (WebDAV) software which can at times leave open connections to the server and can be located by command of ps aufx grep cpdavd providing the active connections. If the date on these processes are 24 hours or older then they are simply a hanging connection and should be closed which can be completed by /scripts/restartsrv_cpdavd instead of killing the PIDs individually. Exim & Daemons Considering that all cpanel servers utilize Exim as the MTA (Mail Transfer Agent), you will quickly become familiar with it. There are a large variety of sources that can cause the mail server to experience a high amount of activity such as mailing scripts, exploited scripts or accounts. A server with a large volume of mail in the queue can undergo very high load spikes while Exim attempts to deal with the messages. Exim handles receiving, sending and delivering messages, but it does not handle interfacing with mail clients via the IMAP or POP3 protocols and it does not handle client authentication. cpanel supports two different mail services for handling authentication and IMAP/POP3 communication, Courier and Dovecot. Courier is a slightly older mail service but remains stable at elevated load. Dovecot is a newer mail service and it has faster IMAP performance and supports push IMAP service. Most of our shared servers are utilizing Dovecot. The primary issue with Dovecot is that it is not as stable as Courier under an elevated load. Locating the origin of Exim load is normally quite simple as the load Exim generates is generally proportional to the amount of incoming and outgoing mail requests it is handling. The Exim mail queue contains messages that have not yet been processed or that could not be delivered. The current mail queue size can be checked with the command exim -bpc. A server with a large mail queue such as one of 1000 or higher is still worthy of investigation even if the server at the time is stable as this can indicate underlying issues (like spammers trying to abuse the server). Hardware Though hardware is not the most common issue, it still warrants investigation when the server is either overloading unexpectedly or experiencing unusual instability. When taking hardware into consideration you will be investigating disk I/O wait, hard drive health, RAID status, memory usage and in rare occasions CPU temperatures. All of the above can hinder the CPU s processing abilities thus causing processes to snowball and begin to overload the server. To discern that the servers load origin is in fact hardware related typically will only come after you have ruled out other issues. As previously stated, hardware issues are not common however still need to be taken into consideration when all options have been exhausted or of course that they are blatantly obvious. Such scenarios would be if the I/O wait on the disk drive is consistently above 50% or that a RAID configuration is either being rebuilt or verified, as both of these would constitute a hindrance to the overall server performance. Both of these issues can be diagnosed with basic commands that will be explained in the following section. 5 H o s t D i m e. c o m

6 How to Stabilize an Overloaded Server and Optimize Service Efficiency Throughout the rest of this guide I will provide a straightforward method to completely sweep a server and discern the origin of any issues that may be present. Further information for the commands and one liners utilized will be available in the appendix. The sar Log The first step is to get an overview of the server status which can be done with the sar log that covers average server load, memory consumption and disk activity which can be seen below: When checking sar r the most important variable here is %swpused as this is referring to the amount of swap files utilized where memory has been exhausted. If the load is elevated and swap usage is displayed then you can infer that memory use is high at the first point in the log. The memory allocated to swap will not clear the current usage unless manually cleared or all of the processes relying on that data quit or request that data again when memory use is no longer high. The command free mt will display the currently free/used memory for the server. Typing sar q shows you the load average over a period of 1/5/15 minutes allowing you to view any spikes throughout the day and provide possible correlation with other processes that may be present on the server causing the server load to be elevated. The output of sar -u specifically indicates the overall percentage of disk usage for the server and allows us to quickly discern if I/O is an issue on the server as indicated by %iowait (consistent values of 25 or more may indicate problems). If the value does exceed 25% or higher consistently then you may want to proceed to check mount options which is discussed later in the Hardware section of this guide. If no issues are located then continue with the investigation. 6 H o s t D i m e. c o m

7 Processes You should check if there are any processes running that are experiencing a large volume of connections, simply hanging (refusing to close) or are simply intensive and utilizing a large amount of resources. When viewing ps aufx the process list that is shown will provide all of this information including username, start time and process name as well as any threads spawned. The information will be displayed similar to this: In the example presented the user nobody is present for Apache processes and as users initiate process threads, they are displayed under the origin process. If there are a large amount of connections to a particular user or process then you should proceed to check the port connections as provided under Port Connections within the What Contributes to Server Load section of the guide to locate the IP address opening the connections. Intensive processes that are known to cause strain on the server such as tar, gzip or rsync (all part of the cpbackup process) should be noted when running ps aufx however you can use top to see the highest CPU using processes. The output will be similar to this: As you can see from the array of information provided, top is a powerful tool in checking the allocation of server resources as well as the complete path to the highest CPU intensive process which in this case is showing as the MySQL service. Depending on what you find, you may want to try killing or restarting an intensive process, checking the firewall for port connections or checking for domain connections. cpanel Processes As previously discussed, cpanel can cause an extensive amount of load on the server that can originate from a wide variety of services or tasks, including cpbackup, cpanellogd, or cpdavd. These processes can easily be traced with a grep command on ps aufx to ascertain if they are running and if so then the load they generate can be measured by viewing top over a short period of time. However, please note that cpbackup will be intensive in more ways than simply load and will stress the drives and the CPU so top will not be the ideal tool to measure the effects of cpbackup on the server. When investigating a server in regards to the impact of the cpbackup process, the process is reading, copying then writing to the drives which you can view with the iostat command showing real time I/O on the server. If I/O is already high prior to cpbackup running then it can be assumed that there is a large amount of processes being held. In this instance you should kill the cpbackup service. cpbackup can be killed if 7 H o s t D i m e. c o m

8 required by pkill cpbackup but this will still allow the current children processes to complete (or you will need to kill those separately. The Firewall Checking the firewall is done with the iptables or csf commands however the netstat commands query the network connections such as processes, ports, and TCP/UCDP connections which provides the output in relation to the services you request. This can either be done manually or with a provided one-liner. The one liners provided in the Appendix will provide you a list of the ports with all of the active IP addresses connected to the server for the request services. This allows the connections to be traced back to the active services so you can take action as appropriate. For example, connections to mail services on ports 25/26 (SMTP) incoming or 110/143 (POP3, IMAP, insecure) outgoing would lead you to check the /var/log/exim_mainlog (SMTP) or /var/log/maillog (IMAP/POP3). Also, all incoming connections to the firewall, if they try to make connections on a particular port, will be logged at /var/log/messages. You can use the information in these logs to track down abusers. If you want to block an abusive IP address you can use CSF or iptables. When blocking an abusive IP address it is not suggested to do it permanently as this should only be used for repeated offenders. Currently nearly all of servers are installed with ConfigServer Firewall (CSF) which allows an easy backend management of the firewall. Instead of using csf d IPaddress reason to permanently block an IP address you can safely temporarily block an IP address with csf td IPaddress reason to ensure that the user does not experience issues in the long term. All of these IP addresses and reasons are available at /etc/csf/csf.deny (permanent) and /etc/csf/csf.tempban (temporary) Apache and PHP While checking the firewall you may notice a large number of connections on port 80. These are connections to Apache and can be traced to the exact file or site with /usr/local/apache/bin/httpd fullstatus which will provide you with the IP address where the connection is coming from, but also the file location and/or domain name to investigate. When investigating any file you must ensure that it is not malicious in nature. Please note that if you have any questions in this regard then you should contact a senior technician or a member of the Security team. If the file is benign then you can check the account via /etc/httpd/domlogs/domain.tld for how much traffic they have been receiving or /usr/local/cpanel/bin/dcpumonview to obtain how many resources have been utilized on average by that particular domain. MySQL The MySQL service can be quite resource intensive. The ideal configuration of MySQL is specific to the server that it is being run on as there are many variables that are dependent on the physical configuration of the server (amount of RAM, the type of CPUs, hard drive configuration, etc.) 8 H o s t D i m e. c o m

9 To the left is the output of /etc/my.cnf (the main MySQL configuration file) on one of our shared servers. This particular server has a total 8GB of memory so the key_buffer is configured for 1/8 th of the total memory to ensure efficient functionality for MySQL to utilize and the following values allow for only particular parameters to be met. An optimized MySQL configuration can be the difference between an unstable and stable server, so please ensure that this configuration is correct, however do not make changes to this file and ensure to inform your senior technician on staff. As a general rule, if a particular MySQL user as shown in mysqladmin proc has more than 15 concurrent connections then this is considered abusive and the client should be contacted by the Abuse team. When overloaded, MySQL can become unstable and unresponsive, leading to errors or corrupted data. When checking the /usr/local/cpanel/bin/dcpumonview function you will note there is a MySQL column as well. This number should be below 2 for each cpanel user or process, ideally. The MySQL server has its own log at /var/lib/mysql/server.hostname.com.err which will show you any crashed databases that have been accessed or other critical issues with the service. When a crashed database is being queried it will decrease the overall service performance which can be resolved by checking the error log then proceeding to repair all crashed or damaged databases. Repairing MySQL databases can be done through multiple means however the simplest manner to accomplish this is mysqlcheck the database or myisamchk the corrupt table. To complete a mysqlcheck you must ensure that the MySQL service is online and run mysqlcheck r database_name. This will check and repair all tables within the specified database, however if there are a large amount of tables present this can take an extended period of time. If you are aware of the specific table which requires repair then completing a myisamchk will be much more efficient. This can be completed by cd /var/lib/mysql/database_name then executing myisamchk r table_name.myi. If a database cannot be repaired or if MySQL is down and resists attempts at being restarted, please contact your supervisor or a member of the third level team to have them try more advanced repair techniques. Exim Though you may not expect it, mail services can be the some of the most intensive processes for a server to complete. There are many ways to detect high mail traffic or volume. Running ps aufx will allow you to see the active processes for the mail services but large numbers of active processes do not always mean that there are problems. An example of Exim processes would typically appear as: 9 H o s t D i m e. c o m

10 Checking Exim for issues should start with the volume of the mail queue, which can be checked with exim - bpc providing a total count of the messages waiting to be processed. The volume of the queue should generally be less than about 300, as any higher probably means that the queue is experiencing issues processing what is present or that there has been a large influx of mail recently. If it is found that the queue is higher than 300 then you should proceed to ensure that there are not any spammers or problems with the mail server. Messages provided by exim -bp displays the time spent in the queue, message ID, sending and receiving address. This is shown such as below: 6h 9.3K 1RAhtO-0005mF-Gl <root@server.hostname.com> root@server.hostname.com If a particular domain or user is showing up frequently then there may be an issue with spam and this should be investigated by a senior technician or member of the Abuse & Security team. You can check message information from the Exim queue directly as well such as via exim -Mvh msgid. Continuing the previous example, you would execute exim -Mvh 1RAhtO-0005mF-Gl to see the header of the message. This information is important to determine if there is abuse or not. Though frozen messages are not able to be sent, they should not be removed from the queue immediately but should be investigated as they hold very important information as to their. Not all frozen messages are a problem however. Exim is set to try to attempt to resend any message that fails to be delivered from at least 3 days. Sometimes remote mail servers have temporary issues and this ensures that a real effort is made to deliver a message before Exim gives up and deletes the message. When encountering frozen messages or a large volume of mail sent by a particular script, then the offending script should be examined. Our Exim configuration includes the user/script that is sending the message within the header under the X-PHP-Script entry which can be displayed as I noted in the previous paragraph. This information provides the ability to trace the script and ensure that it is not malicious in nature. When checking for mail connections via ps aufx it should be noted that Dovecot is currently the default for all cpanel servers and Courier is present only on servers if manually configured. Dovecot provides more efficient IMAP performance as it was specifically generated for IMAP support but the main feature is cpanel supports push IMAP with Dovecot, but not Courier. There are only specific situations where servers are currently running Courier and should be left as is unless otherwise modified by a senior technician. If there are any doubts in these regards please contact a senior technician. Hardware At this point you should have exhausted all other options in regards to server processes, services, and software. If there are still ongoing server load issues then you should take a closer look at the server hardware. 10 H o s t D i m e. c o m

11 Most of the time, high load caused by hardware problems is going to be related to either the disk drives or memory as these have a direct effect on the server s ability to process information. The most common will be the disk drives which can be viewed quickly by the sar -u command for I/O wait percentage which can be a symptom of additional issues. If you find that this is running above 25% consistently then you should first examine the drive mount attributes which can be done with the mount command. Left is an example of the output with you would receive. The attribute that you should be looking for here is the noatime attribute which removes the need for the system to make a write to the drive when simply reading information. Though this is technically a software issue as it is part of the file system, not particularly hardware, this will increase performance especially on servers which with a large amount of traffic. This should be enabled for the /, /tmp and /backup partitions to ensure proper I/O balancing. This can be completed with few very simple steps. For example if we found that /tmp did not have noatime enabled then we would open /etc/fstab with our editor of choice, add noatime to its line of attributes and save the file. Please note the syntax of this file is all a single line for each partition and there are no spaces but simply commas as delimiters (i.e. noexec,nodev,nosuid,noatime). Once this is saved then you have to apply the new attribute which can be done with the mount command via mount -o remount /tmp and remember you must remount the partition or it will no longer be present for access and remounting the partition without taking the proper precautions can be problematic. In this case all processes using /tmp must be stopped prior to remounting which can be listed via lsof /tmp though if you have questions or difficulties then you should contact a senior technician prior to proceeding with the remount. The next variable that should be taken into consideration is if the server has a RAID configuration. If the server does in fact have a RAID configuration then you should check the RAID drive status and if there is any active rebuild or verification. This can be done with the 3ware /c0 show command as it will provide the following output: If there is a degraded drive it will be listed as degraded under Status which the system administrators should be notified immediately to alleviate downtime and expedite a resolution. If a drive is degraded and requires a replacement then the RAID will undergo a rebuild to integrate the new drive. Depending on the number of drives, data capacity and current data stored this process can take H o s t D i m e. c o m

12 hours or more. During this time the server load may be quite a bit higher than normal (high I/O wait). RAID verification takes place to check the integrity of the RAID configuration for stability purposes. This has the same effect as the RAID rebuild however will not take place for such an elongated period of time. There are other variables that may affect the servers load or overall stability that can be monitored however these are normally handled by the system administrators as they can only be resolved with hardware access. Some of these tools for drive integrity (i.e., smartctl) or overall system functions (i.e. sensors) can provide insight to assist in expediting a resolution however will not be covered here. This is the conclusion of the guide and as stated in the beginning, it is merely a crash course for you to have the ability to complete your tasks. Only through practical application of the information here will you obtain experience to further your personal knowledge and technical skill level. Best of luck on all of your endeavors! 12 H o s t D i m e. c o m

13 Appendix Port Connections Display all current port connections: a=$(date +%s); touch /root/$a; netstat -plan grep ":21\ :25\ :26\ :80\ :110\ :143\ :443\ :965\ :995\ :993" grep -v " " grep -v ":::" grep -v " " grep -E "[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3}" > /root/$a; printf "FTP (21):\n"; grep :21 /root/$a awk '{print $5}' cut -d: -f1 sort uniq -c sort -n tail -n5; printf "\nsmtp (25):\n"; grep :25 /root/$a awk '{print $5}' cut -d: -f1 sort uniq -c sort -n tail -n5; printf "\nsmtp (26):\n"; grep :26 /root/$a awk '{print $5}' cut -d: -f1 sort uniq -c sort -n tail -n5; printf "\nhttp (80):\n"; grep :80 /root/$a awk '{print $5}' cut -d: -f1 sort uniq -c sort -n tail -n 5; printf "\npop3 (110):\n"; grep :110 /root/$a awk '{print $5}' cut -d: -f1 sort uniq -c sort -n tail -n 5; printf "\nimap (143):\n"; grep :143 /root/$a awk '{print $5}' cut -d: -f1 sort uniq -c sort -n tail -n5; printf "\nhttps (443):\n"; grep :443 /root/$a awk '{print $5}' cut -d: -f1 sort uniq -c sort -n tail -n5; printf "\nsmtps (965):\n"; grep :965 /root/$a awk '{print $5}' cut -d: -f1 sort uniq -c sort -n tail -n5; printf "\npop3s (995):\n"; grep :995 /root/$a awk '{print $5}' cut -d: -f1 sort uniq -c sort -n tail -n5; printf "\nimaps (993):\n"; grep :993 /root/$a awk '{print $5}' cut -d: -f1 sort uniq -c sort -n tail -n5; rm -f /root/$a; Display all current port connections (modified with colors): PORTS=([80]=Apache [110]=POP3 [143]=IMAP [25]=SMTP [26]=SMTP [21]=FTP); netstat -plan > /root/stats.txt; for port in ${!PORTS[*]}; do echo "$(tput bold)${ports[$port]}($port):$(tput sgr0)"; grep $port /root/stats.txt awk {'print $5'} grep -Po "\d{1,3}(?:\.\d{1,3}){3}" sort -n -t. -k 1,1 -k 2,2 -k 3,3 -k 4,4 uniq -c sort -nk 1 grep -v " " tail -5 awk '{ if ( $1 > 35 ) { printf "\033[1;31m" } else if ( $1 > 25 ) { printf "\033[1;33m" } else { printf "\033[1;32m" } ; print " ", $1, "\033[0;39m", $2 }'; done; rm -f /root/stats.txt; Display only connections to port 80 (httpd): netstat -plan grep :80 awk '{print $5}' cut -d: -f1 sort uniq -c sort -n tail -10 Apache Connections Display all of the domains that have active connections and the count: lynx -dump -width=200 localhost/whm-server-status grep 'POST\ GET' awk '{print $12}' sort uniq -c sort n Display a formatted httpd fullstatus to view file paths as well: MySQL lynx localhost/whm-server-status --dump --width=160 Display all MySQL queries sorted by MySQL database connection: mysqladmin proc stat grep localhost awk '{ print $8 }' sort uniq -c sort -n tail -5 Display all MySQL queries sorted by MySQL user connections: mysqladmin proc stat grep localhost awk '{ print $4 }' cut -d_ -f1 sort uniq -c sort -n tail H o s t D i m e. c o m

14 Exim Sort all mail addresses in the queue by volume: exim -bp awk {'print $4'} cut -d "<" -f2 cut -d ">" -f1 sed '/^$/d' sort uniq -c sort n Remove all frozen messages from the queue: exim -bp grep frozen awk '{print $3}' xargs exim Mrm Remove all mail for the provided user from the Exim queue: exiqrep if @address.com xargs exim Mrm Hardware Display the overall RAID information on the server: 3ware /c0 show Current IO wait on the server with a 3 second delay: iostat -xn 3 Display the current smartctl status for the RAID: smartctl -a -d 3ware,0 /dev/twa0 14 H o s t D i m e. c o m