Technical Guide Version 01 28/11/2014
Documentation Information File Name Document Author Document Filename Monitoring the gateways with _v1.docx Iván Blesa Monitoring the gateways with _v1.docx Issue 01 Classification Public Document Change Control Version Date Name Change Description 01 28/11/201 Iván Blesa First version Document Approval Approval Name Role Signature Date Approval Approval
Contents 1 Introduction... 4 2 Scope... 4 3 Configuration and Setup... 5 3.1 Overview... 5 3.2 Installation... 5 3.2.1 Create a Linux user account for the monitoring service... 5 3.2.2 Sudoers file configuration... 6 3.2.3 Iptables configuration(linux firewall)... 6 3.2.4 Allowing SSH access... 7 3.2.5 Verify DNS forward and reverse lookup... 7 3.2.6 Import Management Packs... 8 3.2.7 Create a monitoring resource pool... 10 3.2.8 Configure Runas accounts and profiles... 11 3.2.9 Configure Gateways to monitor... 16 4 Monitoring the Gateways... 19 5 Troubleshooting... 22 5.1 No SSH access... 22 5.2 Incorrect account privileges... 22 5.3 Incorrect management pack imported... 23 5.4 General troubleshooting... 23 6 FAQ Frequently Asked Questions... 23
1 Introduction Since the introduction of Version 4 of the Clearswift SECURE Gateways, the products run on a Red Hat Enterprise Linux 64 bit Platform. Having a standard underlying Linux distribution opens a wide range of options and flexibility to fulfil specific clients requirements. This guide explains what is required and how to monitor the Gateways from Microsoft System Center Operations Manager (). The configuration described in this guide has been validated on the below versions: SECURE Email Gateway Version 4.0.0 2 Scope This guide describes how to configure the required software on the Clearswift Gateway side and its integration with. It is not intended to provide an installation or configuration guide for.
3 Configuration and Setup 3.1 Overview Microsoft System Center Operations Manager () provides infrastructure monitoring that is flexible and offers comprehensive monitoring for datacenters and cloud, both private and public. For more information on, visit http://technet.microsoft.com/library/hh205987.aspx monitoring is based on deploying agents in systems to monitor, which collect statstics and send it back to a server. There are specific Management Packs for different operating systems, but new ones can be created to monitor custom applications or non standard stats. In the case of the Gateways, a Red Hat 6 agent is required. This guide does not provide a guide on how to create a custom management pack for the Gateways. 3.2 Installation In order to deploy the agents and monitor the platform, the following steps need to be taken: Create a Linux user account for the monitoring service Sudoers file configuration iptables configuration (Linux firewall) Allowing SSH access Verify DNS forward and reverse lookup Import Management Packs Create a monitoring resource pool Runas accounts and profiles Configure Gateways to monitor These steps are detailed in the next sub-sections. 3.2.1 Create a Linux user account for the monitoring service A new dedicated account needs to be created in the Gateways for monitoring purposes. In order to create the account, log in the Gateway through the command line console or SSH, open a terminal session, and as root run: # useradd opsmgrsvc # passwd opsmgrsvc Changing password for user opsmgrsvc. New password: Retype new password: passwd: all authentication tokens updated successfully.
3.2.2 Sudoers file configuration The newly created user must be able to run commands as root, so an entry in the sudoers file must be created. Editing the sudoers file can be done by running the visudo command. Then, at the bottom of the file, the below must be added: Defaults:opsmgrsvc!requiretty opsmgrsvc ALL=(root) NOPASSWD: ALL Please note that this configuration allow the user to run as root anything without asking for a password. If a tighter configuration is required, the below configuration can be used as explained in http://social.technet.microsoft.com/wiki/contents/articles/7375.configuringsudo-elevation-for-unix-and-linux-monitoring-with-system-center-2012-operationsmanager.aspx: #----------------------------------------------------------------------------------- #User configuration for Operations Manager agent for a user with the name: opsmgrsvc #General requirements Defaults:opsmgrsvc!requiretty #Agent maintenance (discovery, install, uninstall, upgrade, restart, cert signing) opsmgrsvc ALL=(root) NOPASSWD: /bin/sh -c cp /tmp/scx-opsmgrsvc/scx.pem /etc/opt/microsoft/scx/ssl/scx.pem; rm -rf /tmp/scx-opsmgrsvc; /opt/microsoft/scx/bin/tools/scxadmin -restart opsmgrsvc ALL=(root) NOPASSWD: /bin/sh -c sh /tmp/scx-opsmgrsvc/getosversion.sh; EC=$?; rm -rf /tmp/scx-opsmgrsvc; exit $EC opsmgrsvc ALL=(root) NOPASSWD: /bin/sh -c cat /etc/opt/microsoft/scx/ssl/scx.pem opsmgrsvc ALL=(root) NOPASSWD: /bin/sh -c rpm -e scx opsmgrsvc ALL=(root) NOPASSWD: /bin/sh -c /bin/rpm -F --force /tmp/scx-opsmgrsvc/scx-1.[0-9].[0-9]-[0-9][0-9][0-9].rhel.[0-9].x[6-8][4-6].rpm; EC=$?; cd /tmp; rm -rf /tmp/scxopsmgrsvc; exit $EC opsmgrsvc ALL=(root) NOPASSWD: /bin/sh -c /bin/rpm -U --force /tmp/scx-opsmgrsvc/scx-1.[0-9].[0-9]-[0-9][0-9][0-9].rhel.[0-9].x[6-8][4-6].rpm; EC=$?; cd /tmp; rm -rf /tmp/scxopsmgrsvc; exit $EC #Log file monitoring opsmgrsvc ALL=(root) NOPASSWD: /opt/microsoft/scx/bin/scxlogfilereader -p ###Examples #Custom shell command monitoring example replace <shell command> with the correct command string #opsmgrsvc ALL=(root) NOPASSWD: /bin/bash -c <shell command> #Daemon diagnostic and restart recovery tasks example (using cron) #opsmgrsvc ALL=(root) NOPASSWD: /bin/sh -c ps -ef grep cron grep -v grep #opsmgrsvc ALL=(root) NOPASSWD: /usr/sbin/cron & #End user configuration for Operations Manager agent #----------------------------------------------------------------------------------- 3.2.3 Iptables configuration(linux firewall) Even though Operations Manager uses SSH to transfer information and to execute scripts, it first requires connecting to the agent via port 1270. The firewall settings in the Gateways will prevent this from happening unless the firewall is configured to allow this access. It can be done by creating or editing the file /opt/cs-gateway/custom/custom.rules and adding the below line: -I INPUT 1 -p tcp -s 192.168.2.116/32 --dport 1270 -m state --state NEW,ESTABLISH -j ACCEPT
Policy must be applied through the Web UI for the rule to be configured in the firewall. 3.2.4 Allowing SSH access As the Operations Manager sever needs to access the agent through SSH, it is needed to allow access to it through the Gateway Web UI by accessing System- >System Settings->SSH Access and adding the Operations Manager s IP address to the list of SSH allowed addresses: Figure 1: SSH access settings The policy must be applied for changes to take effect. 3.2.5 Verify DNS forward and reverse lookup The Operations Manager will not allow the definition of a server to be monitored if its name o IP cannot be resolved. So the DNS configuration in the Operations Manager server must ensure that both forward and reverse DNS lookups will resolve the Gateway name and IP to be monitored: Figure 2: Adding a DNS record
3.2.6 Import Management Packs The core Unix/Linux libraries are already imported after the installation of Operations Manager 2012, but not the detailed Management Packs for each OS version. These packs are available on the installation media, in the \ManagementPacks directory. Figure 3: Available management packs For the v4 Gateways, the following Management Packs are required: Microsoft.ACS.Linux.RHEL6.mp Microsoft.ACS.Unix.Library.mp Microsoft.Linux.Library.mp Microsoft.Linux.RedHat.Library.mp Microsoft.Linux.RHEL6.mpb These MP s can be imported through the Operations Manager UI browsing to Administration->Management Packs and selecting the Import Management Packs action on the right pane:
Figure 4: Importing management packs After adding the above list from disk, the install process can start. If there are unmet dependencies, the import process will show them and the appropriate management packs will need to be added. Figure 5: Management packs installation Now these packages are made available to deploy on the managed servers.
3.2.7 Create a monitoring resource pool A resource pool is a group of Operations Manager monitoring servers dedicated to monitoring a specific type of servers, e.g. Networking, UNIX/Linux, It is common practice to create a group for each type of platform to monitor. This is done to be able to escalate properly and to have dedicated monitoring resources for servers that might generate a high number of events. This step is not mandatory and it is possible that a resource pool will already exist in a production environment to monitor such servers. In any case, the steps are described below. Resource pools can be created through the Operations Manager console under Administration -> Resource Pools. Figure 6: Create a new resource pool The New Resource Pool Wizard window opens, where a name can be defined:
Figure 7: New resource group wizard Assign the selected monitoring servers to the resource group and complete the wizard. The resource pool definition is complete. Figure 8: Resource pool definition 3.2.8 Configure Runas accounts and profiles Computers and devices could now be added to the system to be monitored. However, unless the monitoring accounts are configured in the system, Operations Manager will only report whether the system is running or not. By assigning accounts to monitoring profiles, Operations Manager will be able to monitor the status of the different components of the system.
Two types of accounts will be configured: a standard monitoring one and a privileged account. 3.2.8.1 Monitoring Account A new account can be created in Administration -> Run As Configuration -> UNIX/Linux Accounts by selecting Create Run As Account in the right pane. Figure 9: Create a Run As account As this is a non-privileged account, the opsmgrsvc details will be configured, and Do not use elevation with this account will be selected. Figure 10: Account credentials The next step in the configuration asks for the distribution security options. For this account, More secure should be selected.
Figure 11: Account distribution security The next account to configure is the elevated monitoring account. 3.2.8.2 Monitoring elevated monitoring account This account can be created in exactly the same way as the previous one, but selecting different account credential options. As the same user can be configured in both types of accounts, the display name must be configured so that it is possible to differentiate both types. In this example, the account will be named opsmgrsvc privileged. Figure 12: Privileged account creation
In the account credentials details, the option Elevate this account using sudo for privileged access must be selected. Figure 13: Account credential details As done previously, the distribution security option should be set to More secure. 3.2.8.3 Associate accounts with profiles The newly created accounts have to be associated with monitoring profiles to be used to monitor the Gateways. There are two profiles to associate the accounts with: UNIX/Linux Action Account: The opsmgrsvc account will be associated to this profile UNIX/Linux Privileged Account: The opsmgrsvc - privileged account will be associated to this profile Both are associated exactly in the same way and the only difference is the profile that is selected to do the association. Under Administration -> Run As Configuration -> Profiles, double click on the appropriate profile.
Figure 14: Associating accounts to profiles New accounts can then be added to the profile. The appropriate account must be selected based on the profile. The scope can be left to All targeted objects, as afterwards the account will be associated with the resource group created before. Figure 15: Select account to associate Once saved, the wizard allows clicking on the account name to select where to use the configured account. This is described in the final wizard page, as the account has been created as More secure, which means that it should be defined manually where to use those credentials.
Figure 16: Associate account wizard final page After clicking on the account name, it is possible to select the resource group to which this account must be distributed. In this case, the Clearswift SECURE Gateways resource group will be used. Figure 17: Assign distribution lists This process must be completed for both profiles: UNIX/Linux Action Account with the opsmgrsvc account, and UNIX/Linux Privileged Account with the opsmgrsvcprivileged account. 3.2.9 Configure Gateways to monitor The Clearswift Gateways can now be configured in Operations Manager to be monitored. Computers and devices can be added in the Operations Manager interface in Administration -> Device Management:
Figure 18: Adding the Gateways to Operations Manager The Computer and Device Management Wizard opens. After selecting UNIX/Linux computers as the device time, the appropriate criteria to discover the gateways will need to be added. The Clearswift SECURE Gateways Resource Pool should be selected to have the proper accounts utilized to monitor the Gateways: Figure 19: Discovery criteria configuration Either by IP address or hostname, define in the discovery scope as many entries as needed. Credentials will need to be configured using user name and password. Following the configuration previously done, the opsmgrsvc user will be configured
and stated that it has no privilege access, so that it uses sudo escalation to achieve it: Figure 20: Discovery criteria definition Once the criteria are saved and the discovery process run, the configured gateways must be available for selection: Figure 21: Discovered gateways
After selecting them and clicking on Manage, Operations Manager will connect to the Gateways through SSH to install the management agent: Figure 22: Management pack deployment Once this step is completed, Operations Manager starts monitoring the Gateways. 4 Monitoring the Gateways The installed management packs can review the internal status of Red Hat systems. A fast view of the status can be found under Monitoring->UNIX/Linux Computers in the Operations Manager interface: Figure 23: Monitoring the Gateways In case of finding that a gateway is in a critical status, a detailed view can be used to find out which parts of the system are failing.
Figure 24: Monitored Gateway in a Critical status Right clicking on the failed server allows having different views, which will allow doing a root cause analysis. Figure 25: Root cause analysis The Diagram View allows pinpointing the area in which the problem has been found: Figure 26: Diagram view However, this view does not allow taking actions on the problems found. The Alert view, though, allows managing the alerts found in the system.
Figure 27: Alert view of a Clearswift Gateway A number of alerts are triggered after the monitoring starts. The management packs for Red Hat monitor for a range of services that assume must always be running, including Syslog, the ACPI daemon, NFS and all the NFS related services. As the Clearswift Gateways make a customized installation of Red Hat, some of the Operations Manager default monitored services are not running in the Gateways. These monitors can be disabled for the gateways by right clicking on the alert row and selecting Overrides -> Disable the Monitor -> For the object: Figure 28: Disabling unnecessary monitors This configuration could be saved in the Red Hat management pack, but it should instead be saved in a new management pack:
Figure 29: Override monitoring in a new management pack After overriding the monitoring on a new management pack, the system reports back that the Gateways are in a correct status: 5 Troubleshooting Figure 30: New Gateways status The configuration process can fail for a number of reasons. The most common ones and how to troubleshoot them are shown in this section. 5.1 No SSH access The Operations Manager servers that monitor the Gateways must have access to them through SSH. So access must be allowed through the Gateways UI. If access has not been granted, it will be reported when trying to discover the Gateways. 5.2 Incorrect account privileges The account configured to be used by the Operations Manager server after logging through SSH must be able to sudo without being asked for a password. If the configuration hasn t been set correctly, monitoring will fail.
5.3 Incorrect management pack imported If the Red Hat 6 management pack and all its dependencies have not been installed, the agent will not be installed. The management packs downloaded from the catalogue are not always the most appropriate. It is recommended to install the ones provided on the installation media and upgrade them if necessary with the ones on the online catalogue. 5.4 General troubleshooting The Gateways will log access to SSH in the /var/log/cs-gateway/auth-yyyy-mm- DD.log file. It will also show whenever a sudo command is executed and the command it is used with. This can be used to validate that the monitoring agent is being correctly executed. Nov 28 19:25:20 cs-gw-23358 sudo: opsmgrsvc : TTY=unknown ; PWD=/var/opt/microsoft/scx/run ; USER=root ; COMMAND=/opt/microsoft/scx/bin/scxlogfilereader -p Nov 28 19:30:20 cs-gw-23358 sudo: opsmgrsvc : TTY=unknown ; PWD=/var/opt/microsoft/scx/run ; USER=root ; COMMAND=/opt/microsoft/scx/bin/scxlogfilereader -p Nov 28 19:35:20 cs-gw-23358 sudo: opsmgrsvc : TTY=unknown ; PWD=/var/opt/microsoft/scx/run ; USER=root ; COMMAND=/opt/microsoft/scx/bin/scxlogfilereader -p 6 FAQ Frequently Asked Questions Q: Is there a Management Pack available to monitor the internal status of the Clearswift Gateways? A: Clearswift does not provide a Management Pack to monitor the status of Clearswift Gateways. However, administrators can create a new one based on the standard templates provided by to monitor the status of the gateway s processes.