Nagios and Cloud Computing

Transcription

1 Nagios and Cloud Computing Presentation by William Leibzon Nagios Thanks for being here! Open Source System Management Conference May 10, 2012 Bolzano, Italy

2 Cloud Computing What is Cloud Computing? Virtualized systems, independent of hardware and leased to customers in what is referred to as Infrastructure as a Service Virtualization is the Core of Cloud Computing Separates Hardware from Operating System Efficient use of modern multi-core processors More Servers with Less Hardware Unused system resources can be utilized in other types of servers with different resource usage Less energy, Less Rack Space, More Efficient efficient use of resources

3 Benefits of Cloud Architecture Virtualized Systems in a Cloud Can be managed entirely remotely Can move (even live) from one hardware to another Can be shutdown, saved to disk and started again when required Can be easily cloned to have another alike system started exactly when it is needed Cloud allows to automate scaling up of infrastructure to handle peak traffic load while scaling down after to keep overall cost low This requires monitoring of all system resources!

4 Cloud Solutions and Vendors Hypervisors (Viritualization Kernels): Commercial: VMware ESX, IBM Z/VM, Microsoft VirtualPC Open-Source: Xen, KVM, OpenVZ, Quemu, VirtualBox Xen originally implemented paravirtualization, requirying Linux with modified cloud os kernel. KVM and Xen-HVM do full virtualization with Quemu and CPU virtualization extensions (Intel's VT or AMD's SVM). OpenVZ is hybrid of parvirtualizer and user-mode linux and requires both host and cloud os to be same version of Linux, sharing kernel. Virtualization and Cloud Software Suites Commercial: VMware vcloud, Microsoft Azure Open-Source: Eucalyptus, OpenStack, OpenNebula, Baracus Commercial based on Open-Source: Citrix XenServer, Oracle VM, Ubuntu Enterprise Cloud, Redhat CloudForms, Parallels Virtuozzo Cloud Infrastructure providers Amazon EC2 (Xen), Rackspace (modified Xen), Linode (Xen), Savvis (Vmware), many many more...

5 Of above I recommend OpenStack with KVM or Eucalyptus with Xen. OpenVZ provides best peformance but no true isoluation. Open-Source Cloud Software Open-Source Hypervisors used in Cloud Systems Xen - KVM - OpenVZ - Open-Source Cloud Management Software Eucalyptus - OpenStack OpenNebula - Baracus Proxmox -

6 Monitoring for the Cloud Monitoring of hardware (host OS) & hypervisor More static, hardware does not change as often Monitoring of system resources often integrated into virtualizer and info not available to cloud customer Monitoring of virtual systems Dynamic, should be able to handle addition and removal of server instances Focus on application and network performance Ideally should monitor utilization and be able to launch new server instances (auto-scaling) Monitoring system should itself be robust and handle more servers without impacting performance

7 This means cluster!!! Cloud Monitoring Architecture Horizontal Scaling Clouds can be as small as 10 servers and as as large as 10,000+. When developing architecture, you need to support its future growth. Scaling on Demand A pro-active system should handle big changes in the number of cloud instances. You may have 2 webserver instances at 6am and grow to 20 at 10pm. High Availability Good system design should be fully fault-tolerant and application as a whole should continue to function without interruption if any one server instance dies

8 Nagios Cluster Options The base nagios-core package is for stand-alone monitoring where server does all service checks. It can be extended to Nagios Cluster with : Passive Service Checks (Classic Distributed Model) Old Way - NCSA used to forward results of checks from clients to main nagios server, not robust Shared database (Central Dashboard Model) NDO-Mod and Merlin projects implement this with a combination of NEB modules, daemon & database Worker Nodes (Load Balancing of Checks) DNX and Mod-Gearman do it with combination of loaded NEB module, server daemon & client servers

9 Passive Service Checks NCSA NCSA Nagios Client Server Nagios Client Server How - One central server with all services, it does not do any checks listing them all passive - Separate client nagios servers run plugins and do checks for specific sets of hosts, each has its own subset of full nagios config - Scripts are setup that capture results from each client host and send them to central server using NSCA, it puts them into nagios command queue Advantages This will work with any nagios server, organizations have been doing it from at least 2002 Disadvantages Requires a lot of custom scripting to organize nagios configs. Not reliable if server dies. Not robust to automate cloud instances being added and deleted

10 Shared Database Who: NDO-DB and Merlin How - Multiple Peer Nagios servers, each has different config file specifying which services it would check - All servers use common database to share results of checks and status of services they are monitoring Advantages - There is no master nagios server. There is master DB server, however it is a better understood topic how to create a db cluster - Using NEB avoids slow command-queue processing Disadvantages Partioning of monitoring infrastructure among servers is still manual process. It is not easy to use this for dynamic cloud environment, however it works very well for fault-tolerance

11 DNX and Mod-Gearman Worker Nodes How - Similarly to Passive Service Checks, there is a central Nagios Server, it does not execute any plugins. - Unlike with Passive Checks, nagios does schedule checks. Thereafter NEB module takes over. - Module passes information on which plugin(s) to run to DNX server (or Gearman server for Mod-Gearman) which manages worker nodes. - Worker nodes are separate servers, each has special worker daemon running. The daemon communicates with management server and gets information (plugin command) on what to run. It then passes results back to management server and NEB module writes these results directly into nagios memory.

12 Advantages and Disadvantages of DNX and Mod-Gearman Advantages Robust: checks are automatically distributed among all cluster worker nodes (default round-robin on equal basis) Scalable: Fully achieves Horizontal Scaling of nagios checks Easy to Use in a Cloud Environment: - Existing worker node can be replicated with no special config to start it. Lets expand cluster on demand - All worker nodes are essentially the same and there is no additional re-configuration necessary to add a new node Efficient Integration with Nagios: Using NEB loaded modules achieves low-level integration with nagios, much better than NCSA and command queue Disadvantages Still relies on a single central nagios server, If central nagios server dies entire system is out

13 DNX vs Mod-Gearman Single package, no external dependencies. Includes all job cluster control components DNX Hard to maintain and test for non-linux environment Can use localcheckpattern in server configuration to direct jobs. But it is not documented Supports nagios-2.x with a patch and nagios-3.x as is Client can be extended with nagiosspecific features. Planned are: - embedded Perl, check_icmp, - check_snmp, check_nrpe Mod-Gearman is built around Gearman Project Better maintained since Gearman has many uses Enjoys benefits of wider testing on new releases Easy to configure and direct to separate queues depending on hostgroup & servicegroup Only supports nagios 3.x Mod-Gearman Supports eventhandlers and not just checks! Nagios-only features are hard to add at node level

14 Combining Checks Together Combining data collection for multiple services together is a great way to off-load Nagios. There are several approaches on how to do it: Old Way - cron jobs run plugins and submit results with NSCA Check-Multisite This allows to combine multiple checks together, output status is multiple lines, new feature of nagios 3.0 A separate collection daemon on a each server. This is like Old way but no longer using cron and instead dedicated process on server doing collection. Two such popular open-source packages: - Munin : - Collectd: (/wiki/index.php/collectd-nagios) I recommend Collectd because its faster and munin relies on NSCA to push data to nagios from each host where as Collectd can send data using its own protocol to nagios server and then nagios can check this data locally. For munin you can do without NSCA using (I haven't tried it myself though) And in general use plugins that can have multiple thresholds and do checks together instead of ones that have to be called separately for each check

15 Combining All Options Together All Nagios cluster options can be combined! DNX and Mod-Gearman offers horizontal scaling for all checks and relieaves Nagios of need to run them Merlin or ADO-DB can be used for failover and scaling of Nagios Server itself Munix or Collectd can run on cloud hosts to off-load gathering of data from hosts for standard checks and provide data to nagios together Collectd Collectd on each server

16 Ideal Fully Fault-Tolerant Nagios Cluster Architecture Nagios Web Interface Server Merlin/ADO DB Replication Merlin/ADO DB Backup Backup Nagios Web Interface Server DB Proxy heartbeat Standby DB Proxy Performance Data (RRD) Server (like NagiosGrapher) udpecho Nagios Server crossmonitor Backup Nagios Server Backup Performance Data (RRD) Server udp udp Worker Node Worker Node Worker Node Worker Node Ideally you would have each of the above as a separate cloud server, but even those with 1000s of servers may find this hard to maintain

17 Nagios Cloud Cluster with 4 hosts MAIN NAGIOS SERVER Apache PNP w/ RRD N P C D Mysql DB Merlin replication STANDBY NAGIOS SERVER Apache Mysql DB Merlin PNP w/ RRD N P C D Standby Server has all checks disabled (except cross-monitor of other nagios which should not use DNX) If main server dies, backup takes over and registers itself in dyndns server replacing primary. Nagios Daemon DNX Server crossmonitor Nagios Daemon DNX Server DNX Clients use dyndns address, they are restarted on server switch DNX Client DNX Client Note: I'm working a new nagios add-on for failover which would be a NEB module that will take care of cross-monitoring, switching on failure and syncing. This should be ready in late in 2012.

18 Configuration of a cloud host The best way to configure monitoring of cloud hosts with multiple instances is to have a template and define all services by hostgroups define host { use host_name alias address hostgroups parents contact_groups admins } wprod-server <--- Template for all Webservers w1 webserv1 <---- This is second way to search w1.dynamic.cloud1.mydomain <---- Local DNS production,loadbalanced,linux_centos5,webserv loadbalancer1,loadbalancer2 Instead of adding new host of same type when its instance starts up, I recommend adding a bunch of hosts (up to maximum you could auto-scale for the day) and disable checks by default. Checks are then enabled using nagios cmd when new server is launched and ready.

19 Auto-Scaling Event handlers can be used or custom check. Trigger based on total number of open http sockets (check_netstat, check_apache_status) from all servers Write custom script that keeps number of currently active servers in DB or local file to set name of new server. Have new server name as a parameter for launching cloud instance. Write startup scripts that use this to set hostname and register ip in local dynamic dns server. For Amazon EC2, aws utility is very useful to automate launching of new servers. Get it at Extra nagios worker node is launched similarly and this is triggered when enough servers have been launched. Can also do it based on nagios stats (check_nagios) Scale down after an hour or more of low resource usage, you can do it with a check that relies on RRD data

20 Custom check to see if new server should be started: $count=sqlexec("select COUNT(id) FROM ServerData") $sumit=sqlexec("select SUM(Connections) FROM ServerData") $lastlaunched=sqlexec("select MAX(started_on) FROM ServerData") if $sumit/$count > $threshold && ($now-$lastlatched)<600 { <figure out the name and id> launch_new_server_instance($newname) sqlexec( INSERT INTO ServerData VALUES ($newid, $newname,0,curdate()) ) enable_nagios_service_checks($newname) } Use of SQL DB for Auto-Scaling This is for illustration of logic only. Not real code. CREATE TABLE ServerData ( id bigint(10) unsigned NOT NULL, name varchar(50) unsigned default NULL, connections bigint(20) unsigned default 0, started_on date default NULL, PRIMARY KEY(id)); After you got results of server check (like event handler) you do this: UPDATE ServerData SET connections=<data from nagios check> WHERE name=<server host>

21 Additional Cloud Monitoring Tips Cloud Servers are not entirely independent, and other servers on same hardware server may effect yours System load checks may not be as useful with some virtualizers, and can show false spikes in load. Put larger emphasis on 15-minute load. For determening if system is too loaded and you need to launch another server, use number of TCP connections and time to process each request. If you control the cloud, find way to get their info. Create check that shows physical server name and gives link to hardware host monitoring graphs. Remember, you can just launch a new server Do not spend too much time investigating cause, take it out of production first, replace, and investigate later

22 Nagios Cluster Software Nagios, NDO-Utils, NCSA DNX (Distributed Nagios executor) - Mod-Gearman - Gearman - Merlin (Module for Effortless Redundancy and Loadbalancing by OP5) Check-Multisite (collect data from multiple servers) Ganglia (open-source computing cluster monitoring, can be integrated with nagios)

23 Questions?

24 More Questions? Feedback? William Leibzon My Nagios Page (mostly plugins) :