Best Practices on monitoring Solaris Global/Local Zones using IBM Tivoli Monitoring

Transcription

1 Best Practices on monitoring Solaris Global/Local Zones using IBM Tivoli Monitoring Document version 1.0 Gianluca Della Corte, IBM Tivoli Monitoring software engineer Antonio Sgro, IBM Tivoli Monitoring and APM resources agents architect Copyright International Business Machines Corporation All rights reserved. US Government Users Restricted Rights Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp.

2 Table of Contents 1. Overview on Oracle Solaris LDOMs, Containers and Zones ITM UNIX OS Agent and Solaris Zones Installing ITM Unix OS Agent: Global zone or Local zones? Installing in a Global zone: pros and cons Pros Cons Installing in a Local zones: pros and cons Pros Cons Conclusions... 10

3 1. Overview on Oracle Solaris LDOMs, Containers and Zones Solaris LDOMs and Containers are part of Oracle s comprehensive offering of virtualization technologies. LDOMs (or Oracle VM for SPARC) are the HW-Virtualization technology of the SPARC-T (CMT, ChipMultiThreading, Coolthread, sun4v, etc) server series, it is their ability to carve up the server into Logical DOMains, running on a hypervisor that runs in the firmware. LDOMs are a hardware virtualization, the main feature provided by LDOMs is Kernel level separation. In fact, in a LDOM solution, you could run different updates of Solaris 10 within a box and you could run Solaris 10 and Solaris 11 right next to each other within a box. Containers provide isolation between software applications or services using flexible, softwaredefined boundaries. Applications can be managed independently of each other, even while running in the same instance of the Oracle Solaris Operating System (Solaris OS). Solaris Containers create an execution environment within a single instance of the Solaris OS and provide: full resource containment and control for more predictable service levels software fault isolation to minimize fault propagation and unplanned downtime security isolation to prevent unauthorized access as well as unintentional intrusions A Solaris container is an implementation of operating system-level virtualization technology, characterized by: resource and namespace isolation a scalable architecture hardware independence possible use, in some configurations, of a single operating system, which eliminates redundant administration (in that condition the guest operating system is the same as the host operating system, but appears isolated) This implementation is in stark contrast to VMware virtualization technique, where:

4 a variety of operating systems can be mounted on the guests, and each guest functions as an independent kernel the modified guest operating systems use a special API hardware virtualization is performed in addition to OS virtualization Solaris Containers is an implementation of operating system-level virtualization technology for x86 and SPARC systems, first released publicly in February 2004 in build 51 beta of Solaris 10, and subsequently in the first full release of Solaris 10, This technology is now available in both Oracle Solaris 10 and Oracle Solaris 11. Solaris Containers are built using one or more of the following technologies: Solaris Resource Manager, for workload resource management; Resource Pools, for partitioning Zones All systems that run Oracle Solaris 10 or 11 contain a master zone, called the Global zone. The Global zone is the original Oracle Solaris OS instance. It has access to the physical hardware and can control all processes. It also has the authority to create and control new zones, called non- Global zones (Local zones), in which applications run. Solaris Zones and LDOMs are not rivaling, but complementary technologies. If an Oracle Administrator requires kernelspace separation, he should use LDOMs, if he simply requires applications that are cleanly separated from each other, he should use Zones.. A Solaris Zone is a complete execution environment for a set of software services a separate, virtual Solaris environment within a Solaris instance. A Zone provides a virtual mapping from software services to platform resources, and allows application components to be isolated from each other even though they share a single Solaris OS instance. It establishes boundaries for resource consumption and provides isolation from other Zones on the same system. The boundaries can be changed dynamically to adapt to changing processing requirements of the applications running in the Zone.

5 The Global zone encompasses the entire system and is comparable to a normal Solaris OS instance. It has access to the physical hardware and can see and control all processes. The administrator of the Global zone can control the system as a whole. The Global zone always exists, even when no other zones are configured. Within the Global zone are Local zones. These zones are isolated from the physical hardware characteristics of the machine by the virtual platform layer. This layer provides the zones with a virtual network interface, one or more file systems and a virtual console. Even though the virtual network interfaces might map to the same physical network interface, applications in different zones are prevented from seeing traffic from applications in other zones. Each zone has its own process environment and runs its own set of core Solaris OS services, including inetd, syslogd, rpcbind, and more. Applications running in a zone are unable to see applications running in other zones because of this private process environment. There are three general different types of Oracle Solaris Zones to pick from during zone creation: Small Zone (also known as a "Sparse Root Zone"): this is a partial replica of the Global zone using a loopback file system (LOFS) to access its shared libraries. It inherits packages and patches distributed through the Global zone. It shares the same operating system instance and therefore runs the same Solaris OS version. Big Zone (also known as a "Whole Root Zone"): this is a whole replica of the Global zone using its own physical copy of the system s shared libraries. It also inherits the entire package and patch database, and maintains its own physical copy of product contents. It shares the same operating system instance and therefore run the same Solaris OS version. Branded Zone: this is a different virtualization method that permits the creation of Local zones with different operating systems. o Branded Solaris 10 inside a Solaris 11 Global zone o Branded Solaris 9 inside a Solaris 10 Global zone (available only on SPARC systems)

6 o Branded Solaris 8 inside a Solaris 10 Global zone (available only on SPARC systems) o Branded Red Hat Enterprise Linux inside a Solaris 10 Global zone Large virtualized Solaris environments are often characterized by a large number of zones (small, big or branded) hosting critical server applications. Since the zones actively share resources with the base OS, any performance deficiency or resource contention at the base could adversely impact the performance of the server applications executing on the zone. It is therefore imperative to closely monitor the functioning of the virtual Solaris host and the zones executing on it from the perspective of resource usage, so that resource-intensive applications or processes can be instantly identified and corrective measures immediately initiated. Figure 1: Oracle operation stack: LDOMS for HW virtualization, Zones for SW virtualization

7 2. ITM UNIX OS Agent and Solaris Zones ITM Unix OS Agent has a dedicated attribute group and a set of out of the box workspaces to monitor Solaris Local zones from an agent installed in the Global zone. Figure 2: Solaris Zones view ITM Unix OS Agent attributes available in Solaris Zones attribute group: Name: The name of a zone. Valid entry is a simple text string, alphanumeric with a maximum length 192 characters. Zone ID: The full name of a user. Path: The path to the root directory of the zone. Init Process ID: The process ID of the init process for this zone. Status: The status of a zone. The following values are valid: Uninitialized, Ready, Booting, Running, Shutting Down, Empty, Down, Dying, Dead, Initialized. Scheduler: The scheduler used by the zone. The following values are valid: Fair Share Scheduler, Time Share, Interactive, Fixed Priority, System, and Real Time. Pool ID: The numeric pool ID this zone is bound to. Zone CPU Usage: The CPU usage of all processes in the zone. CPU Share PCT: The percent of the processor set available to this zone. CPU Shares: The weight used by the fair share scheduler to control CPU usage. Dedicated CPU: The number of CPUs that are assigned for zone exclusive use. Capped CPU: The cap configured in the zone on CPU units. Total CPUs: The number of CPUs that are in the processor set. Physical Memory: The physical memory in kilobytes used by all processes in the zone. Virtual Memory: The virtual memory in kilobytes used by all processes in the zone. Capped Memory: The cap configured in the zone on physical memory (KB).

8 ITM Unix OS Agent also includes a workspace called Solaris Zone Processes workspace that includes the Top 10 Processes by CPU Time in zone, Top 10 Processes by Virtual Size in zone, and Top 10 Processes by Memory % used in zone views: Figure 3: Top 10 Processes by CPU percent in zone Figure 4: Top 10 processes by Virtual Size in zone Figure 5: Top 10 processes by Memory percent in zone

9 3. Installing ITM Unix OS Agent: Global zone or Local zones? 3.1 Installing in a Global zone: pros and cons Using the metrics reported by ITM Unix OS Agent installed in the Global zone, Oracle Solaris OS administrators can find quick answers to the following questions: How many zones are defined on each Oracle Solaris server, what is the zone ID of each of the zones, and which is its current status? Does the base operating system have sufficient memory available to support the zones that it is hosting (Physical memory and Virtual Memory consumptions)? What is the CPU utilization of the base operating system and which of the zones is taking up excessive CPU (Zone CPU Usage)? Which are zones configuration parameters in terms of CPU Shares, dedicated CPU and CPU and Memory capping? Which application(s)/process running on each of the zones is consuming CPU, physical memory, and virtual memory resources? Pros Installing an ITM OS Agent only in the Oracle Solaris Global zone provides the following advantages: Reduced TCO: since you do not need to deploy the ITM OS agents in the Local zones, you only need to maintain and upgrade a single agent. Out-of-the-box Global view: with this approach it is possible to have a common Solaris Zones view and you don t need to create a custom logical views filled with data from each zone agent. No duplicated data into Tivoli Data Warehouse Cons Installing an ITM OS Agent only in the Oracle Solaris Global zone results in the following limitations: Missing detailed information on file systems and files of the single zones

10 More complex processes management because the agent installed in a Global Zone is able to see all running processes in all zones more zones, possibly more mess It is not possible to have details on network consumption related to each zone If applications agents are installed on the Local zones, you may not control remotely these agents through the OS Agent or control using PAS (aka Watchdog) 3.2 Installing in a Local zones: pros and cons Pros Installing an ITM OS Agent in each Local zone provides the following advantages: Oracle Administrator can have detailed information on file systems, files and network of the single zones ITM Administrator can not remotely control ITM applications agents installed on the Local zones using PAS without OS Agents installed in the same local zone ITM Administrator can create simpler situations because he does not need to create filters to skip other zones resources (like processes, filesystems, etc) Cons Installing an ITM OS agents in Local Zones results in the following limitations: Increased TCO: more agents need to be on each zone A Global machine view is not available and the ITM Administrator needs to create a custom logical view filled with data from each zone agent Filesystems usage increases, as each agent installation requires about 500MB to 1GB Historical collections could generate duplicated and useless data, increasing resource consumption on the Solaris machine (needed to collect duplicated data) and a data size increase in the Warehouse tables. 4. Conclusions We suggest that you select the installation type (Global or Local) based on your monitoring needs. To best address most monitoring needs:

11 1. Install an ITM OS Agent in the Global zone and execute on it general historical data collections and general situations, using the Solaris Zones attribute group and main general attribute groups (like CPU, Memory, Processes, Disks) 2. Install an ITM OS Agent in each critical Local zone where the OS/Application Administrator needs detailed information (deep monitoring) or where it is needed to deploy application agents (in order to use PAS and OS Agent control capabilities). When you have installed agents in both Global and Local zones, and you require disk performance situations and historical data collections, execute them on the agent running in the Global zone. And set the following parameter in the ux.ini file of agents that are running in the Local zones. (INSERT PARAMETER HERE) This parameter reduces OS Agent CPU consumption and eliminates data collections on disks that are already monitored in the Global zone: KUX_DISABLE_UNIXDPERF=TRUE