HP OO Community Guides Monitoring HP OO 10 This document describes the specifications of components we want to monitor, and the means to monitor them, in order to achieve effective monitoring of HP Operations Orchestration 10.0x. Overview This document includes components to monitor, technical monitoring options, and some demonstrations of usage of monitoring tools. It is focused on monitoring Central and RAS through all the relevant layers. This includes using OO REST API calls to monitor the application level, monitoring the JVM performance and monitoring the infrastructure level components such as DB, Services, CPU, memory, etc. HP Operations Orchestration HP Operations Orchestration (HP OO) is a next generation IT Process Automation solution that is designed from the ground up to increase automation adoption whether in a traditional data center or hybrid cloud environment. OO 10 Community This guide is created for the OO community. To get started with HP OO 10 check out our community onboarding kit that contains three persona-based learning tracks for administrators, flow authors, and end users. Each track provides knowledge assets of the following types: Videos Tutorials Quick Guides Presentations This document does not contain recommended thresholds, tuning and sizing guidelines. Available Tools In order to ease the process of building a monitoring infrastructure for OO, you can utilize the following tools. Java Melody: Java Melody is implemented in the OO web application. HP SiteScope (SiS): We chose to demo some of the monitors utilizing HP SiteScope as the monitoring application. HP OO Flows and Operations: Our base content pack contains several operations which can be easily used in order to build your own health check flows. For example: JVM Low Memory Check JVM Memory Usage Statistics Http Client Get LDAP Search Service Status Some of these tools such as Java Melody, SiteScope generic monitors and OO flows, are available Out-Of-The-Box.
Monitoring Layers The figure above illustrates relevant monitors and available tools for monitoring Central and RAS. Central OO Application Layer: In this layer we can use the OO REST API, as well as logs in order to get feedback in the functional level. Java Application Server and JVM Layer: In this layer we can monitor the JVM using JMX, logs and OO operations. We can also get valuable information regarding the Java Application status and performance using Java Melody (See Appendix 2). Infrastructure Layer: Includes OS, services, database, etc. This layer can be monitored via standard monitoring as well as using Out-Of-The-Box OO Flows. RAS RAS is a java based application; it is not an application server (thus it cannot contain Java Melody). And as a remote agent it is of course much thinner then Central. When monitoring RAS, we should focus on its connectivity to Central, and on the JVM and OS related metrics. The next sections describe the detailed monitoring for each layer in a top-bottom fashion.
OO Application Layer Application layer monitoring data resides in the database and logs. It can be retrieved by using the following: REST API Logs Central REST API Metric REST Syntax Remarks RAS Servers Status Get /workers Example JSON result: [{ "uuid":"16fcc0d1-17e2-48d6-8d60-4abcf4fd3571", "installpath":"c:/program Files/Hewlett-Packard/HP Operations Orchestration/central", "os":"windows Server 2008 R2", "jvm":"1.7.0_40", "description":"16fcc0d1-17e2-48d6-8d60-4abcf4fd3571", "dotnetversion":"4.x", "hostname":"myhost.mydomain.corp", "groups":[ "RAS_Operator_Path" ], "status":"running", "active":true LDAP Communication Status }] Request: POST/authns/ldap-config/{id}/test Description: Test an existing LDAP configuration. Request: POST/authns/ldap-config/test Description: Test an ad hoc LDAP configuration. The body of this request must contain a JSON object with the search details. Example: {"type":"open_dj", "domain":"catlab", "addresses":[{"host":"myhost.mydomain.corp","port":389}], "securedchannel":false, "enabled":true, "privilegeduser":"cn=ooadmin,ou=services,dc=catlab,dc=corp", "privilegeduserpassword":"********", "groupsdns":["ou=oo,ou=applications,dc=catlab,dc=corp"], "groupssearchrecursive":true, "groupsfilter":"(&(objectclass=groupofnames)(member={0}))", "groupnameattribute":"cn", "usersfilter":"(&(uid={0})( (objectclass=inetorgperson)(objectclass=organizationalperson) (objectclass=person)))", "usersdns":["ou=users,dc=catlab,dc=corp"], "userssearchrecursive":true, "usercommonnameattribute":"cn", "useremailattribute":null, "groupmembershipattribute":null, "username":"admin", "userpassword":"admin", "customgroupsattributesnames":null} Several REST options (any operations which perform an LDAP search) can be used for this purpose. This specific example tests the LDAP configuration and returns the attributes of the user you have tested with (The same as clicking the test button in the UI). Utilizing the REST API has the advantage of an end-to-end check from the OO application to the LDAP Server. Central Logs Metric Log Remarks\Examples Failed Executions Execution.log 2013-11-10 23:11:00,713 [WorkerExecutionThread-2_142895324] (ExecutionServiceImpl.java:498) ERROR - Error occurred during operation execution. Execution id: 142895324
com.hp.oo.internal.sdk.execution.flowexecutionexception: Failed to execute content action: com.hp.oo:oo-base-legacy-plugin:1.0.135-snapshot -> com.iconclude.content.actions.cmd.ping.localping In general it is recommended to capture ERROR and FATAL statements in the log file. Java Application Server and Java Virtual Machine Layer Application Server level monitoring can be done by using the following tools: For Central Servers JMX (for the JVM - in case you can/will not use JMX, please view Appendix 1) Java Melody (please view Appendix 2) OO Operations OO Logs For RAS Servers JMX (for the JVM - in case you can/will not use JMX, please view Appendix 1) OO Operations RAS Log The quickest way to achieve Java Application Monitoring will be to use the Out-Of-The-Box Site Scope JMX monitor for both Central and RAS. Using JMX In the following screenshot, you can see an Out-of-the-Box SiS JMX performance monitor, showing data of an OO 10.0x standalone RAS. The following table describes the java metrics: Counter java.lang/garbagecollector/ps MarkSweep/CollectionCount java.lang/garbagecollector/ps MarkSweep/CollectionTime java.lang/garbagecollector/ps Scavenge/CollectionCount java.lang/garbagecollector/ps Scavenge/CollectionTime Description Number of Major garbage collection since the start of the JVM and the cumulative CPU time these GC took Indicates the amount of minor garbage collections - meaning garbage collection from the young/eden and survivor spaces in the heap Cumulative amount of CPU time minor garbage collection
Counter java.lang/garbagecollector/ps Scavenge/LastGcInfo/duration java.lang/garbagecollector/ps MarkSweep/LastGcInfo/duration java.lang/memory/heapmemoryusage/used java.lang/memory/nonheapmemoryusage/used java.lang/threading/threadcount OO/WorkerManagerMBean/workerManagerMBean/InBufferSize OO/WorkerManagerMBean/workerManagerMBean/OutBufferSize Description Duration of the last minor GC Duration of the last major GC, long periods of time may indicate a memory problem Amount of memory used, if the amount of memory used is close to the maximum available memory to the heap this can indicate excessive memory usage PERM memory, used mainly to store loaded classes Amount of threads in the JVM Current size of the inbuffer which holds the steps the worker is supposed to run Current size of the outbuffer, which holds the steps that need to be persisted to the DB. Note JMX offers 2 optional methods for authentication and authorization against the JMX server: 1. Using text files, where the password is not encrypted. The files are protected using standard file system security. 2. Using LDAP. In order to secure JMX, it is highly recommended to configure it with LDAP Authentication and enable SSL. Using the OO operations There are 2 very useful OO operations for the JVM health: JVM Low Memory Check: This operation retrieves the JVM memory pool info, including the maximum memory, used memory, memory pools, and returns a response if there is a low memory issue or memory is within thresholds. JVM Memory Usage Statistics: A detailed description of these operations is available in the operation s description tab. OO Logs The OO Log files are located under <OO installation Folder>\central\var\logs and it is recommended to monitor for ERROR and FATAL events. The following log files are available for Central (while only the wrapper and execution logs are available for RAS): Metric Log Remarks Failed HTTP requests to Central Access The access log contains every request sent to the OO by HTTP/HTTPS the server Log JVM and wrapper issues Wrapper.log The wrapper log contains messages such as startup/shutdown of the server, memory issues in the server etc. Execution errors Execution.log All logged messages General.log Infrastructure Layer OS It is advised to use standard OS monitoring:
CPU Utilization Disk Space Memory Usage Services HP Operations Orchestration Central service status HP Operations Orchestration RAS service status Both OS and Services can be monitored in several ways: SiS Monitors. OO flows, utilizing the following operations: Get CPU Usage. Get Available Memory in Megabytes Service Status Windows File System, Network, Memory diagnostics and system health check OOB flows. Any standard monitoring tool. DB The OO application rolls data in tables and the growth should be relatively consistent. We recommend checking the DB size growth consistency. A good threshold will be an alert if the DB size is doubled over the last day. In addition, we advise to use standard DB monitoring: Available free space for the OO schema DB size Database error logs Swap file usage HA (In case you use High availability ) Monitor backup validity Note: Indications regarding the DB performance are also available via Java Melody. DB monitoring recommendations and monitoring relevant SQL commands and tools are vendor specific. Please also consult your DBA regarding the server s CPU, system IOPS ( I/O per Second), memory usage, DB connections, open DB files, temp usage and more. SiS has Out-Of-The-Box monitors for Oracle and SQL Server. For other Databases you can use the custom DB monitor or other solutions. LDAP Verify that there is access from the Central Server to the relevant LDAP (Default ports are 389 for simple bind, 636 for LDAP over SSL). Verify that the OO application User logins successfully to the LDAP server. Note: This can also be done in the application layer. And that s the end of guide. If you have any questions then please post them on the OO community forums: https://hpln.hp.com/node/21/og/forum/37
Appendix 1: Logging the JVM Memory Usage Information Some organizations chose not to enable the JMX debug console for Java based applications due to e.g. security concerns. In this case you can log Java memory usage data by turning on several java-logging options in the Java Additional Parameters section of the ras-wrapper.conf file. The following example outputs garbage collection information to a separated log: wrapper.java.additional.22="-xloggc:%ras_home%/var/logs/jvm.log" wrapper.java.additional.23="-xx:-usegclogfilerotation" wrapper.java.additional.25="-xx:gclogfilesize=10240k" These settings will produce the following information: [GC 138232K->17428K(494976K), 0.0038747 secs] The above line indicates: [GC/Full GC <amount of memory in young/old generation before the GC> -> <amount of memory in young/old generation after the GC>(Committed size of the heap in Kb), <duration of the GC in seconds>] This data can indicate memory issues if we see the following: Similar or close Amount of memory before and after a Full GC can indicate insufficient heap or a memory leak over time Long periods of garbage collections (more than a few seconds) can indicate insufficient memory or a memory leak and can also indicate and over loaded system There many online resources that can provide addition information and suggestions for logging JVM metrics to a file.
Appendix 2: Using Java Melody Java Melody is a Java Application Server monitoring tool which is implemented in the OO Central Application. Java Melody displays statistics regarding memory usage, CPU usage, http sessions and the database. To access the JavaMelody statistics, please go to: http://<centralserver>:<port>/oo/monitoring Metric Details Remarks Used Memory Memory utilization metrics % CPU CPU Usage. Active jdbc Currently working JDBC Provides metrics regarding DB connections. connections connections Used jdbc Currently opened JDBC Provides information regarding DB connections pooling connections connections http hits per Amount of http requests to Provides information regarding web server load. minute the server per minute HTTP Mean Times Sql hits per minute SQL Mean Times Average times for the HTTP requests sent to the server (note that these requests can be drilled down to in statistics http section Provides information regarding DB server load. Average times for the SQL requests generated by the server Amount of SQL requests generated by the server Java Melody has a REST API, in case you want to query it with your own monitoring tool. The results can be returned in JSON or XML format. The API documentation is available here: https://code.google.com/p/javamelody/wiki/externalapi