Monitoring and Tuning Your ZENworks Environment Rashmi Ranjan Mohanty Specialist, Novell mrashmiranjan@novell.com Mike Brunger Novell mbrunger@novell.com Patrick TenHoopen ZCM Admin, CHE Trinity Health tenhoopa@trinity-health.org
Agenda
Agenda Introduction What to Monitor Methods of Monitoring ZENworks Diagnostics And Probe Using Nagios to Monitor ZENworks JVisualVM and MBeans ZENworks Tuning Questions and Answers Appendix 3
Introduction
Introduction Why Monitor? Proactive Issue Resolution End-User Satisfaction Create Performance Baseline Why Tune? Scalability (# of Devices, # of events or interval i.e.: inventory refresh ) Better Performance Better Stability 5
What to Monitor
What to Monitor Servers ZENworks Services Web service URIs Database Connection Parameters Logs Collection Files Status Satellite Authentication Server last contact date Database (Covered in separate session) 7
ZENworks Services Non-running/Stopped Services Crashed Services ZENMonitor Service Is Your Friend Monitors essential services (e.g., ZENLoader and ZENServer) Primary Server Service List Windows https://www.novell.com/documentation/zenworks113/zen11_sys_servers/d ata/bb2m6eh.html Linux https://www.novell.com/documentation/zenworks113/zen11_sys_servers/d ata/bb2mp9s.html 8
Web Service URIs Certain ZCM services return status strings when they are connected to via their URI ZENworks Ping http://<device>/zenworks-ping returns "ZENworks Server Ping" Preboot Services http://<device>/zenworks-preboot returns "ZENworks Preboot Work Lookup" Agent http://<device>:7628/rurunning returns "ZENworks Agent is Running" 9
Database Connection Parameters ZENworks uses C3P0 to manage DB connection pool Maximum pool size, Used pool size, Threads waiting for connection CLOSE_WAIT connection count DB connections closed from server end, but waiting to be closed from DB side Large count indicates DB performance issue A script can be created that counts the CLOSE_WAIT states (e.g., using netstat -an) and alerts if certain thresholds are met 10
Logs Primary Server Logs server-messages.log loader-messages.log zcc.log Log Gathering zac zeninfo Scripts Log Analysis Log Parser Utility (smparse) Java application written by Craig Wilson (Novell Support) that generates graphs from loader-messages and services-messages logs 11
Collection File Status Inventory, Message, Status, Audit Check date of oldest file waiting to be processed Check count of files waiting to be processed Queue directory (Windows): inventory, message, status subdirectories in %ZENworks_Home%\work\collection Queue directory (Linux): inventory, message, status subdirectories in /opt/novell/zenworks/work/collection 12
Satellite Authentication Check authentication subsystem of Satellite servers periodically (~once a day) Use casatester program written by Susan Perrin (Novell Support) Alert if non-zero return code: -1 = Invalid response on GetAuthPolicy -2 = No auth Policy returned -3 = Password Authentication not enabled -4 = Invalid response on Authenticate -5 = Error Authenticating -6 = No token found -7 = Invalid response on GetAuthToken -8 = Error getting Auth Token -9 = Exception -999 = Syntax error in command line 13
Server Last Contact Date Check last contact date/time of servers periodically Alert if older than zone's refresh interval (e.g., 12 hours) 14
Methods of Monitoring
Monitoring Methods ZENworks Diagnostics and Probe Monitoring Systems Nagios, HP OpenView, IBM Tivoli, etc. Jvisualvm and MBeans Automated Checks Using Scripts Bash, PowerShell, Batch, Perl, Python, etc. Manual Checks 16
ZENworks Diagnostics & Probe
ZENworks Diagnostics Page Resource baseline or troubleshooting High CPU High Memory Utilization Thread Utilization Database connectivity issues 18
ZENworks Probe Probe is an open source toolkit used in monitoring Java Applications, Threads, Systems, and Connections Threads can be dumped to analyze which thread is causing a problem From the Threads tab in Probe, click Dumps, All Threads Memory or Heap dumps at regular intervals can be helpful in determining memory leaks Click on the System tab, click Dump Heap 19
Using Nagios to Monitor ZENworks
Nagios Nagios (www.nagios.org) is a monitoring and alerting system that can be used to monitor the ZCM infrastructure Nagios Core edition is available for free Business edition offers more advanced and userfriendly features for a cost 21
Nagios Configuration Files (Partial List) commands.cfg Definitions of checks services.cfg Things on hosts to monitor/check serverhosts.cfg Hosts to monitor hostgroups.cfg Logical groups of hosts (Primaries, Satellites, etc.) contacts.cfg Who to contact if an alert is generated 22
Nagios Checks and Plugins Nagios has many check commands, including: check_ping check_http check_disk check_log Community-developed plugins http://exchange.nagios.org/directory/plugins 23
Nagios Custom Checks and Plugins Can create custom check commands using scripts or programs The script/program needs to return one of the exit statuses that Nagios expects: 3 = UNKNOWN 2 = CRITICAL 1 = WARNING 0 = OK 24
Free Nagios Addons Nagios Remote Plugin Executor (NRPE) Execute checks on a remote Linux host http://exchange.nagios.org/directory/addons/monitoring- Agents/NRPE--2D-Nagios-Remote-Plugin-Executor/details NSClient++ Easily set up checks on Windows servers Also acts as an NRPE listener on the Windows server http://exchange.nagios.org/directory/addons/monitoring- Agents/NSClient%2B%2B/details 25
Example 1 ZCM Ping Uses check_http with these options: Primary Server IP: -I <Primary Server IP> URI: -u /zenworks-ping Expected Content String: -s ZENworks Server Ping Follow: -f sticky The sticky follow option lets the check find the redirected response 26
Example 1 ZCM Ping Command and Service Definitions # 'check_zcmping' command definition define command { command_name check_zcmping command_line $USER1$/check_http -I $HOSTADDRESS$ -u /zenworks-ping s 'ZENworks Server Ping' f sticky } # 'check_zcmping' service definition define service { use generic-service host_name service_description check_command } PRIMARY1 ZCM Ping check_zcmping 27
Jvisualvm and MBeans
JVisualVM Java tool to monitor running JVM Refer to best practices guide section 4.3 (Monitoring Your ZENworks Server Using Visual VM) 29
MBeans Enhanced Visual VM capabilities Reload logging configurations during runtime 30
ZENworks Tuning
ZENworks Primary Server Memory Tuning Primary Server Memory During install, ZENworks 11.3 sets ZENLoader and ZENServer Java memory Defaults to 50% Server and 25% Loader If amount of RAM changes, values need manual adjusting Can be adjusted to match role of the server as well: Configuration server: Increase Server/Decrease Loader Collection server: Increase Loader/Decrease Server Use Diagnostics page or JVisualVM to monitor memory usage Best practices guide section 4.1.2 (Heap Memory Size for ZENworks Services) has steps to configure 32
ZENworks Tuning Primary Server Manual Memory Tuning Using registry settings to set 2048 MB (0x800): For ZENLoader: [HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Apache Software Foundation\Procrun 2.0\ZENLoader\Parameters\Java] "JvmMx"=dword:00000800 For ZENServer: [HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Apache Software Foundation\Procrun 2.0\ZENServer\Parameters\Java] "JvmMx"=dword:00000800 33
ZENworks Tuning Primary Server HTTP/HTTPS Threads Config and Auth servers need more HTTPS threads Collection servers needs more HTTP threads Monitor web service requests/success using probe Monitor web services performance using ZENRequestValue logging Enable using tomcat server.xml or MBeans operation No of requests, successful, busy, time taken More busy responses needs increase in threads Best practices guide section 4.1.1 (Maximum HTTP/HTTPS tomcat threads) has details how to configure 34
ZENworks Tuning C3P0 Connection Pool Tuning Web services taking more time (max 100 secs) could be becuase of DB connection non-availability Monitor using C3P0 visual VM MBeans 35
ZENworks Tuning C3P0 Connection Pool Tuning Thread dump can indicate if lots of threads are waiting for DB connections If higher tomcat threads, need more DB connections Different pool configuration for zenserver and zenloader in same zdm.xml Have different configuration for various pools used Audit, Batch, Inventory UI, zenserver/zenloader See best practices guide section 4.1.4 (Connection Pool Tuning for ZENworks Primary Server) 36
ZENworks Tuning Primary Server ehcache Tuning ZENworks uses ehcache to cache objects in memory Database query is avoided if object is already cached Different cache for different type of objects Bundle, policies, etc. Devices, assignments, etc. Default encache sizes increased in ZENworks 11.3 Can monitor cache performance using ZENworks DMCache MBeans tab in Visual VM 37
ZENworks Tuning Primary Server ehcache Tuning Poor cache hit-to-miss ratio (<1.0) causes increased DB operation and poor performance Refer to best practices guide section 4.3.5 (Monitoring encahce) for steps to tune 38
ZENworks Tuning ZENLoader Storer Module Threads ZENworks 11.3 onwards inventory, message, status, audit, patch storer modules are multi-threaded Monitor processing speed using Nagios or custom scripts Many files lying around for long time requires tuning Configure the number of threads of each module to match the processing need of the environment Refer to best practices guide section 4.1.5 (Tuning Threads Allocated to Loader Storer Processes) for steps to configure Nagios example to monitor inventory files follows 39
Nagios Example Inventory File Count Uses NRPE, NSClient++, check_nrpe, and Check Filecount community plugin Checks the number of files waiting to be processed in the queue on a Windows Primary server 40
Nagios Example Inventory File Count Command, Service, and NSClient++ Definitions define command { command_name check_inv_files command_line $USER1$/check_nrpe -H $HOSTADDRESS$ -c check_inv_files } define service { host_name PRIMARY2 service_description Inventory Collection File Count check_command check_inv_files #<--remote check } ; From nsclient.ini on monitored Windows server [/settings/external scripts/scripts] check_inv_files=c:\windows\system32\cscript.exe //NoLogo //T:30 D:\nrpe\directory_file_count\directory_file_count.wsf d: \\novell\\zenworks\\work\\collection\\inventory 50 100 41
ZENworks Tuning Load Balancing through Locations Primary servers in a Location's Closest Server Rules should be randomized Inside the Server tab for a location, add the local satellite Create a group and add the desired primaries to the group The location response is added to the database in a random order zac zc l will show you the order on a device 42
Questions and Answers
Appendix
ZENworks LDAP User Source Tuning Define user containers properly for good performance LDAP replica for edirectory servers Nested group support for active directory Dynamic group support for edirectory Enable LDAP round robin to balance LDAP queries Increase LDAP caching value 45
Enhanced Logging Best Practices Guide Section 5 ZENRequestValue for tomcat Captures statistics about all web service requests Requesting device, service name, time taken, response status Hibernate logging Logs hibernate queries and parameters C3P0 logging Additional logging of C3P0 library internals JDBC logging using p6spy Logs actual JDBC queries executed on DB Queries vary depending on DB type 46
Corporate Headquarters 1800 South, Novell Place Provo, Utah 84606 801.861.7000 (Worldwide) 800.453.1267 (Toll-free) Join us on: www.novell.com 47
This document could include technical inaccuracies or typographical errors. Changes are periodically made to the information herein. These changes may be incorporated in new editions of this document. Novell, Inc. may make improvements in or changes to the software described in this document at any time. Copyright 2014 Novell, Inc. All rights reserved. All Novell marks referenced in this presentation are trademarks or registered trademarks of Novell, Inc. in the United States. All third-party trademarks are the property of their respective owners.