Monitoring and Tuning Your ZENworks

Transcription

1 Monitoring and Tuning Your ZENworks Environment Rashmi Ranjan Mohanty Specialist, Novell Mike Brunger Novell Patrick TenHoopen ZCM Admin, CHE Trinity Health

2 Agenda

3 Agenda Introduction What to Monitor Methods of Monitoring ZENworks Diagnostics And Probe Using Nagios to Monitor ZENworks JVisualVM and MBeans ZENworks Tuning Questions and Answers Appendix 3

4 Introduction

5 Introduction Why Monitor? Proactive Issue Resolution End-User Satisfaction Create Performance Baseline Why Tune? Scalability (# of Devices, # of events or interval i.e.: inventory refresh ) Better Performance Better Stability 5

6 What to Monitor

7 What to Monitor Servers ZENworks Services Web service URIs Database Connection Parameters Logs Collection Files Status Satellite Authentication Server last contact date Database (Covered in separate session) 7

8 ZENworks Services Non-running/Stopped Services Crashed Services ZENMonitor Service Is Your Friend Monitors essential services (e.g., ZENLoader and ZENServer) Primary Server Service List Windows ata/bb2m6eh.html Linux ata/bb2mp9s.html 8

9 Web Service URIs Certain ZCM services return status strings when they are connected to via their URI ZENworks Ping returns "ZENworks Server Ping" Preboot Services returns "ZENworks Preboot Work Lookup" Agent returns "ZENworks Agent is Running" 9

10 Database Connection Parameters ZENworks uses C3P0 to manage DB connection pool Maximum pool size, Used pool size, Threads waiting for connection CLOSE_WAIT connection count DB connections closed from server end, but waiting to be closed from DB side Large count indicates DB performance issue A script can be created that counts the CLOSE_WAIT states (e.g., using netstat -an) and alerts if certain thresholds are met 10

11 Logs Primary Server Logs server-messages.log loader-messages.log zcc.log Log Gathering zac zeninfo Scripts Log Analysis Log Parser Utility (smparse) Java application written by Craig Wilson (Novell Support) that generates graphs from loader-messages and services-messages logs 11

12 Collection File Status Inventory, Message, Status, Audit Check date of oldest file waiting to be processed Check count of files waiting to be processed Queue directory (Windows): inventory, message, status subdirectories in %ZENworks_Home%\work\collection Queue directory (Linux): inventory, message, status subdirectories in /opt/novell/zenworks/work/collection 12

13 Satellite Authentication Check authentication subsystem of Satellite servers periodically (~once a day) Use casatester program written by Susan Perrin (Novell Support) Alert if non-zero return code: -1 = Invalid response on GetAuthPolicy -2 = No auth Policy returned -3 = Password Authentication not enabled -4 = Invalid response on Authenticate -5 = Error Authenticating -6 = No token found -7 = Invalid response on GetAuthToken -8 = Error getting Auth Token -9 = Exception -999 = Syntax error in command line 13

14 Server Last Contact Date Check last contact date/time of servers periodically Alert if older than zone's refresh interval (e.g., 12 hours) 14

15 Methods of Monitoring

16 Monitoring Methods ZENworks Diagnostics and Probe Monitoring Systems Nagios, HP OpenView, IBM Tivoli, etc. Jvisualvm and MBeans Automated Checks Using Scripts Bash, PowerShell, Batch, Perl, Python, etc. Manual Checks 16

17 ZENworks Diagnostics & Probe

18 ZENworks Diagnostics Page Resource baseline or troubleshooting High CPU High Memory Utilization Thread Utilization Database connectivity issues 18

19 ZENworks Probe Probe is an open source toolkit used in monitoring Java Applications, Threads, Systems, and Connections Threads can be dumped to analyze which thread is causing a problem From the Threads tab in Probe, click Dumps, All Threads Memory or Heap dumps at regular intervals can be helpful in determining memory leaks Click on the System tab, click Dump Heap 19

20 Using Nagios to Monitor ZENworks

21 Nagios Nagios ( is a monitoring and alerting system that can be used to monitor the ZCM infrastructure Nagios Core edition is available for free Business edition offers more advanced and userfriendly features for a cost 21

22 Nagios Configuration Files (Partial List) commands.cfg Definitions of checks services.cfg Things on hosts to monitor/check serverhosts.cfg Hosts to monitor hostgroups.cfg Logical groups of hosts (Primaries, Satellites, etc.) contacts.cfg Who to contact if an alert is generated 22

23 Nagios Checks and Plugins Nagios has many check commands, including: check_ping check_http check_disk check_log Community-developed plugins 23

24 Nagios Custom Checks and Plugins Can create custom check commands using scripts or programs The script/program needs to return one of the exit statuses that Nagios expects: 3 = UNKNOWN 2 = CRITICAL 1 = WARNING 0 = OK 24

25 Free Nagios Addons Nagios Remote Plugin Executor (NRPE) Execute checks on a remote Linux host Agents/NRPE--2D-Nagios-Remote-Plugin-Executor/details NSClient++ Easily set up checks on Windows servers Also acts as an NRPE listener on the Windows server Agents/NSClient%2B%2B/details 25

26 Example 1 ZCM Ping Uses check_http with these options: Primary Server IP: -I <Primary Server IP> URI: -u /zenworks-ping Expected Content String: -s ZENworks Server Ping Follow: -f sticky The sticky follow option lets the check find the redirected response 26

27 Example 1 ZCM Ping Command and Service Definitions # 'check_zcmping' command definition define command { command_name check_zcmping command_line $USER1$/check_http -I $HOSTADDRESS$ -u /zenworks-ping s 'ZENworks Server Ping' f sticky } # 'check_zcmping' service definition define service { use generic-service host_name service_description check_command } PRIMARY1 ZCM Ping check_zcmping 27

28 Jvisualvm and MBeans

29 JVisualVM Java tool to monitor running JVM Refer to best practices guide section 4.3 (Monitoring Your ZENworks Server Using Visual VM) 29

30 MBeans Enhanced Visual VM capabilities Reload logging configurations during runtime 30

31 ZENworks Tuning

32 ZENworks Primary Server Memory Tuning Primary Server Memory During install, ZENworks 11.3 sets ZENLoader and ZENServer Java memory Defaults to 50% Server and 25% Loader If amount of RAM changes, values need manual adjusting Can be adjusted to match role of the server as well: Configuration server: Increase Server/Decrease Loader Collection server: Increase Loader/Decrease Server Use Diagnostics page or JVisualVM to monitor memory usage Best practices guide section (Heap Memory Size for ZENworks Services) has steps to configure 32

33 ZENworks Tuning Primary Server Manual Memory Tuning Using registry settings to set 2048 MB (0x800): For ZENLoader: [HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Apache Software Foundation\Procrun 2.0\ZENLoader\Parameters\Java] "JvmMx"=dword: For ZENServer: [HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Apache Software Foundation\Procrun 2.0\ZENServer\Parameters\Java] "JvmMx"=dword:

34 ZENworks Tuning Primary Server HTTP/HTTPS Threads Config and Auth servers need more HTTPS threads Collection servers needs more HTTP threads Monitor web service requests/success using probe Monitor web services performance using ZENRequestValue logging Enable using tomcat server.xml or MBeans operation No of requests, successful, busy, time taken More busy responses needs increase in threads Best practices guide section (Maximum HTTP/HTTPS tomcat threads) has details how to configure 34

35 ZENworks Tuning C3P0 Connection Pool Tuning Web services taking more time (max 100 secs) could be becuase of DB connection non-availability Monitor using C3P0 visual VM MBeans 35

36 ZENworks Tuning C3P0 Connection Pool Tuning Thread dump can indicate if lots of threads are waiting for DB connections If higher tomcat threads, need more DB connections Different pool configuration for zenserver and zenloader in same zdm.xml Have different configuration for various pools used Audit, Batch, Inventory UI, zenserver/zenloader See best practices guide section (Connection Pool Tuning for ZENworks Primary Server) 36

37 ZENworks Tuning Primary Server ehcache Tuning ZENworks uses ehcache to cache objects in memory Database query is avoided if object is already cached Different cache for different type of objects Bundle, policies, etc. Devices, assignments, etc. Default encache sizes increased in ZENworks 11.3 Can monitor cache performance using ZENworks DMCache MBeans tab in Visual VM 37

38 ZENworks Tuning Primary Server ehcache Tuning Poor cache hit-to-miss ratio (<1.0) causes increased DB operation and poor performance Refer to best practices guide section (Monitoring encahce) for steps to tune 38

39 ZENworks Tuning ZENLoader Storer Module Threads ZENworks 11.3 onwards inventory, message, status, audit, patch storer modules are multi-threaded Monitor processing speed using Nagios or custom scripts Many files lying around for long time requires tuning Configure the number of threads of each module to match the processing need of the environment Refer to best practices guide section (Tuning Threads Allocated to Loader Storer Processes) for steps to configure Nagios example to monitor inventory files follows 39

40 Nagios Example Inventory File Count Uses NRPE, NSClient++, check_nrpe, and Check Filecount community plugin Checks the number of files waiting to be processed in the queue on a Windows Primary server 40

41 Nagios Example Inventory File Count Command, Service, and NSClient++ Definitions define command { command_name check_inv_files command_line $USER1$/check_nrpe -H $HOSTADDRESS$ -c check_inv_files } define service { host_name PRIMARY2 service_description Inventory Collection File Count check_command check_inv_files #<--remote check } ; From nsclient.ini on monitored Windows server [/settings/external scripts/scripts] check_inv_files=c:\windows\system32\cscript.exe //NoLogo //T:30 D:\nrpe\directory_file_count\directory_file_count.wsf d: \\novell\\zenworks\\work\\collection\\inventory

42 ZENworks Tuning Load Balancing through Locations Primary servers in a Location's Closest Server Rules should be randomized Inside the Server tab for a location, add the local satellite Create a group and add the desired primaries to the group The location response is added to the database in a random order zac zc l will show you the order on a device 42

43 Questions and Answers

44 Appendix

45 ZENworks LDAP User Source Tuning Define user containers properly for good performance LDAP replica for edirectory servers Nested group support for active directory Dynamic group support for edirectory Enable LDAP round robin to balance LDAP queries Increase LDAP caching value 45

46 Enhanced Logging Best Practices Guide Section 5 ZENRequestValue for tomcat Captures statistics about all web service requests Requesting device, service name, time taken, response status Hibernate logging Logs hibernate queries and parameters C3P0 logging Additional logging of C3P0 library internals JDBC logging using p6spy Logs actual JDBC queries executed on DB Queries vary depending on DB type 46

47 Corporate Headquarters 1800 South, Novell Place Provo, Utah (Worldwide) (Toll-free) Join us on: 47

48 This document could include technical inaccuracies or typographical errors. Changes are periodically made to the information herein. These changes may be incorporated in new editions of this document. Novell, Inc. may make improvements in or changes to the software described in this document at any time. Copyright 2014 Novell, Inc. All rights reserved. All Novell marks referenced in this presentation are trademarks or registered trademarks of Novell, Inc. in the United States. All third-party trademarks are the property of their respective owners.