Moving beyond hardware

Transcription

1 Moving beyond hardware

2 These slides represent the work and opinions of the author and do not constitute official positions of any organization sponsoring the author s work This material has not been peer reviewed and is presented here with the permission of the author. The author assumes no liability for any content or opinion expressed in this presentation and or use of content herein.

3 Michael Medin I work with Integration Not system administration Currently at Oracle Corporation But do not worry! I am NOT here to sell you anything I am also A developer An avid fan of open source The author of NSClient++ Trying to convince everyone monitoring is good!

4 Introduction Integration Applications Monitoring and Application Servers JMX JMX Tools Nagios perspective Zenoss perspective Using JMX Exploring JMX Monitoring JMX Extending JMX

5 Who monitors their applications today? Who uses JMX Who uses WMI Who uses SNMP Who uses custom built things What else do you use?

6 What do you monitor? Application Errors? Memory? System Memory? Application Memory? Component memory? Threads? Database pools? Internal Errors? Messages? Non events? Performance? System Performance? Database Performance? Application Performance? Component Performance?

7 My field of expertiese

8

9 In this case: FTP Web Services Databases Application servers IPC But often: Files Queues EDIFact EBXML messages Network VPN External components etc Metrics: Disks System Load Memory utilisation Database Utilization Performance Processes Business Process Messages

10 How would you approach something like this? A lot of custom scripts? Use the supplied GUI? Some basic standard and custom things? Maybe simpler to skip monitoring?

11 Nagios (op5) Grid Control Scripts Files Databases (oracle) Applications FTP Queues Web services Disks Database Utilization Database System Load Performance Application servers Memory utilization BPEL Console BAM (Business Application Monitor) B2B Home Brew Processes Services (some) Business Process EDIFact EBXML RosettaNet messages etc

12 The Tech Support Team Nagios Scripts The DBA Grid Control Scripts The ICC Operations Team Home brew (usually part of the application) BPEL Console BAM B2B

13 Many systems means; Many places to look Many people to ask NOT many people who knows it is broken (Often no one at all) Hard to; Correlate data Be proactive Find performance bottlenecks Find missing files Debug system wide applications

14

15 We spend millions of euro to build one integrated system for business messages. Why not save money using one integrated system for monitoring?

16 In other words, the boring stuff

17

18

19 Still not convinced?

20 A net work administrator changes a firewall rule Preformed during the service window The service windows is 23:00 -> 02:00 (Friday) The network admin Checks the monitoring and checklists Everything is green! Goes home happy (grabs a beer on the way home) The integration staff is not on call (only 8-17) An integrations service breaks (as a database pool gets corrupted) No one sees this (no one is checking the BPEL Console ) This is discovered on Monday at 09:00 Why?

21 What can you monitor?

22 Application memory Make sure the application has available memory (not the same as the system memory usage) Threads Make sure we have active threads But not too many Sessions Make sure the application is used But not to much Response Time How long are requests taking HTTP, EJB, Database, etc etc

23 Database pool Make sure the application can access the database Make sure the database is being used Transactions How long did transactions take How many transactions are committed How many are rolled back Queues How many messages are waiting (are the queues filling up) And this is just a few examples Your applications needs will be different

24 What is an application?

25 It depends on who you ask 20 years ago I would have said: This assembler thing I wrote 10 years ago I would have said: This C++ thing we wrote. Now I usually answer this J2EE system This means Java (and.net) And they usually run in an application server Java Inside a Java J2EE Application server JBoss, WLS, WebSphere, OC4J, Orion, etc etc....net Inside.Net framework Microsoft.Net Framework

26

27 Main tasks are to: Manage applications Provide resources Deliver utility functions Manage means to manage; State Resources Configuration Connections (database etc) Pools (thread, memory, database, etc) Caches etc Wait Is this not what we want to monitor?

28

29 JDBC Database connections (and pools) JMS Queues (and such) JTA Transactions <Random acronym here> Whatever buzzword is popular today And the best things is: All this is provided and managed by the application server

30 Out of the box we can monitor: Database connections Database pools Queues Transactions Etc etc etc etc etc etc (yes, there is a lot more) Almost all resources are managed! So we can out of the box monitor most things Now if only there was a way to monitor this

31 The default and standard way is: JMX Other options; SNMP Web Services Scripts Various proprietary solutions (often provided by the Application Server) Custom Written things But JMX is the standard way in Java And it works everywhere!

32 .Net provides the same thing WMI (Windows Management Instrumentation) (but not as extensive)

33 Java Managment Extensions

34 Java Management Extensions API for management communication Think of it as snmp or Wmi but for Java

35 JMX Consists of: Agent A repository of MBeans Client The user (in your case check_jmx) Mbeans Providers (in your case server components) Which can be monitored

36 Managed Bean Accessed through the Agent You can: Probe information Change parameters Execute administrative functions Subscribe to events There is a lot more But this is all you really need to know

37 Attributes Set Write attributes Get Read attributes Operations Execute Run commands Notifications Subscribe Subscribe to events Nagios Check commands Read attributes Event handlers Execute operations Write attributes Passive Checking (NSCA) Notifications

38 A big list of objects. Will change live as the application runs BUT not always the same repository For instance: WLS uses their own

39 A managed object Has metadata (descriptions) Identified by: A name Usually key/value pairs For instance: Name=Something, Type=Something, etc etc A part of your application (server) Delivering live metrics

40 The JMX specification does NOT support remote checking! Usually this is circumvented by using RMI The sad word here is Usually! Remember There are other options then JMX For instance SNMP agent

41 Ho to monitor Monitoring

42 Nagios is script driven Means you have a check_xxx for everything check_cpu check_mem check_nrpe check_... Application Server Exposes Jmx (Java) WMI (.Net) So you need a check_wmi and a check_jmx!

43 Generic JMX Plugin Jmxquery/Syabru/* (check_jmx) check_jmx4perl JBoss Plugin JBossNagiosPlugin Monju Others (non JMX) JSend NSCA

44 Written in python/java Remote or local JMX connection Forks java (once per check) Only > check Does not support custom protocols Without modification

45 Notes: Always use vvvv when testing Or no errors will be displayed Command syntax: check_jmx -U <URL> -O <Object> -A <Attribute> -w <warning> -c <critical> Sample command (from -help): check_jmx -U service:jmx:rmi:///jndi/rmi://localhost:8888/jmxrmi -O java.lang:type=memory -A NonHeapMemoryUsage -K used -w c

46 Command: jbossjmx_plugin Written in Perl Only local checks (ie. NRPE/SSH) Forks Java (once per check) Only > checks Presumably works with other servers but requires JBoss

47 Notes: Version is only to find Jboss Command syntax: jbossjmx_plugin <JBoss Server URL> <JBoss Version 3 4> <JBoss MBean Object Name> <JBoss MBean Attribute> <Warn Threshhold> <Critical Threshhold> Sample command: jbossjmx_plugin jboss.jca:name=oracledsn,service=managedconnectionpool InUseConnectionCount 20 50

48 Written in Python (check) and Java (servlet) Based on a servlet (agent) No forking (requires deploy on a webserver) Only > checks Pretty hard to write queries Presumably works with non Jboss servers

49 Notes: Very complex and hard to use Command syntax:./check_jboss.sh 'nagios/jmx? obj_name=<object name>&attribute=$<attribute name>&is_int=<integer>&warn=<warn value>&crit=<crit value>' Sample Command: check_jboss.sh nagios/jmx?obj_name=jboss.system:type=serverinfo &attribute=freememory&is_int=true&warn= &crit=

50 Written in Perl (check) and Java (servlet) Based on a servlet (agent) No forking (requires deploy on a webserver)

51 Notes: URL is NOT! JMX Url (it is java component) Command Syntax: check_jmx4perl --url <j4p URL> --name <alias> --mbean <Mbean> --attribute <atribute> --path <part> --critical <crit>--warning <warn> Sample Command: check_jmx4perl --url --name memory_used --mbean java.lang:type=memory --attribute HeapMemoryUsage --path used --critical warning

52 Jmx4perl The good: Powerfull syntax Nagios standard No forking The Bad: Requires a custom component on the server No security model check_jmx The good: Java Standard Standard security model The bad: Very basic usage Requires Java fork on check

53 Check_perfect_jmx Local Java agent (one process) With a simple socket/nrpe/* interface Which connects to Java agents via RMI/* And checks remotly

54 The Zenoss perspective

55 First off I have only looked into this briefly! Never managed to figure out how Zenoss works So I could be wrong Relays on Java Runs in the background (as a demon locally) One demon per target server Does not support events

56 Finding things to monitor

57 JConsole Comes with Java (1.5 and later) Can attach to local processes Without any configuration Can attach to remote processes Given a connection string Third party browsers Some Application Servers have them builtin

58

59

60 default repository (18) WLS repository (4500)

61 Mbean name: com.bea:name=soa_server1,type=serverlifecycleruntime Breaking it down we get com.bea Namespace Name=soa_server1 Property (name) Type=ServerLifeCycleRuntime Property (type) This looks like a query but it is not a query! com.bea:type=serverlifecycleruntime (does not work)

62 A quick demo

63 Some useful things to monitor

64 Heap space (what usually fills up) java.lang:type=memory getheapmemoryusage() Memory Pools java.lang:type=memorypool,name=pool's name Eden Space (Heap Memory) Pool from which memory is initially allocated for most objects Survivor Space (Heap Memory) Pool containing objects that have survived GC of eden space. Tenured Generation (Heap Memory) Pool containing objects that have existed for some time in the survivor space. Permanent Generation (Non-Heap) Holds all the reflective data of the virtual machine itself, such as class and method objects. With JVMs that use class data sharing, this generation is divided into read-only and read-write areas.

65 Thread Pools java.lang:type=threading ThreadCount Number of current threads PeakThreadCount Maximum thread count

66 Will be different depending on your application server as will most other things unfortunately

67 What your applications relay on is nothing I know you need to talk to your application developers! But the why to do it is JMX! (or WMI for windows )

68 Monitoring YOUR applications

69 Writing custom Mbeans is very simple Always better then a home brew solution Ask your developers

70 Simple example: public interface HelloMBean { public String getname(); } public class Hello implements HelloMBean { public String getname() { return Hello World ; } } class Sample { public static void main(string[] args) { MBeanServer mbs = ManagementFactory.getPlatformMBeanServer(); ObjectName name = new ObjectName("com.example:type=Hello"); mbs.registermbean(new Hello(), name); } }

71 Wrapping up

72 Application Monitoring is fun JMX is pretty simple JMX is very powerful JMX is better JMX is the standard For windows replace JMX with WMI above! Talk to your developers!

73 Questions/Thoughts/Ideas?

74 Information about NSClient++ Slides, and examples at: