Monitoring and Auditing WebSphere MQ

Transcription

1 Monitoring and Auditing Webphere MQ Morag Hughson ession # 8720 Monitoring and Auditing Webphere MQ his session will provide illustrations and insight on the various techniques and capabilities available for MQ Administrators to meet "Monitoring" and "Auditing" demands for Webphere MQ on distributed non-mainframe platforms. With the ever increasing demands to provide system runtime "high availability" along with the need to comply with the constantly changing internal and external audit demands, this session will provide information on what can be done from a Webphere MQ perspective.

2 Different types of Monitoring n-line status commands vent messages ff-line accounting and statistics Different types of Monitoring - otes here are several different types of monitoring features in Webphere MQ. You will likely use some of each type in order to look after your queue managers. ach serves a different purpose. here are immediate, on-line, commands that show the status of resources in the queue manager, such as queues and channels. hese can be useful for diagnosing problems right now, such why a channel is not moving messages, or why a queue has a deep current depth. here are event messages which the queue manager emits when certain interesting, or note-worthy occurrences happen. hese can be programmatically collected, processed and acted upon. here are off-line statistics and accounting records cut over longer intervals accumulating information about the application activity in the queue manager which can be programmatically collected and post-processed for charge-back or capacity planning purposes.

3 n-line status commands Queue Manager Resources Queues Channels opics Applications Connections Handles ubscriptions Available via MQC commands Programmable Command Format (PCF) MQ xplorer GUI tarting MQC for queue manager 1. DILAY QMAU n-line status commands otes Many of the resources in the queue manager have status commands. hese commands show the run-time view of that resource. here is a status command for the queue manager, although being able to issue any command of course tells you something about the status of the queue manager! here are also status commands for channels, queues and topics, and commands which show the run-time status of applications currently connected to the queue manager, and subscriptions currently active in the system. ome of these status commands have additional data that you can activate. his additional data is not enabled by default because it requires the capture of time stamp information to produce and collecting time stamps can be expensive on some platforms. You enable this additional data using the MCHL and MQ attributes on channels and queues respectively. hese status commands are available using MQC commands or via PCF (and thus through the MQ xplorer GUI and other such PCF based tools). In the MQ xplorer GUI you will find status display available through context menus on a single object, or from the object folder in the avigator (left hand pane) for status of many objects in one view.

4 Queue Manager tatus Is the queue manager alive PIG QMGR command xistence of particular processes Does not mean a queue manager can do anything useful dspmq control command qmstatus.ini ot a reliable way to tell if a queue manager is running DIPLAY QMAU command A summary of the run-time state of the queue manager tarting MQC for queue manager 1. PIG QMGR 1 : PIG QMGR AMQ8415: Ping Webphere MQ Queue Manager command complete. DIPLAY QMAU ALL 2 : DIPLAY QMAU ALL AMQ8705: Display Queue Manager tatus Details. QMAM(1) AU(RUIG) C(6) CMDRV(RUIG) CHII(RUIG) ADBY(PRMI) CURRLG( LG) RCLG( LG) MDIALG( LG) Queue Manager tatus otes he basic question you are trying to determine here is whether the queue manager is alive. f course that rather depends on what you mean by alive. In my view it means an application can connect. ne handy to command to remember is PIG QMGR. It needs several components to be working, including an application connecting, in order to complete successfully and so it a good test of whether the queue manager is open for business. HA scripts have typically used this to determine the health of a queue manager. ome people want to test for the existence of particular processes. However, I don t like this. ven if amqzxma0 exists, it does not mean the queue manager can do anything useful. he dspmq control command is also useful as it also does an MQC. It is not design to programmatically use, also the main drawback about translated fields has been solved in V7.0.1, so it could be used in this way successfully now too. ome will use the contents of qmstatus.ini file. his is not a reliable method. It will correctly state if a queue manager is down, but you cannot rely on it if it says a queue manager is running, as the queue manager may have failed in a way that meant the update to this file was not able to be made. he status command for the queue manager, DIPLAY QMAU covers a number of the run-time parts of the queue manager including the command server and channel initiator. It indicates details about the log files if you are using linear logging, and notes whether this is a multi-instance queue manager. he MQ xplorer GUI provides some of the output from this command in the quick view, and the full output can be displayed using the context menu on the queue manager and selecting tatus->general

5 tatus of run-time resources Queues Run-time information about the queue Details of each handle currently open on the queue Channels Partner details Logical UoW details How much work it has done tate of channel opics Resolution of hierarchically inherited attributes Publisher tatus ubscriber tatus tarting MQC for queue manager 1. DIPLAY PAU('ports/Football/London') YP(PIC) ALL 1 : DIPLAY PAU('ports/Football/London') YP(PIC) ALL AMQ8754: Display topic status details. PICR(ports/Football/London) ADMI(LD.FBALL) MDURMDL(YM.DURABL.MDL.QUU) MDURMDL(YM.DURABL.MDL.QUU) DFPI() DFPRY(0) DFPRP(YC) DURUB(Y) PUB(ABLD) UB(ABLD) PMGDLV(ALLDUR) PMGDLV(ALLAVAIL) RAID() PUBCU(0) UBCU(8) PUBCP(ALL) UBCP(ALL) tatus of run-time resources otes here are status commands allowing the run-time view of various Webphere MQ resources to be queried. For queues, there is DIPLAY QAU command which has two variants. DIPLAY QAU YP(QUU) shows overall information about the queue, for example, how many applications have an input or output handle open on that queue, the depth of the queue, whether there is any uncommitted work on this queue and when was the last time the queue was processed either to get or put messages. A second variant, DIPLAY QAU YP(HADL) shows more detailed information about each handle that currently has the queue open, including information about the application that owns the handle. For channels, there is DIPLAY CHAU command which shows the run-time information for each channel instance remembering of course that there may be multiple channel instances of the same name in the case of receiver and server-connection channels. he information displayed includes the details of the partner we are connected to; any logical unit of work information in the case of queue manager to queue manager channels; how much work the channel has done; and of course the state of the channel. For topics, there is DIPLAY PAU command which has three variants. DIPLAY PAU YP(PIC) shows the resolution of any APAR definitional values. opics are defined in a hierarchical structure which we call the topic tree. Any values which are not specifically set on a topic are inherited from the parent in the topic tree. his command makes the actual run-time values easy to see without having to display many different topic objects. wo further variants, DIPLAY PAU YP(PUB) and YP(UB) show more detailed information about each handle that currently has the topic open, including the connection ID which can be used with the DIPLAY C command to discover full information about the application that owns the handle.

6 Command xamples: Queues and opics DIPLAY QAU(Q1) YP(QUU) 1 : DIPLAY QAU(Q1) YP(QUU) AMQ8450: Display queue status details. QUU(Q1) CURDPH(5) LGDA( ) LPUDA( ) MDIALG( LG) MGAG(29) QIM( , ) YP(QUU) IPPRC(0) LGIM( ) LPUIM( ) MQ(HIGH) PPRC(1) UCM() DIPLAY PAU('ports/Football/London') YP(PIC) ALL 1 : DIPLAY PAU('ports/Football/London') YP(PIC) ALL AMQ8754: Display topic status details. PICR(ports/Football/London) ADMI(LD.FBALL) MDURMDL(YM.DURABL.MDL.QUU) MDURMDL(YM.DURABL.MDL.QUU) DFPI() DFPRY(0) DFPRP(YC) DURUB(Y) PUB(ABLD) UB(ABLD) PMGDLV(ALLDUR) PMGDLV(ALLAVAIL) RAID() PUBCU(0) UBCU(8) PUBCP(ALL) UBCP(ALL) DIPLAY QAU(Q1) YP(HADL) 2 : DIPLAY QAU(Q1) YP(HADL) AMQ8450: Display queue status details. QUU(Q1) YP(HADL) APPLDC( ) APPLAG(d:\nttools\q.exe) APPLYP(UR) BRW() CHAL(YM.DF.VRC) CAM( ) AA() HA(IACIV) IPU() IQUIR() UPU(Y) PID(14608) QMURID(0.0) () ID(*) URID(A_FRMAID[ ] A_GRID[] A_BQUAL[]) URYP(QMGR) URID(hughson@R820DGM) DIPLAY PAU('ports/Football/London') YP(PUB) ALL 2 : DIPLAY PAU('ports/Football/London') YP(PUB) ALL AMQ8754: Display topic status details. PICR(ports/Football/London) LPUBDA( ) LPUBIM(16:11:50) ACC(414D D FC16B4D ) UMPUB(1) DIPLAY PAU('ports/Football/London') YP(UB) ALL 3 : DIPLAY PAU('ports/Football/London') YP(UB) ALL AMQ8754: Display topic status details. PICR(ports/Football/London) UBID(414D D FC16B4D A) UBUR(hughson) RMDA( ) RMIM(16:12:49) LMGDA( ) LMGIM(16:12:51) ACC(414D D FC16B4D ) DURABL() UBYP(API) UMMG(4) Command xamples: Channels DIPLAY CHAU(YM.DF.VRC) ALL 1 : DIPLAY CHAU(YM.DF.VRC) ALL AMQ8417: Display Channel tatus details. CHAL(YM.DF.VRC) CHLYP(VRC) BUFRCVD(9) BUF(8) BYRCVD(1688) BY(1516) CHADA( ) CHAI( ) CMPHDR(,) CMPMG(,) CMPRA(0,0) CMPIM(0,0) CAM( ) CURR IIM(0,0) HBI(300) JBAM( B14) LCLADDR( ) LMGDA( ) LMGI( ) MCAA(RUIG) MCAUR(hughson) MCHL(FF) MG(5) RAPPLAG(d:\nttools\q.exe) LCRI( ) LKYDA( ) LKYI( ) LPR( ) LRKY(0) AU(RUIG) PRQ() UBA(RCIV) CURHCV(1) MAHCV(10) DIPLAY CHAU(1MAH..2MAH) ALL 2 : DIPLAY CHAU(1MAH..2MAH) ALL AMQ8417: Display Channel tatus details. CHAL(1MAH..2MAH) CHLYP(DR) BACH(7) BACHZ(50) BUFRCVD(8) BUF(13) BYRCVD(432) BY(5509) CHADA( ) CHAI( ) CMPHDR(,) CMPMG(,) CMPRA(0,0) CMPIM(0,0) CAM( (1502)) CURLUWID(3DC16B4D ) CURMG(0) CURR CURQ(11) IIM(0,0) HBI(300) IDUB() JBAM( ) LCLADDR( (1901)) LGR( ) LLUWID(3DC16B4D ) LMGDA( ) LMGI( ) LQ(11) MCAA(RUIG) MCHL(HIGH) MG(11) IM(110,17) PMPD(FA) RQMAM(2MAH) HRR(10) LCRI( ) LKYDA( ) LKYI( ) LPR( ) LRKY(0) AU(RUIG) PRQ() UBA(MQG) BACHZ(1,1) MIQ(2MAH) QIM(1794,344)

7 tatus of Applications Application connection What Who UoW details Application resources Handles opened on queues, topics and subscriptions ubscription details ame information as in opic status tarting MQC for queue manager 1. DIPLAY C(*) WHR(APPLYP Q UR) YP(ALL) ALL 1 : DIPLAY C(*) WHR(APPLYP Q UR) YP(ALL) ALL AMQ8276: Display Connection details. C(3FC16B4D ) C(414D D ) YP(C) APPLAG(d:\nttools\q.exe) APPLYP(UR) AA() CP(MQC_HARD_BIDIG) URID(hughson) BJAM( ) BJYP(PIC) AA() HA(IACIV) PP(MQ_UPU,MQ_FAIL_IF_QUICIG) PICR(ports/Football/London) tatus of Applications otes DIPLAY C provides information about the applications connected to the queue manager and the handles that they have open. Various other commands output a connection ID which can be used as input into DIPLAY C in order to find out full details about the application that made the connection in question. You can find out what the application is, whether client or locally connected for example; the user id it is running under; and when the last UoW was started helpful for diagnosing long running UoWs. You can also see details of the handles this connection has open. his shows the same information as DIPLAY QAU and DIPLAY PAU YP(PUB/UB) but in a different perspective. hose other commands looked at the information from the perspective of the resource and who is using the resources. his command shows the information in the perspective of the application and what resources it is using. It is useful to know that you can start from either perspective and get the same information. DIPLAY BAU provides the same information about subscriptions as DIPLAY PAU YP(UB) but from the perspective of a subscription rather than the resource, as one subscription may cover a number of topics if it has been made using a wildcarded string.

8 Command xamples: Connections DIPLAY C(*) WHR(APPLYP Q UR) YP(ALL) ALL 1 : DIPLAY C(*) WHR(APPLYP Q UR) YP(ALL) ALL AMQ8276: Display Connection details. C(3FC16B4D ) C(414D D ) YP(C) PID(16932) ID(1) APPLDC( ) APPLAG(d:\nttools\q.exe) APPLYP(UR) AA() CHAL( ) CAM( ) CP(MQC_HARD_BIDIG) URID(hughson) UWLG( ) UWDA( ) UWI( ) UWLGDA( ) UWLGI( ) URYP(QMGR) URID(A_FRMAID[ ] A_GRID[] A_BQUAL[]) QMURID(0.0) UWA() BJAM( ) BJYP(PIC) AA() HA(IACIV) PP(MQ_UPU,MQ_FAIL_IF_QUICIG) RADA() PICR(ports/Football/London) DIPLAY C(*) WHR(APPLYP Q UR) YP(ALL) ALL 1 : DIPLAY C(*) WHR(APPLYP Q UR) YP(ALL) ALL AMQ8276: Display Connection details. C(3FC16B4D20003B01) C(414D D ) YP(C) PID(17740) ID(3) APPLDC( ) APPLAG(d:\nttools\q.exe) APPLYP(UR) AA(ARD) CHAL(YM.DF.VRC) CAM( ) CP(MQC_HADL_HAR_BLCK,MQC_HARD_BIDIG) URID(hughson) UWLG( ) UWDA( ) UWI( ) UWLGDA( ) UWLGI( ) URYP(QMGR) URID(A_FRMAID[ ] A_GRID[] A_BQUAL[]) QMURID(0.57) UWA(ACIV) BJAM(Q1) BJYP(QUU) AA(ACIV) HA(ACIV) PP(MQ_IPU_HARD,MQ_FAIL_IF_QUICIG) RADA() Command xamples: ubscriptions DIPLAY BAU('portsWriter') ALL 48 : DIPLAY BAU('portsWriter') ALL AMQ8099: Webphere MQ subscription status inquired. UB(portsWriter) UBID(414D D FC16B4D ) UBUR(hughson) RMDA( ) RMIM(19:04:48) LMGDA( ) LMGIM(19:05:36) ACC(414D D FC16B4D ) DURABL() UMMG(1) UBYP(API)

9 Webphere MQ vent Messages vent Messages for Auditing ecurity Failures Commands Issued Configuration Changes vent Messages for Monitoring tarts and tops of resources tarts and tops of Channels Channel errors Application errors using resources Performance of message processing ervicing of queues Depth of queues Webphere MQ vent Messages otes When Webphere MQ needs to emit information it often does so in the form of event messages. everal of the auditing features in Webphere MQ make use of this mechanism as well as many monitoring features so we will take a general look at event messages and how they are built and then we will look at the details for the various features that use event messages.

10 vent Messages Written to specific queue Created as LCAL queues in the default set of objects Can be redefined as RM queues Controlled by ALR QMR switch Message Descriptor Format = MQFM_V V Well defined message format MQCFH header PCF sub-structures vent Message YM.ADMI.<feature name>.v ALR QMGR <type>v(abld) PCF Header (MQCFH) PCF Parameters vent Messages - otes vent messages are used for various features of MQ and they share a common mechanism for configuration and format for the emitted events. As we look at each specific event feature later we will see each event type is written to a specific queue name which follows the pattern YM.ADMI.<feature name>.v, and is controlled by a switch, an attribute on ALR QMGR, usually with ABLD/DIABLD as the values (although sometimes there are additional options). he switches also follow a pattern of <type>v as the name. he various event queues are defined as part of the default objects on the queue manager (on distributed platforms) and in CQ4I?? CQIP2 samples (on z/) as LCAL queues. hey can be redefined as RM queues to funnel all event messages to a central queue manager to process if wished. he queue name just needs to be correct. he format of event messages is shown as MQFM_V ( V )in the message descriptor (MQMD) which is a PCF header. his means the message starts with a header using the structure MQCFH. his header is then followed by 1 or more parameter structures which are self-describing sub-structures providing the data of the actual event message.

11 Viewing vent Messages Management or monitoring tool o general event formatter provided with WMQ everal upportpacs available M01 (C) - including source code M0K (C) - including source code M12 (CBL) - including source code M0P WMQ xplorer plug-in M71 GUI Administrator Viewing vent Messages In order to view MQ vent messages we would normally expect there to be some management or monitoring tool to be used. However, event messages are just an MQ Message with a specific published format (which we will take a look at in a moment) so writing an application to view them is also possible. Apart from buying purpose built MQ monitoring tools, there are also a number of freely available MQ upport Pacs that allow the browsing and formatting of MQ event messages. M0P and M71 provide graphical tools that will format event messages (among various other task that they provide). Whether you buy or download a tool, or write your own application, something should be consuming the event messages that you configure you queue manager to emit, and taking appropriate action based on them.

12 vent Message PCF Header PCF Header (MQCFH) ype MQCF_V truclength, Version tandard fields in all MQ headers Command MQCMD_* Indicates what the event is A value for each category of event Msgequmber, Control Helpful with sets of related events CompCode, Reason MQCC_WARIG MQRC_* A value for each type of event within the category ParameterCount How many pieces of data will follow vent Message ype truc Length Version Command Msgeq umber Control Comp Code Reason Parameter Count PCF Header (MQCFH) PCF Parameters vent Message PCF Header - otes We will look at the MQCFH header here and some example (common) substructures, and then for each specific event type later we will see more detail on the sort of data that follows in the sub-structures. he ype field tells you that this is an event message (MQCF_V) since the PCF Header is actually used for other messages as well. PCF is a format that allows self-describing messages so it is very useful for lots of things. he truclength and Version fields are common to all MQ defined headers, allowing future-proofing against changes (extensions) to the structure. ur first important field is the Command field. his will contain a value that will indicate what category of event it is, with a defined MQCMD_* constant to use. his means that even if you have all your event messages aliased to a single central queue, you can still tell what they are. We will see the value used for the specific types of events when we come onto them. his is further qualified by Reason which we will see in a moment.

13 vent Message PCF Header - otes Msgequmber and Control are helpful when a single event is written as multiple messages. Msgequmber contains an incrementing number of the event message within the set, and Control flags the last messages so you know when you have all the messages in the set. When a single event maps to a single message which is what happens most of the time, you will see Msgequmber = 1, and Control = MQCFC_LA. We will see this used with one of the specific event types later. CompCode and Reason indicate the type of event within the category. CompCode is always MQCC_WARIG, so Reason is the important one. ParameterCount tells you how much data to expect to see following this PCF header. he count will vary with event types and sometimes even within event types. PCF Header C and CBL struct tagmqcfh { MQLG ype; MQLG truclength; /* tructure type */ /* tructure length */ MQLG Version; /* tructure version number */ MQLG Command; MQLG Msgequmber; /* Command identifier */ /* Message sequence number */ MQLG Control; /* Control options */ MQLG CompCode; MQLG Reason; /* Completion code */ /* Reason code qualifying completion code */ MQLG ParameterCount; /* Count of parameter structures */ }; ** MQCFH structure 10 MQCFH. ** tructure type 15 MQCFH-YP PIC 9(9) BIARY. ** tructure length 15 MQCFH-RUCLGH PIC 9(9) BIARY. ** tructure version number 15 MQCFH-VRI PIC 9(9) BIARY. ** Command identifier 15 MQCFH-CMMAD PIC 9(9) BIARY. ** Message sequence number 15 MQCFH-MGQUMBR PIC 9(9) BIARY. ** Control options 15 MQCFH-CRL PIC 9(9) BIARY. ** Completion code 15 MQCFH-CMPCD PIC 9(9) BIARY. ** Reason code qualifying completion code 15 MQCFH-RA PIC 9(9) BIARY. ** Count of parameter structures 15 MQCFH-PARAMRCU PIC 9(9) BIARY.

14 xample parameters tring (MQCF) field example ype MQCF_RIG truclength Parameter MQCA_Q_MGR_AM for our example CodedCharetId (CCID) he codepage that the string characters are represented in tringlength How long the string following is tring he actual data Integer field example ype MQCF_IGR truclength Parameter MQIACF_RA_QUALIFIR Value he actual data MQRQ_* MQCFI sub-structure MQCF sub-structure ype truc Length Parameter Value ype truc Length Parameter CCID tring Length tring xample Parameters We show a couple of example parameters here to give you an idea of the structure of the data in an event message. We have chosen one of each of the most common data types, a string parameter and an integer parameter. he example, the string QMgrame field is used by many different types of event messages as it records the name of the queue manager generating the event. ur second example, the integer ReasonQualifier field is used by a number of event types to further qualify the type of event within a category we will see one of the events we look at later using it.

15 ecurity Failures Audit rail of security access failures on your queue manager Queue Manager Attribute AUHRV PCF Header fields Command MQCMD_Q_MGR_V YM.ADMI.QMGR.V Reasons MQRC AUHRIZD Reason Qualifier MQRQ_C AUHRIZD MQRQ_P AUHRIZD MQRQ_CL AUHRIZD MQRQ_CMD AUHRIZD MQRQ_UB AUHRIZD MQRQ_UB_D AUHRIZD ALR QMGR AUHRV(ABLD) ecurity Failures otes n the distributed platforms, an audit trail of access failures is kept by means of event messages which are written to the YM.ADMI.QMGR.V queue. You can enable these events to be written by means of the AUHRV switch on ALR QMGR. here are several different types of MQRC AUHRIZD events showing specifically what kind of access was attempted. ach of these types has a different reason qualifier recorded in the event message. MQRQ_C AUHRIZD MQRQ_P AUHRIZD MQPU1 ==> MQP MQRQ_CL AUHRIZD For deletion of dynamic queues MQRQ_CMD AUHRIZD Webphere MQ MQC/PCF commands MQRQ_UB AUHRIZD subscribe check failed MQRQ_UB_D AUHRIZD destination queue check failed and, where applicable, there is information in each event message to show the user ID and application that made the failed access attempt.

16 ot Authorized vent Message Details Reason Qualifier QMgr ame User Identifier Appl ype Appl ame ptions Dest pen ptions bject QMgr ame bject ame * opic tring Admin opic ames Command MQRQ_C AUHRIZD MQRQ_P AUHRIZD MQRQ_CL AUHRIZD MQRQ_CMD AUHRIZD MQRQ_UB AUHRIZD MQRQ_UB_D AUHRIZD * Might be Q ame, Process ame (P only), amelist ame (P only), ub ame (CL only) ot Authorized vent Message Details - otes Much of the contents of the ot Authorized event messages are fairly selfexplanatory, but we will look at a few of the interesting fields in a bit more detail. f course you can read about the details of event messages in the Monitoring Webphere MQ book too. he not authorised events that are from API calls, all except MQRQ_CMD AUHRIZD, provide details of the application making the call as well as the user ID that the application was running under that did not have the required access. hese application identity fields are the same fields you will see on DIPLAY C, APPLAG and APPLYP along with URID. MQP and MQUB authorisation failures will provide the options used on the verb to allow you to work out what access was attempted on the resource. hese are integer fields that contain all the options and can be decoded most easily working in hex (or of course using a tool such as in M0P or M71 to decode these for you).

17 Identifying the Application vent message contains same info you d see on DIPLAY C command APPLAG APPLYP URID (ot Authorized events only) User Identifier Appl ype Appl ame AMQ8276: Display Connection details. C(3C75FA4B2001BA01) YP(C) PID(9428) ID(1) APPLAG(d:\nttools\q.exe) APPLYP(UR) URID(hughson) CHAL( ) CAM( ) CP(MQC_HARD_BIDIG) Decoding ptions An integer field containing the options used xamples ptions (from MQP) 0x MQ_UPU 0x MQ_IPU_HARD 0x ptions Dest pen ptions ptions (from MQUB) MQ_CRA MQ_DURABL 0x A 0x x

18 opic ecurity Apples MQUB Price/Fruit/Apples Using Q1 MQG (Q1) Fruit Q1 FRUI ranges YM.BA.PIC Price Authority check on topic objects Walk up the tree May be more than one check Failed event records all topic objects used for authority check Authority check on destination queue When not using MQ_MAAGD Check is for PU to that queue opic tring Admin opic ames opic ecurity When MQPing a topic (MQ_PIC) for MQ_UPU that is, in order to publish, or when making an MQUB call to subscribe to a topic, a security check is done to see if your user ID has authority to use that topic. In our example we have called MQUB at the point in the topic tree, Price/Fruit/Apples. here is no topic object at this point in the topic tree, so to find the profile we need to check authorities against we walk up the topic tree to find a node which does have a topic object. he next point is Price/Fruit. his does have a topic object, FRUI, so we will check that this user ID has subscribe authority on the profile for the FRUI topic. If that user ID does have authority, our search stops there. If it does not, we carry on searching up the topic tree and will check the YM.BA.PIC to see if this user ID has subscribe authority there. An additional authorisation check is done for an MQUB call when the application wishes to use a specific destination queue (i.e. is not using the MQ_MAAGD option). In this case we also check that this user ID has authority to PU to that destination queue.

19 Commands Issued Audit rail of MQC/PCF commands issued on your queue manager Queue Manager Attribute CMDV DIPLAY Command Failed => o event PCF Header fields Command MQCMD_CMMAD_V YM.ADMI.CMMAD.V ALR QMGR CMDV(ABLD) Possible Reasons MQRC_CMMAD_MQC MQRC_CMMAD_PCF Distributed platform Command events in V7.0.1 M0P contains a limited Command vent generator for PCF admin For older Distributed systems Commands Issued - otes An audit trail of commands issued is kept by means of event messages which are written to the YM.ADMI.CMMAD.V queue. You can enable these events to be written by means of the CMDV switch on ALR QMGR. You can choose to record all commands that are issued, or perhaps more usefully, all commands except DIPLAY commands (PCF Inquire commands), so that you only capture a record of those potentially destructive or interesting commands. his is done using CMDV(DIPLAY). If the command issued failed, for example a syntax error, then no command event is generated. he PCF Header of a command event message will record the Command field as MQCMD_CMMAD_V and can have one of two possible Reasons, MQRC_CMMAD_MQC or MQRC_CMMAD_PCF. he next page will show the differences. Command events are available on z/ in V6 and Distributed platforms in V upportpac M0P also provides an API xit that can be used to log all commands sent to the Command erver on the Distributed platforms for earlier versions.

20 Command vent Message Details Command Context vent rigin vent User ID vent QMgr vent Accounting oken vent Identity Data vent Appl ype vent Appl ame vent Appl rigin Command Command Data * MQV_CL MQV_II MQV_MG MQV_IRAL MQV_HR * ither PCF message as it was submitted, or text of MQC command Command vent Message Details - otes Much of the contents of the Command event messages are fairly self-explanatory, but we will look at a few of the interesting fields in a bit more detail. f course you can read about the details of event messages in the Monitoring Webphere MQ book too. he contents of the command event message varies depending on how the command was issued. If the command was a PCF message then the content of the input PCF message is part of the command event. Alternatively, if the command was an MQC message then this text string will be found in the event message instead of the PCF input message. If the command was issued by putting a message on the command server queue (MQV_MG) then there will be more application identifying information than in other cases because the Message Descriptor (MQMD) of the command message written by the application contains lots of extra data. In all cases you will get the user ID issuing the command, the queue manager where the command was entered, and one of the two aforementioned command data variants.

21 Configuration Changes Audit trail of changes to the configuration of the queue manager. Commands acting on objects MQ calls Queue Manager Attribute CFIGV Create a base-line view with RFRH QMGR PCF Header fields Command MQCMD_CFIG_V Possible Reasons MQRC_CFIG_CHAG_BJC MQRC_CFIG_CRA_BJC MQRC_CFIG_DL_BJC MQRC_CFIG_RFRH_BJC YM.ADMI.CFIG.V ALR QMGR CFIGV(ABLD) RFRH QMGR YP(CFIGV) BJC(ALL) AM(*) Configuration Changes - otes An audit trail of changes to the queue manager configuration is kept by means of event messages which are written to the YM.ADMI.CFIG.V queue. You can enable these events to be written by means of the CFIGV switch on ALR QMGR. hese events will be generated when a DFI, ALR or DL command acts upon an object, or an MQ command is used. A base-line picture of the current queue manager configuration can be created by using the RFRH QMGR YP(CFIGV) command which will create an event message for every object in the queue manager. ince this could be a heavyweight operation if you have a lot of objects, you can break it down into smaller sets of objects using the AM and BJC qualifiers on the command. he PCF Header of a configuration event message will record the Command field as MQCMD_CFIG_V and can have one of four possible Reasons, MQRC_CFIG_CHAG_BJC, MQRC_CFIG_CRA_BJC or MQRC_CFIG_DL_BJC for the respective MQC or PCF commands that you might issue upon an object or MQRC_CFIG_RFRH_BJC for those event messages written when creating the base-line picture. he next page will show the details. Config events are available on z/ in V5.3 and Distributed platforms in V7.0.1.

22 Config vent Message Details All config events Command Context just as Command vent Message Details bject ype bject ame Disposition (z/ only) PCF Header When 2 event messages Msgequmber = 1, 2 Control = MQCFC LA, MQCFC_LA MQRC_CFIG_CHAG_BJC 2 event messages Attributes before change Attributes after change MQRC_CFIG_CRA_BJC 1 event message Attributes after create MQRC_CFIG_DL_BJC 1 event message Attributes before deletion MQRC_CFIG_RFRH_BJC 1 event message Current attributes of object Config vent Message Details - otes Much of the contents of the Config event messages contain object attributes. hese are PCF sub-structures when one issues a PCF command to make or alter an object, and exactly the same sub-structures are used in these event messages. f course you can read about the details of event messages in the Monitoring Webphere MQ book too. he contents of the command event message varies depending on how the command was issued just as with command events. If the command was issued by putting a message on the command server queue (MQV_MG) then there will be more application identifying information than in other cases because the Message Descriptor (MQMD) of the command message written by the application contains lots of extra data. In all cases you will get the user ID issuing the command, the queue manager where the command was entered. In the specific case of the MQRC_CFIG_CHAG_BJC, you will get two messages, one containing the object attributes before the change and one containing those after the change. hese will be indicated using the Msgequmber field and the Control field in the PCF Header.

23 Combining Command and Config vents ALR Q(FRD) MADPH(1) Command vent Before Change Config vent After Change Config vent ALR QMGR CMDV(DIPLAY) CFIGV(ABLD) Correl ID = 1234 Correl ID = 1234 YM.ADMI.CMMAD.V YM.ADMI.CFIG.V Combining Command and Config vents - otes If you have both Command events and Configuration events enabled, then when an object is changed, the event messages will share the same correlation ID in their MQMDs.

24 tart and top vents otification of queue manager start and stop Queue Manager Attribute RPV DFPI of event queue YM.ADMI.QMGR.V PCF Header fields Command MQCMD_Q_MGR_V ALR QMGR RPV(ABLD) Possible Reasons MQRC_Q_MGR_ACIV MQRC_Q_MGR ACIV Reason Qualifier MQRQ_Q_MGR_PPIG MQRQ_Q_MGR_QUICIG Reason MQRC_Q_MGR_ACIV MQRC_Q_MGR ACIV QMgr ame Reason Qualifier tart and top vents - otes Whenever a queue manager starts or stops, an event messages is written to the YM.ADMI.QMGR.V queue. You can enable these events to be written by means of the RPV switch on ALR QMGR. he PCF Header of a start or stop event message will record the Command field as MQCMD_Q_MGR_V and can have two possible Reasons where the MQRC_Q_MGR ACIV can have two possible Reason Qualifiers as shown. A stop event is only recorded if the DFPI attribute of the YM.ADMI.QMGR.V queue is defined as persistent.

25 Channel vents otification of channel starts, stops and errors, including L errors Queue Manager Attributes CHLV and LV CPI erver-connection channels do not cause start or stop events PCF Header fields Command MQCMD_CHAL_V YM.ADMI.CHAL.V ALR QMGR CHLV(CPI) LV(ABLD) Possible Reasons MQRC_CHAL_ACIVAD MQRC_CHAL ACIVAD MQRC_CHAL_ARD MQRC_CHAL_PPD MQRC_CHAL_PPD_BY_UR MQRC_CHAL_CVRI_RRR MQRC_CHAL_L_RRR MQRC_CHAL_L_WARIG Reason Qualifier MQRQ_CHAL_PPD_K MQRQ_CHAL_PPD_RRR MQRQ_CHAL_PPD_RRY MQRQ_CHAL_PPD_DIABLD MQRQ_L_HADHAK_RRR MQRQ_L_CIPHR_PC_RRR MQRQ_L_PR_AM_RRR MQRQ_L_CLI_AUH_RRR MQRQ_L_UKW_RVCAI Channel vents - otes Whenever a channel does something noteworthy, an event messages is written to the YM.ADMI.CHAL.V queue. You can enable these events to be written by means of the CHLV switch on ALR QMGR. Additionally, if you are using L on your channels, you can get detailed event messages on the reason for any L failures by enabling L error events by means of the LV switch on ALR QMGR. hese are written to the same event queue. You can choose to record all channel events that are issued, or perhaps more usefully, all the ones relating to error situations. his is done using CHLV(CPI). he PCF Header of a channel event message will record the Command field as MQCMD_CHAL_V and can have several possible Reasons and Reason Qualifiers as shown.

26 Channel Auto-definition vents otification of attempts to automatically define channels Queue Manager Attributes CHAD CHADV CHADI PCF Header fields Command MQCMD_CHAL_V YM.ADMI.CHAL.V ALR QMGR CHAD(ABLD) CHADV(ABLD) CHADI(exit-name) Possible Reasons MQRC_CHAL_AU_DF_RRR MQRC_CHAL_AU_DF_K Channel Auto-defintion vents - otes If you have enabled channel auto-definition by means of the CHAD switch on ALR QMGR, then an inbound channel connection who wishes to use a channel that is not defined will cause it to be created (modeled off YM.AU.* channels). You can further configure this behaviour with an exit whose name is specified in the CHADI attribute, and you can also have events emitted whenever an auto-definition of a channel is attempted. Whenever a channel is automatically defined, an event messages is written to the YM.ADMI.CHAL.V queue. You can enable these events to be written by means of the CHADV switch on ALR QMGR. he PCF Header of a channel event message will record the Command field as MQCMD_CHAL_V and can have one of two possible Reasons, MQRC_CHAL_AU_DF_K or MQRC_CHAL_AU_DF_RRR.

27 Channel vent Message Details Reason QMgr ame Channel ame mitq ame Connection ame rror Identifier Aux rror Data Int 1/2 Aux rror Data tring 1/2/3 Conversion Reason Code Format Channel ype L Handshake tage L Return Code L Peer ame MQRC_CHAL_ACIVAD MQRC_CHAL ACIVAD MQRC_CHAL_ARD MQRC_CHAL_PPD MQRC_CHAL_PPD_BY_UR MQRC_CHAL_CV_RRR MQRC_CHAL_L_RRR MQRC_CHAL_L_WARIG MQRC_CHAL_AU_DF_K MQRC_CHAL_AU_DF_RRR Channel vent Message Details - otes Much of the contents of the Channel event messages are fairly self-explanatory, but we will look at a few of the interesting fields in a bit more detail. f course you can read about the details of event messages in the Monitoring Webphere MQ book too. he channel events all provide details of the channel in questions. hese channel detail fields are the same fields you will see on DIPLAY CHAU, the channel name, MIQ and CAM. he events that report channels failing due to an error, which are probably the most interesting ones for most people, include some interesting fields that it is helpful to know how to interpret.

28 Identifying the Channel vent message contains same info you d see on DIPLAY CHAU command Channel name MIQ CAM Channel ame mitq ame Connection ame AMQ8417: Display Channel tatus details. CHAL(1MAH..2MAH) CHLYP(DR) CAM( (1502)) CURR RQMAM(2MAH) AU(RUIG) UBA(MQG) MIQ(2MAH) Decoding rror Information rror Identifier xample 0x Use mqrc tool Can enter number in hex or decimal Receive Failed C:\>mqrc 0x x rrc_rciv_faild rror Identifier Aux rror Data Int 1/2 Aux rror Data tring 1/2/3 Auxiliary data Contains data that would be seen in the inserts of the equivalent error log message C:\>mqrc AMQ9208 MAG: rror on receive from host <insert one>. PLAAI: An error occurred receiving data from <insert one> over <insert two>. his may be due to a communications failure. ACI: he return code from the <insert two><insert three> call was 1111 ('8A'). Record these values and tell the systems administrator.

29 Application errors using resources otification of application failures to use local or remote queues, inhibited resources Queue Manager Attributes LCALV RMV IHIBIV PCF Header fields Command MQCMD_Q_MGR_V YM.ADMI.QMGR.V Possible Reasons MQRC_ALIA_BA_Q_YP_RRR MQRC_UKW_ALIA_BA_Q MQRC_UKW_BJC_AM MQRC_DF_MI_Q_YP/UAG_RRR MQRC_Q_YP_RRR MQRC_RM_Q_AM_RRR MQRC_MI_Q_YP/UAG_RRR MQRC_UKW_DF_MI_Q MQRC_UKW_RM_Q_MGR MQRC_UKW_MI_Q MQRC_G_IHIBID MQRC_PU_IHIBID ALR QMGR LCALV(ABLD) RMV(ABLD) IHIBIV(ABLD) Reflects an MQRC also given back to the application Handy when application forgets to check! Application errors using resources otes hese events reflects error that are also returned to applications. hey indicate problems where either the object that the application needs to use is defined incorrectly, or the application is using the wrong name of an object, perhaps more likely is cases such as MQRC_UKW_BJC_AM. he inhibit events reflect a problem where an application has attempted to use a resource which is inhibited for the operation required. MQRC_PU_IHIBI events are created for PU(DIABLD) queues and PUB(DIABLD) topics. hese events can also be generated by internal operations in the queue manager, such as the writing of an event message. In case it would reflect the ReplyoQ and ReplyoQMgr fields of an originating application message containing incorrect information or pointing at a resource that is not correctly defined.

30 Local/Remote/Inhibit vent Message Details Reason MQRC_ALIA_BA_Q_YP_RRR QMgr ame bject ame * bject QMgr ame Base bject ame mit Q ame Base ype Q ype Appl ype Appl ame opic tirng MQRC_UKW_ALIA_BA_Q MQRC_UKW_BJC_AM MQRC_DF_MI_Q_YP_RRR MQRC_DF_MI_Q_UAG_RRR MQRC_Q_YP_RRR MQRC_RM_Q_AM_RRR MQRC_MI_Q_YP_RRR MQRC_MI_Q_UAG_RRR MQRC_UKW_DF_MI_Q MQRC_UKW_RM_Q_MGR MQRC_UKW_MI_Q MQRC_G_IHIBID MQRC_PU_IHIBID * Might be Q ame, Process ame, amelist ame or opic ame Local/Remote/Inhibit vent Message Details - otes Much of the contents of these queue manager event messages are fairly selfexplanatory, but we will look at a few of the interesting fields in a bit more detail. f course you can read about the details of event messages in the Monitoring Webphere MQ book too. hese events are all caused by some application making an MQ API call (mostly MQPs) which failed. o they all provide details of the application making the call. hese application identity fields are the same fields you will see on DIPLAY C, APPLAG and APPLYP. We saw an example of this earlier with the ot Authorized events. When diagnosing an issue reported by one of these events, either the application has coded the wrong thing, or there is an administrative definition missing or incorrect that the application needs to use. You can see the details of exactly what the application coded in the MQD.bjectame and MQD.bjectQMgrame fields by looking at the equivalent fields in the event message.

31 Application MQD details MQP API call MQD describes the object to open bjectame bjectqmgrame MQMD describes the object to open for a Response/Report message ReplyoQ ReplyoQMgr vent message data provides these MQD bjdesc = {MQD_DFAUL}; bject ame * bject QMgr ame bjdesc.bjectype strncpy(bjdesc.bjectame, MY.APP.QUU, MQ_Q_AM_LGH); = MQ_Q;

32 Queue ervice Interval vents otification of whether queues are being processed in a timely manner MQGs or CLAR QLCAL Queue Manager Attribute PRFMV Queue Attributes QVCIV vent switch QVCI Interval PCF Header fields Command MQCMD_PRFM_V Possible Reasons MQRC_Q_RVIC_IRVAL_HIGH MQRC_Q_RVIC_IRVAL_K YM.ADMI.PRFM.V ALR QMGR PRFMV(ABLD) ALR QLCAL(q-name) QVCIV(HIGH) QVCI(10000) Queue ervice Interval vents otes Queue ervice Interval events give you notification that a queue is not being processed in a timely enough manner. You can set the interval on the queue (in milliseconds) for how quickly you expect gets to be issued after some messages are there to be processed, and if a get hasn t been used within that interval you will get an event. A CLAR QLCAL command will also count as a get for the purposes of these events. Both these High/Low events and Queue Depth High/Low events work in cooperation with each other as we will see in a moment.

33 Queue Depth vents otification of when queues start to fill up with messages Queue Manager Attribute PRFMV Queue Attributes QDPHIV High Depth vent witch QDPLV Low Depth vent witch QDPMAV Max Depth vent witch QDPHHI - % of Max Depth QDPHL - % of Max Depth MADPH PCF Header fields Command MQCMD_PRFM_V Possible Reasons MQRC_Q_DPH_HIGH MQRC_Q_DPH_LW MQRC_Q_FULL YM.ADMI.PRFM.V ALR QMGR PRFMV(ABLD) ALR QLCAL(q-name) MADPH(1000) QDPHHI(80) QDPHL(20) QDPHIV(ABLD) Queue Depth vents otes Queue Depth events give you prior warning that your queue is getting full. A queue full event of course tells you that the queue is full too late!! However, you can get prior warning by using the Queue High event, and setting the point you wish to be notified as a percentage of the maximum depth of the queue. here are also Queue Low events which tell you when the panic is over and the alarms can be turned off again! Both these High/Low events and Queue ervice Interval High/Low events work in co-operation with each other as we will see in a moment.

34 Performance vent Message Details Reason QMgr ame Q ame ime ince Reset tatistics High Q Depth Msg nq Count Msg Deq Count Performance vent tatistics are reset A performance event is emitted he queue manager restarts Reset Queue tatistics (PCF) command is issued R QA (MQC) command is issued (z/ only) ime ince Reset hows time of last one of the above actions MQRC_Q_RVIC_IRVAL_HIGH MQRC_Q_RVIC_IRVAL_K MQRC_Q_DPH_HIGH MQRC_Q_DPH_LW MQRC_Q_FULL Performance vent Message Details otes o this is not the most interesting table we ve seen today. All these different event messages have exactly the same content. What is perhaps more interesting is the way these events operate which we will look at on the next page. ote that the statistics that are provided as part of a performance event are reset are various points, when a performance event is emitted; when the queue manager restarts; and when a command is issued to reset the statistics. he ime ince Reset parameter in the event message details when the last one of these happens. If no reset has happened it will contain the time of the last event message.

35 he Highs and Lows of Performance vents Queue High vent emitted Queue High Disabled Queue Low nabled Queue Full nabled Queue Full vent emitted Queue Full Disabled ALR QMGR PRFMV(ABLD) ALR QLCAL(q-name) MADPH(1000) QDPHHI(80) QDPHL(20) QDPHIV(ABLD) Queue Low vent emitted Queue High nabled Queue Low Disabled Queue Full nabled he Highs and Lows of Performance vents - otes Queue ervice Interval and Queue Depth events are unusual compared to the other events we have looked at in that they operate in pairs. here is a High event which alerts the monitor of a problem, a slow processing application or backlog of messages, and then when the problem is solved, and the queue is being serviced in a timely manner again, or the depth has returned to a nice low value, the corresponding Low event is emitted. his means that only one of the High or Low event from the pair is set at any one time and when the event is generated, let s say the High one, then the High event switch is disabled and the Low event switch is enabled. he reverse happens when the Low event is reached. For Depth events there is also the Queue Full event to consider. his is automatically switched on when the Queue High state is reached. here isn t an equivalent event for the Queue ervice Interval pair though.

36 Accounting Messages Collects information about the applications which connect to the Queue Manager verall Per Queue Queue Manager Attribute ACCI YM.ADMI.ACCUIG.QUU verall MQI Accounting Queue Manager Attribute ACCMQI Queue Accounting Queue Manager Attribute ACCQ Queue Attribute ACCQ PCF Header fields Command MQCMD_ACCUIG_MQI overall information MQCMD_ACCUIG_Q queues message ALR QMGR ACCI(1800) ACCMQI() ACCQ() ALR QLCAL(q-name) ACCQ(QMGR) Accounting Messages otes Accounting Monitoring Data collects information about the applications which connect to the Queue Manager. his data, when enabled by means of the ACCMQI switch on ALR QMGR, is written in the form of PCF records to the YM.ADMI.ACCUIG.QUU and can be post-processed to give information on the activity of each application connected. he PCF formatted message is written upon MQDIC (or at regular intervals for long running tasks, controlled by the ACCI attribute on ALR QMGR). As well as the standard collection details, detailed queue information for each connection may also be collected. Written at the same time as the standard accounting data collection, a number of PCF formatted messages may we written which includes accounting information for each queue opened by the connection (up to 100 queue details per message). he queues that this information is written for is controlled by the ACCQ attribute on queues and on the queue manager (providing a way to switch on and off for multiple queues at once). he PCF Header of an accounting data message will record the Command field as MQCMD_ACCUIG_MQI for the main accounting information and MQCMD_ACCUIG_Q for the additional messages written for queues.

37 tatistics Messages Collects information about Webphere MQ resources Queue Manager Per Queue Per Channel Queue Manager Attribute AI verall MQI tatistics Queue Manager Attribute AMQI Queue and Channel tatistics Queue Manager Attributes AQ, ACHL and AACL Queue Attribute AQ Channel Attribute ACHL PCF Header fields Command MQCMD_AIIC_MQI MQCMD_AIIC_Q MQCMD_AIIC_CHAL YM.ADMI.AIIC.QUU ALR QMGR AI(1800) AMQI() AQ() ACHL(HIGH) AACL(QMGR) ALR QLCAL(q-name) AQ(QMGR) ALR CHL(chl-name) CHLYP(DR) ACHL(QMGR) tatistics Messages otes tatistics Monitoring Data collects information about Webphere MQ resources and writes this information at configured intervals (using the AI attribute on ALR QMGR) to the YM.ADMI.AIIC.QUU. his data, written in the form of PCF records, can be post-processed to give information on the activity of the system. Data collection is split into 3 classes, statistics based on the activity of the whole system; statistics on the activity of the queue (per queue); and statistics on the activity of the channel (per channel). he PCF Header of an accounting data message will record the Command field as MQCMD_AIIC_MQI for the activity of the whole system; MQCMD_AIIC_Q for the per queue activity and MQCMD_AIIC_CHAL for the per channel activity.

38 Accounting Message Details MQCMD_ACCUIG_MQI Connection Details (application name / process Id / connection type / connect time) API counts (MQP / MQCL / MQG / MQPU /...) MQPU / MQG / MQG (browse) message counts: persistent / non-persistent bytes counts: persistent / non-persistent message sizes: persistent / non-persistent MQCMD_ACCUIG_Q Queue details: name, type pen details: first open time, last close time MQPU details: count, total bytes, msg-sizes (min / max) (persistent / non-persistent) MQG details: count, total bytes, msg-size (min / max), time-on-queue (min / avg / max) (persistent / non-persistent) MQG(browse): count, total bytes, msgsizes (min / max) (persistent / nonpersistent) tatistics Message Details MQCMD_AIIC_MQI API counts for each of the MQ API's (MQC / MQDIC / MQPU / MQG/...) otal message/bytes put to queues (persistent / non-persistent messages) otal message/bytes got from queues (persistent / non-persistent messages) MQCMD_AIIC_Q Minimum / Maximum depth of queue Average time-on-queue for messages retrieved from the queue API counts for G / PU / BRW (persistent / non-persistent) otal byte counts for G / PU / BRW (persistent / non-persistent) MQCMD_AIIC_CHAL umber of messages transferred (persistent / non-persistent) umber of bytes transferred (persistent / non-persistent) etwork times (min / avg / max) : measured on heartbeats xit imes (min / avg / max) Batch Info (total / number full / avg size) umber of PU retries