IBM WebSphere Message Broker Message Monitoring, Auditing, Record and Replay. Tim Kimber WebSphere Message Broker Development IBM Hursley Park, UK

Transcription

1 IBM WebSphere Message Broker Message Monitoring, Auditing, Record and Replay Tim Kimber WebSphere Message Broker Development IBM Hursley Park, UK 1

2 Agenda Overview of Monitoring Monitoring support in WebSphere Message Broker Configuring monitoring events Record and Replay in WebSphere Message Broker V8

3 Agenda Overview of Monitoring Monitoring support in WebSphere Message Broker Configuring monitoring events Record and Replay in WebSphere Message Broker V8 3

4 Business critical applications and in-flight data Business applications rely on information exchange Application to application communication represents business interactions E.g. Web to Order system, Warehouse to Distribution, Payments to Accounts Business data passes through the ESB Message Broker has full visibility of the data content in any format Uniquely positioned to report on the status of the business data as it is being processed Can emit events containing relevant data as well as the success of failure of transactions Supports Business Activity Monitoring (BAM), auditing or data capture for potential later replay

5 What is Business Activity Monitoring? Ability to monitor your business performance Giving a real-time view of what is happening in your business Identifying problems in your business processes Opportunity to improve your business processes and business competitiveness A way of making the business more transparent Allows evidence-based decision making X-Ray for business processes WebSphere Business Monitor A BAM software product that subscribes to business events and displays the event information in a dashboard. 5

6 Business Activity Monitoring Why? Decision-makers need Key Performance Indicators (KPIs) Where? Best source for KPIs are the applications which run the business The ESB has visibility of all of those applications How? A Monitor Model describes how to process events and how to collect the information in dashboards. Applications can send the monitoring events which are collected by Business Monitor. 6

7 Agenda Overview of Monitoring Monitoring support in WebSphere Message Broker Configuring monitoring events Record and Replay in WebSphere Message Broker V8

8 Monitoring support in Message Broker Events are published to an MQ topic To allow multiple subscribers To allow each subscriber to choose the level of granularity Domain / Broker / Execution group / Message flow Event format is XML (published schema) Designed to be compatible with CBE Allows message broker to integrate with other monitoring applications Allows entire message to be captured and logged to a database for audit purposes Events can be forwarded to monitoring tools, e.g. WebSphere Business Monitor A WBM message driven bean is provided with the monitoring sample The bean is hosted on a WebSphere Application Server and wraps the message broker event with a Common Base Event wrapper and then submits it to the CEI server. 8

9 Monitoring Events A message flow can emit two kinds of events: Transaction events Each input node in a message flow contains three events sources (in addition to terminal events): transaction.start transaction.end transaction.rollback Terminal events Any terminal in a message flow can be an event source. If the event source is active, it emits an event each time a message passes through the terminal. All events are optional, and fully configurable Events can contain simple or complex data from message payload

10 Event Configuration Events are configured in two ways: Via the message flow editor Excellent support in message flow editor via Monitoring panel on all nodes. Via the command line Using a monitoring profile configurable service. Activate monitoring for the flow using mqsichangeflowmonitoring -c Enable event sources using mqsichangeflowmonitoring -i Ensure that the event sources are enabled using: mqsireportflowmonitoring The use of a monitoring profile configurable service overrides the monitoring properties of a message flow. The monitoring profile can be used to customize events without re-deploying the message flow.

11 Notes Terminal events Event is only emitted if the message passes through the terminal in a forward direction Events for error conditions should be configured on the nodes attached to the failure/catch terminals of the input node(s) Configuration Purposely designed to be administered without the toolkit So command-line / monitoring profile can do anything that the toolkit can do and toolkit config can be exported as monitoring profile to ease the transition ( see later slide re: mqsireportflowmonitoring ) 11

12 Monitoring event format details EventData EventIdentity eventname creationtime EventPointData EventSequence EventCorrelation counter localtransactionid parenttransactionid globaltransactionid MessageFlowData broker brokername / uuid executiongroup executiongroupname / uuid messageflow flowname / uuid ApplicationData SimpleContent node nodename terminalname name value datatype ComplexContent name BitstreamData Encoding Bitstream

13 Notes EventPointData/EventData Provides the 3 main items of data for a monitoring application Identity, Correlation, Sequence EventPointData/MessageFlowData Should be self-explanatory no surprises here ApplicationData Can come from message headers or body Automatically included as XML, even if source message was non-xml If the XPath/ESQL returned a simple element, it is placed in simplecontent, else it goes into complexcontent BitstreamData More details in later slides

14 Agenda Overview of Monitoring Monitoring support in WebSphere Message Broker Configuring monitoring events Record and Replay in WebSphere Message Broker V8

15 Configuring monitoring events using the toolkit Note: terminals are coloured red on this slide to highlight those terminals with monitoring events defined.

16 Notes Note the Monitoring page in the Properties view The canvas of the message flow has been clicked, so it is displaying configured event sources for the message flow not all event sources Clicking an individual node would show event sources for that node How many potential event sources are there in this flow? 15-3 terminal events on each node(though the compute node could have more if more output terminals are added) + the 3 transaction events on the input node ) Highlight the Event Source Address column ESA is used to address an event source from the command line, or from a monitoring profile. It will be unique within a message flow, provided that the flow does not contain duplicate node names. Terminals highlighted in red have events Not displayed like this in the editor!

17 Configuring monitoring events using the toolkit These event sources are internal to the input node. They are not located on a terminal Note: terminals are coloured red on this slide to highlight those terminals with monitoring events defined.

18 Configuring monitoring events using the toolkit Note: terminals are coloured red on this slide to highlight those terminals with monitoring events defined.

19 Configuring monitoring events using the toolkit Note: terminals are coloured red on this slide to highlight those terminals with monitoring events defined.

20 Configuring: Adding an event to a node

21 Configuring: Adding an event to a node Select the event source. Input nodes include the special transaction event sources

22 Configuring: Customizing an event The event name can be a literal value, or can be extracted from the message payload using an expression

23 Configuring: Adding a filter to an event XPath expression to indicate if this event should be emitted or not

24 Configuring: Adding a filter to an event (2) Expression evaluates to True event emitted False event not emitted Evaluated at runtime Set on event source definition Expression can reference fields from anywhere in the message assembly XPath expression builder support available Event filter appears on Monitoring summary table

25 Notes: Event Filter section in the Monitoring tab (V7 onwards) Enables user to filter out events which do not match a business rule. So events can be filtered at Message Broker rather than emitting to Business Monitor and filtering there - which will help performance The event filter can be set to a numeric, boolean or string XPath expression which will evaluate to boolean true or false. Typically the result of an a = 'b' or x > 'y' type test If the expression evaluates to true then events are emitted If the expression evaluates to false then events are not emitted The default setting is true()

26 Customizing an event bitstream data Click here to add part or all of the bitstream to the event

27 Customizing an event bitstream data Include headers, body or entire bitstream

28 Customizing an event bitstream data Encode bitstream as hexbinary, base64 or CData

29 Notes CData format for bitstream Not safe unless you know that the XML is free from invalid characters NB: CData does not protect you from invalid XML characters So not usually safe for use with All Because that will include headers which may contain binary data. Recommendation: Use CData encoding with care, and only with content set to Body

30 When to include bitstream data Bitstream data for monitoring Not expected to be used in standard BAM scenarios. Bitstream data for auditing Events can be captured and written to a database Often required by compliance bodies Bitstream data for resubmission Used by the Record & Replay facility in WMB V8 More later For V7 and earlier, WMBTM offering can provide capture/repair/resubmit based on these monitoring events

31 Notes: Command Line: mqsichangeflowmonitoring mqsichangeflowmonitoring activate monitoring for the specified message flow(s) set name of monitoring profile to use for the message flow(s) enable and disable individual event sources in a message flow Multiple event sources can be modified in a single command invocation No need to edit message flow and redeploy

32 Notes: Command Line: mqsireportflowmonitoring mqsireportflowmonitoring Reports whether monitoring is active, and name of monitoring profile report all configured event sources for a single message flow report all available event sources in a single message flow Export the current monitoring properties as a monitoring profile. If monitoring profile is in use, registry contents are written to file If node properties are in use, XML is constructed from them Tip: Use this to easily construct a monitoring profile, rather than hand-crafting it in a schema editor.

33 Notes: Example output from mqsireportflowmonitoring Example output from mqsireportflowmonitoring command with n option BIP8911I: Monitoring settings for flow 'TotalPurchaseOrderFlow' in execution group 'EventsEmitter.1' - State?: active, ProfileName: ''. BIP8912I: Event: 'InputOrder.transaction.Start', Event name: 'InputOrder.Trans ctionstart', Configured?: yes, State?: enabled BIP8912I: Event: 'InputOrder.transaction.End', Event name: 'InputOrder.Transac ionend', Configured?: yes, State?: enabled BIP8912I: Event: 'InputOrder.transaction.Rollback', Event name: 'InputOrder.Tr nsactionrollback', Configured?: yes, State?: enabled BIP8912I: Event: 'GoldOrderTotal.terminal.in', Event name: 'GoldOrderTotal.InT rminal', Configured?: yes, State?: enabled BIP8912I: Event: 'RegularOrderTotal.terminal.in', Event name: 'RegularOrderTot l.interminal', Configured?: yes, State?: enabled BIP8071I: Successful command completion.

34 Agenda Overview of Monitoring Monitoring support in WebSphere Message Broker Configuring monitoring events Record and Replay in WebSphere Message Broker V8

35 Message Broker Version 8 Record and Replay New Message Broker feature to enable record and replay in-flight data Universal support for file, messaging, web service, ERP & other data Browser tooling for access to end users, auditors and administrators Multiple data formats (binary, text, XML) Data Recording provides view of in-flight data for transfers in MB connected network Replay Information to target applications for resubmission or redelivery processing Multiple sources and targets WMB handles messages, databases, SAP, JMS and more Broad Range of Data Formats Handles binary, text and XML message formats Web based tooling graphical facilities for record, edit and replay; no need to program Secure sensitive information can be encrypted on the wire as necessary Source and Target Independent - information providers and consumers do not require change Simple Deployment Integrates with existing infrastructure record Message Database JMS SAP replay Record and Replay WebSphere Message Broker

36 High level lifecycle 1 A TRIGGER occurs that causes message details to be enqueued 2 The broker will RECORD the message in a database 3 Web user can QUERY saved messages 4 Web user can REPLAY messages to the same or a different source

37 1 Message Flow(s) 2 Data source (ODBC) 4 3 WMB V8 Broker HTTP/ REST Web User

38 1 Business Activity Monitoring message emission WMB s existing Business Monitoring framework provides a way of capturing messages Configure event emission at design time or operationally Customisable payload allows you to capture entire bit-streams or only the message elements you require Messages are emitted to a queue The Record part of this feature builds on this BAM framework

39 2 Message record Configure the broker to listen for these emitted messages Execution groups will asynchronously copy the messages to a target data source Execution group properties (via configurable service): Monitoring Topic $SYS/#/FLOW1 $SYS/#/FLOW2 Target data source AUDITDB AUDITDB Scalable solution with flexible recording from multiple execution groups to multiple datasources Design allows for future implementation of tracking and auditing of messages that are not sent through WMB (e.g. XI50, custom applications)

40

41 3 Querying messages Connect to the broker from a web browser to list available messages Sophisticated filters to narrow down to just the messages you are interested in e.g. by transaction, by message ID, by time Fully secured access (including encryption) using existing facilities (e.g. HTTPS) Design allows for future role based access to transactions and destinations. Uses a new public REST API (more on this later) This allows you to develop your own web interfaces, for custom branding and advanced query/edit scenarios

42 4 Replaying messages Web UI provides an option to submit selected message(s) to an MQ queue for reprocessing in a message flow (either the same or different) for retransmission directly to the destination application Destinations predefined by administrator Named set of MQ queues User selects message(s) and the target destination, and the broker will redeliver automatically. Remember that not all of the message payload may have been captured If you replay messages back to the original point of capture, or to a target application make sure that you record the entire bitstream!

43 4 Replaying messages to an application Replay messages directly to an application Request redelivery from the web UI or custom REST application The target application can request redelivery direct from the broker Intermediate message flow not required! HTTP/ REST Deliver MSGID x to destination MYDEST WMB Destination Name Destination Type Destination parameters MYDEST MQ MYTARGETAPP.INPUTQ

44 4 Replaying messages to a message flow Replay to the same or a different node Define available entry points operationally Replay scenarios may require a different path through the message flow You may need to write your message flow differently to cope with replay Destination Name Destination Type Destination parameters MYDEST2 MQ myeg / myflow / Replay Input node

45 Flexible system design Supports full multiplicity of all components Can configure multiple recorders, re-players, viewers Highly scalable Minimal performance impact Event emission from v7 brokers supported Different datasources to support different QoS Broker 1 Broker 2 Events Recording Broker REST Broker n Viewing Broker

46 New Web Interface In V7, the broker includes a HTTP listener for remote administration of service proxies (service federation management) WMB V8 Broker V8 has extended this listener to include general broker administration capabilities Including providing the interface for Record and Replay HTTP/ REST Accessed through a zero-footprint general purpose administration web UI Initially provided as a product sample Built on Record and Replay web UI The interface this web UI uses will be made an external product interface and use standard technology throughout e.g. REST Complements existing CMP interface Web User

47 The REST Administration Interface The web browser communicates with the broker through a series of REST verbs The URI identifies the object being manipulated Options on the URI describe parameters addressing the object (e.g. filters) HTTP payload provides details of the object Lists of objects returned WMB V8 Broker http/s port GET /admin/eg/myegname POST /admin/messages/mydatasource/msgid Web User

48 Summary Message Broker has built-in support for Business Activity Monitoring Designed for WebSphere Business Monitor / CBE integration Highly configurable Can be administered without the toolkit Also supports audit, capture/replay scenarios WMB Record and Replay addresses significant recurring problems Create or populate System of Record Repair malformed data for resubmission without original data source Replay historical data to target systems for reprocessing Feature Rich Record in-flight data: messages, files, services Query, View & Edit captured data Replay data on demand REST API for zero-footprint administration

49 Copyright and Trademarks IBM Corporation All Rights Reserved. IBM, the IBM logo, ibm.com are trademarks or registered trademarks of International Business Machines Corp., registered in many jurisdictions worldwide. Other product and service names might be trademarks of IBM or other companies. A current list of IBM trademarks is available on the Web at Copyright and trademark information at