Part 3 - Performance: How to Fine-tune Your ODM Solution. An InformationWeek Webcast Sponsored by

Transcription

1 Part 3 - Performance: How to Fine-tune Your ODM Solution An InformationWeek Webcast Sponsored by

2 Webcast Logistics

3 Today s Presenters David Granshaw WODM Performance Architect (Events) Pierre-André Paumelle WODM Performance Architect (Rules)

4 Demo Series: Best Practices for Implementing Operational Decision Management Solutions Part 3: Performance: How to Fine-tune Your ODM Solution April 3 rd, 2012

5 WebSphere Operational Decision Management Relations between components WebSphere Decision Center Ruledocs Decision Center Console Rule Solutions for Office Decision Center for Business Space Session 1 focus Session 2 focus Session 3 focus Deploy Decision Center Repository Deploy Synchronize Synchronize Deploy Deploy 5 Rule Execution Server Rule Designer Event Designer WebSphere Decision Server Event Execution Runtime

6 Agenda WebSphere Operational Decision Management 1. Tuning Decision Server Events 2. Tuning Decision Server Rules 3. Tuning Decision Center 4. Performance Improvements in V7.5 Note the performance figures given in this presentation are just examples based on a typical workload. The performance impact of specific design and runtime changes will vary depending upon the details of the workload.

7 Decision Server Events Runtime Overview WODM Server DB Server Events Actions Technology Connectors Technology Connectors JMS (durable) Event Destination JMS (durable) Action Topic Events Runtime Events Repository Database All events and actions ultimately end up on the JMS event and action destinations (except events delivered via Extreme Scale). Technology Connectors can be used to send actions and receive events from a range of sources including JMS, File, HTTP, JDBC etc The Events Repository can be used to store information about the occurrence of events, history and any data that may be shared across events (CSBOs).

8 Event Designer : Durability of Events and Actions Non Durable processing can be 2.5 times faster than durable processing so if actions and events do not need to survive a server restart use non durable processing For further information on durability see the WODM Infocenter : Configuring durable and nondurable events and actions

9 Event Designer : Simple and Complex Event Rules Simple event rules By definition, Simple event rules can be evaluated without reference to previous events or actions and are therefore significantly faster e.g. if the deposit amount is > $500 then Complex Event Rules By definition, Complex event rules have a dependency on previous events or actions with the same context ID (e.g. customer reference number) e.g. If this customer has withdrawn money 3 times in the last week then For optimal performance do not add a Context Id to a Simple event rule unless required. Simple rules should only have a context ID defined if: Information about the occurrence of the events and action processed by this event rule are actually required by other, Complex event rules. Context Scoped Business Objects are being used

10 Relative Throughput (evts/s) Relative Throughput (evts/s) Event Designer : Simple Rules versus Complex Rules Simple event rules are faster than Complex rules by: Non Durable Processing: 87% Durable Processing 22% Simple Rule Simple Rule with Ctx ID Complex Rule Simple Rule Simple Rule with Ctx ID Complex Rule Non Durable 0 Durable Don t define a context ID on a Simple rule unless required Consider using Simple rules for filtering

11 Event Designer : Ordering Multiple Expressions/Filters in an Event Rule Rules will evaluate expressions intelligently from the top down, so place Simple expressions first For example, the second part of an and expression will only be evaluated if the first part is true. Complex expression first : More costly Complex expression is always evaluated Simple expression first : More costly Complex expression is only evaluated when the Simple expression is true

12 Relative Throughput (evts/s) Relative Throughput (evts/s) Event Designer : Ordering multiple filters in a rule Placing Simple filters first may be faster by: Non Durable Processing: 51% Durable Processing 11% Simple Filter First 1.6 Simple Filter First 1.4 Complex Filter First 1.4 Complex Filter First Non Durable 0 Durable Note: The performance difference will depend how often the first filter evaluates to true. In the graphs above the first filter evaluated to true 1 in every 100 events.

13 Event Designer : Ordering multiple filters in a rule For optimal performance: Always place Simple filters ahead of Complex filters. The same principle applies to any filters that require more processing, for example, simplistic filters should come before filters that involve database fetches or Context Scoped Business Objects Also, if an event triggers multiple Complex rules to be evaluated, group the rules together into one event rule group.

14 Relative Throughput (evts/s) Relative Throughput (evts/s) Event Designer : Technology Connectors These examples show an example of the performance impact of using the new JMS and HTTP connectors shipped in V7.5. Event Connector Action Connector No Connector HTTP Connector JMS Connector No Connector HTTP Connector JMS Connector Event Connector 0 Action Connector For optimal performance, send events in the correct format directly to Decision Server JMS event or durable event destinations. Also receive actions directly from the action or durable action topic

15 Event Runtime Performance : Top Ten Recommendations 1. Monitor system resources (cpu, disk, network, memory) This will ensure you target your performance tuning effectively 2. Check you are not writing large amounts of data to log, error log, ffdc or trace files It will significantly impact performance and may indicate an underlying problem in the project or configuration, which will degrade throughput. 3. Ensure you are not recording history unless you intend to. History is needed for user defined charts or reports of events, actions, filters and data. It is not needed for Complex rule evaluation or Context Scoped Business Objects. History records large amounts of data. It can impact performance by 50 to 80%.

16 Event Runtime Performance : Top Ten Recommendations 4. Tune the JVM A heap size of 1280MB is a good starting point for 32 bit systems; 4096MB for 64bit. Consider using generational garbage collection 5. Ensure you have a sufficient number of rule processing threads You should consider increasing the number of rule processing threads if events are building up on the event destination If you increase the number of rule processing threads you also need to consider increasing the connection pools for database connections, JMS Connections and data connection objects 6. Periodically (either manually or automatically) prune unnecessary context data from the Decision Server Event Repository to improve the database performance

17 Event Runtime Performance : Top Ten Recommendations 7. Ensure the repository database is tuned (or auto tuned) for the workload use separate, fast disk subsystems (SAN/RAID) for all logs and tables. 8. When using persistent Context Scoped Business Objects, the following database tuning may improve performance up to a factor of 3: For db2 consider using the statement concentrator: UPDATE DB CFG USING STMT_CONC LITERALS IMMEDIATE; For Oracle consider using cursor sharing: ALTER SYSTEM SET CURSOR_SHARING=FORCE SCOPE=BOTH; 9. Tune the JMS provider and monitor the queue/topic message depths 10. Use the Performance Monitoring Infrastructure to monitor Decision Server Events This will show statistics about events, actions, rules and other aspects of the runtime. See WODM Infocenter for more details

18 Note the next 4 slides were not presented at the webinar but provide additional detail on some of the top ten tuning recommendations.

19 Runtime : Minimising Disk Access - logging WODM has a number of log and error files. If these are growing significantly over time it can have a large impact on performance and may indicate an underlying problem. Check that the following files and ensure they are not growing significantly over time. If they are, read the logs to find the cause and rectify the problem: <was_install_dir>\profiles\ <profilename> \logs\systemout.log <was_install_dir>\profiles\ <profilename> \logs\systemerr.log <was_install_dir>\profiles\ <profilename> \ffdc\ (all files) Also confirm that large trace files are not being generated in <was_install_dir>\profiles\<profilename>\logs\trace.out <DecisionServer_install_dir>\director\logs\connectors.log (standalone connectors only) For further information on trace see the WODM Infocenter : enabling trace and Configuring the technology connectors log file.

20 Runtime : JVM Tuning The tuning of the JVM and the garbage collection policy can have a significant impact on performance. Settings will depend upon the WODM project and the OS but here are some recommended starting configurations 32 bit WODM installation 64 bit WODM installation Minimum heap size 1280 (MB) 4096 (MB) Maximum heap size 1280 (MB) 4096 (MB) Generic JVM arguments -Xgcpolicy:gencon Xmn1024M -Xgcpolicy:gencon Xmn2048M For further information on JVM tuning see the WAS Infocenter : JVM Tuning

21 Runtime : Decision Server Events Tuning If there is spare cpu on the WODM server and events are building up on the event destination this may indicate an insufficient number of rule processing threads. The number of threads can be increased in the WAS administrative console: Resources > Resource Environment > Resource environment entries > WbeSrv01 > Custom properties In a cluster it is recommended to use a prime number for the number of threads. This will provide the greatest concurrency.

22 Runtime : Decision Server Events Tuning If you do increase the number of rule processor threads: As each rule processor may use a database connection, ensure there are a sufficient number of JDBC threads Resources > JDBC > Data sources > Event Runtime Datasource > Connection pools Set Maximum connections to rule processor threads + history MDB concurrency As each rule processor thread may use a JMS connection, ensure there are a sufficient number of JMS connections Resources > JMS > Topic connection factories > WbeTopicConnectionFactory > Connection pools Set Maximum connections to rule processor threads + 10 When using database enrichment ensure there is a connection available for each rule processor thread. This is set in event designer data connections page. Set the connection pool limit to at least the number of rule processor threads

23 HA, Scalability and Performance When using WebSphere Default Messaging for JMS There are two key clustering architectures mentioned in the WODM Infocenter, the Silver Topology and the Golden Topology. The Golden topology has separate messaging and WODM clusters The Silver topology contains a messaging engine and a Decision Server Runtime in each cluster member There is also a choice of messaging engine configurations The scalability options will offer the opportunity to have multiple messaging engines and will provide the greatest throughput. For optimal performance use the Silver Topology with a Scalability or High availability with Scalability messaging engine configuration.

24 HA, Scalability and Performance Golden versus Silver Topology Relative Throughput (evts/s) Relative Throughput (evts/s) The Silver topology can out perform the Golden topology by 40 80% due to the co-location of the messaging engine and events runtimes Non Durable Processing Durable Processing Silver Topology 1.8 Silver Topology Golden Topology Golden Topology Non Durable 0 Durable

25 Agenda WebSphere Operational Decision Management 1. Tuning Decision Server Events 2. Tuning Decision Server Rules 3. Tuning Decision Center 4. Performance Improvements in V7.5

26 Ruleflow Decision Service architecture Rule Engine Execution IN IN Rules OUT BOM to XOM XOM BOM: Business Object Model XOM: executable Object Model 26

27 Decision Service performance best practices The performance cost for a Decision Service may look something like Execution Time XOM Ruleflow Functions B2X Misc. Rule engine JVM (GC) 27

28 Decision Service performance best practices XOM type choices: JAVA XOM better performance XML XOM Dynamicity Useful in case of XML model Ruleflow: Limit the size and the complexity of the ruleflow, it is interpreted. Always use the same engine algorithm to save memory

29 Decision Service performance best practices Choose the correct engine algorithm depending on your Decision Service. RetePlus (The default mode) Stateful application Rule chaining application May be useful in the case of many objects Sequential Application with many rules and few objects Most of the customer cases. Really efficient in multi-thread environment. Fastpath Application with rules implementing a decision structure and many objects May have longer compilation but faster at run time. Really efficient in multi-thread environment.

30 Decision Service performance best practices Choose your architecture depending on your Decision Service characteristics: Smallest memory consumption Integration with JEE animals Engine alone (EJB, MDB, POJO) No concurrent execution Engine alone JSE Rule Execution Server Concurrent executions Rule Execution Server JSE Rule Execution Server JEE Rule Execution Server JEE Call as a Web Service HTDS MTDS Call from COBOL application zres 30

31 Decision Server Rules tuning The log level in the Rule Execution Server should be set to level Severe or Warning in the production environment to increase performance. This property (TraceLevel) is accessible in the resource adaptor of the Rule Execution Server or in the ra.xml. Tune the GC and memory size. Starting configuration 64bits -Xgcpolicy:gencon Xmn2048M Xmx4096M Xms4096M Tune the RES pool size. A sizing methodology is available at :

32 Decision Server Rules tuning Tune the usage of the execution trace and the Decision Warehouse by filtering. A ruleset on an XML XOM should be configured to run in multiple simultaneous. This is configured using the ruleset property ruleset.xmldocumentdriverpool.maxsize. Use the ruleset caching information (XU dump) from the Rule Execution Server console to get the Rule Execution Server status.

33 TPS Decision Server Rules Performance Impact of the execution trace

34 TPS Decision Server Rules Performance Impact of the XOM type

35 TPS Decision Server Rules Performance Remote Web Service call vs. Local call

36 TPS Decision Server Rules Performance Fastpath Algorithm vs. Sequential Algorithm

37 Agenda WebSphere Operational Decision Management 1. Tuning Decision Server Events 2. Tuning Decision Server Rules 3. Tuning Decision Center 4. Performance Improvements in V7.5

38 Decision Center tuning Limit memory consumption Parsing a ruleset consumes memory so you can disable the Archive parsing flag option (Project > Edit project options > Check the ruleset archive). Use Automatic build to avoid a huge ruleset generation cost at first ruleset generation. Set in the installation manager the configuration parameter teamserver.build.archive.storage to file instead of memory. Build performance Disable Archive parsing flag. Disable Rule analysis checks.

39 Decision Center tuning Memory consumption estimation on a repository of business rules Memory foot print of a session is ~10 MB Ruleset generation : ~150 MB of short-lived objects created If you believe the number of users connected simultaneously to a single server will not fit into the memory allocated for a single VM, you should deploy Decision Center to a cluster of servers and use loadbalancing, so that HTTP sessions are dispatched according to the server s available memory.

40 Agenda WebSphere Operational Decision Management 1. Tuning Decision Server Events 2. Tuning Decision Server Rules 3. Tuning Decision Center 4. Performance Improvements in V7.5

41 Decision Server Rule Benchmark Scenarios (relative to GA) Rule Execution Server execution improvement (TPS) Between 1.01x to 3.97x faster than Average improvement factor = 25% The new Web Service stack (HTDS) for rulesets based on java XOM offers performance improvements Improvement factor 2x to 4x versus

42 Decision Center (relative to GA) Decision Center delivers a new ruleset build chain. Performance gains from 50% to 150% versus Rule Team Server V This optimization offers a better scalability of Decision Center as performance gains increase with rule project size. 42

43 Customer Based Scenario 1 : Credit Limit Check This scenario processes withdrawals from a bank and looks for customers who repeatedly exceed their credit limits. The majority of processing is with Simple event rules. Withdrawal Events Enrich Event: Get current balance and credit limit from DB Simple rule: Check if limit exceeded Synthetic event Generated 1 in every 100 events Complex rule: Has balance been exceeded before? Warn Customer Action Performance in V7.5 improved by (relative to V ): 18% for non durable processing 10% for durable processing

44 Customer Based Scenario 2 : Missed Payments This scenario looks for customers who have repeatedly missed payments on multiple accounts : credit card, overdraft and loan. It also uses CSBOs to keep track of the total of all missed payments. All event rules are Complex. Credit card, overdraft, loan missed payment events Enrich Event: Get current customer details from DB Complex rule: Have payments been missed on all accounts Synthetic event Generated 1 in every 100 events Complex rule: Has this occurred more than once and is the total debt > $5000 Warn Customer Action Performance in V7.5 improved by: 31% for non durable processing 14% for durable processing

45 Customer Based Scenario 3 : Rules and Events Scenario 3 is a customer based scenario, which looks for customers who are eligible for a discount as part of a new marketing campaign. The scenario involves DSE and DSR. Quote Requested Event Simple rule: determine discount from Decision Server Rules Decision Server Events (DSE) Action Event Complex rule: Has this customer applied before Offer Customer Discount Action SOAP Technology Connector Decision Server Rules (DSR) Performance in V7.5 improved by: 27% for non durable processing 16% for durable processing

46 Where to go for further information WebSphere Operational Decision Management Redbook on BRMS Tuning Proven Practices for Enhancing Performance: A Q & A for IBM WebSphere ILOG BRMS 7.1 Whitepaper on WebSphere Business Events Tuning Getting the best performance from your complex event processing system with WebSphere Business Events DeveloperWorks tool (System Integration Bus Performance) that automatically engages and displays PMI statistics relevant to Decision Server Event runtime performance WODM Performance Monitoring Tool

47 Q&A Session David Granshaw WODM Performance Architect (Events) Pierre-André Paumelle WODM Performance Architect (Rules)

48 Resources To view this or other events on-demand please visit: For more information please visit: WebSphere Operational Decision Management