Performance Best Practices for Oracle Enterprise Service Bus and Advanced Queueing

Transcription

1 Performance Best Practices for Oracle Enterprise Service Bus and Advanced Queueing A technical guide for SOA performance tuning Prepared by Oracle Corporation Inc. Creation Date: December 19, 2006 Last Updated: April 9, 2007 Version: 2.0 Author: hong.hua.chin@oracle.com Page 1 of 13

2 Chapter 1: Enterprise Service Bus Performance Tuning Step 1 Install and start the Oracle Enterprise Service Bus (ESB) using instructions found in: Step 2 Be sure to download the performance patch updates from the Internal ESB site (URL: D/dave.berry/Published%20Documents/ESB/Training/SW/ESB perf.zip) before you engage in a POC. Alternatively, contact your local Oracle Fusion Middleware sales consultant for assistance with the performance patch download. Step 3 Follow only the relevant instructions and familiarize with the topics found in Oracle Application Server Performance Guide (URL: Eg: If HTTP Server is not in use in your deployment environment, do not follow instructions in the Performance Guide in HTTP Server tuning. Step 4 Follow the relevant techniques described in the Performance Guide for Oracle Enterprise Manager: Step 5 Disable logging in Enterprise Manager. (URL: Step 6 Turn off instance tracking in ESB Control. (URL: DJGFD) Page 2 of 13

3 NOTE: Since ESB is stateless, the database performance is less of a concern if instance tracking was turned off. If you must leave instance tracking on then migrate away from thee fault Oracle Lite DB installed as part of SOA Suite. Step 7 Use syncronous services wherever possible avoiding costly file or database writes. (URL: DGHFG) Step 8 Minimize the use of transformations as much as possible. (URL: GA) Step 9 Use Domain-Value Maps in lieu of Database lookup within XSL transformations in order to minimize file I/O (URL: Note: Steps 10 to 16 are for File or In-Memory JMS use cases involving OC4J JMS provider. If you do not use File or In-Memory JMS, proceed to Step 17. JMS Adapter Activation Step 10 Locate the following snippet within the adapter.wsdl file.: <jca:operation ActivationSpec="oracle.tip.adapter.jms.inbound.JmsConsumeActivationSpec" DestinationName="jms/OSAOutQueue" UseMessageListener="false" PayloadType="TextMessage" OpaqueSchema="false" > </jca:operation> Step 11 Manually edit the adapter.wsdl file using Notepad or VI to include the UseServerTimeout="true" tag. <jca:operation Page 3 of 13

4 ActivationSpec="oracle.tip.adapter.jms.inbound.JmsConsumeActivationSpec" DestinationName="jms/OSAOutQueue" UseMessageListener="false" UseServerTimeout="true" PayloadType="TextMessage" OpaqueSchema="false" > </jca:operation> Step 12 View the amended.wsdl file in JDeveloper before re-registering the service. (URL: JMS Adapter Thread Increase Step 13 Manually edit the JMS adapter.esbsvc file using VI or Notepad to include the tag in bold: <invocation> <targetservice guid=" ad211dbaf173b29ccb6c0ef" qname="pearsona.osaoutqueue_rs"/> <interface> <wsdlurl>osaoutqueue.wsdl</wsdlurl> <porttype xmlns:tns=" ptt</porttype> </interface> </invocation> <endpointproperties> <property name="adapter.jms.receive.threads" value="5"/> </endpointproperties> Please increase the number of inbound threads in small increments in order to improve CPU usage of the container. Please note that the ability to parallelly process synchronous messages in ESB is governed by the message source - JCA adapters, SOAP Provider etc. The higher the number of threads you set on the message source, the higher the number of messages ESB engine will be able to process parallelly. Step 14 Edit the ESB Control Adapter service Properties tab in JDeveloper and re-register your service. (URL: JDHB) Page 4 of 13

5 Note: Increase the value of adapter.jms.receive.threads with small increments after testing and with much caution. Unsuitably high values can result in system unstability. Step 15 Edit <Oracle_Home>\j2ee\home\config\jms.xml Step 16 Turn off server instrumentation by setting "oc4j.jms.nodms" to "true" and "oc4j.jms.messagepoll" : <config-properties> <config-property name="oc4j.jms.nodms" value="true"/> <config-property name="oc4j.jms.messagepoll" value="100"/> </config-properties> Step 17 Set JVM Heap Size in <Oracle_Home>\opmn\conf\opmn.xml : eg: <process-type id="oc4j_esb" module-id="oc4j"> <module-data> <category id="start-parameters"> <data id="java-options" value="-xrs -server -Xmn1228m -XX:+AggressiveHeap - Xms2048m -Xmx2048m -XX:AppendRatio=3 - Djava.security.policy=$ORACLE_HOME/j2ee/oc4j_soa/config/java2.policy - Djava.awt.headless=true -Dhttp.webdir.enable=false - Doraesb.home=C:\oracle\soa\integration\esb -Dhttp.proxySet=false -Doc4j.userThreads=true - Doracle.mdb.fastUndeploy=60 -Dorabpel.home=C:\oracle\soa\bpel - Xbootclasspath^/p:C:\oracle\soa\bpel/lib/orabpel-boot.jar -Dhttp.proxySet=false" /> </category>.. </module-data> </process-type> Note: Explanations of the settings: -Xmx 2048m: Set the maximum heap size to be 2Gig. Because our ESB servers run on dedicated machines, we set the JVM heap size as high as possible. Please note this number is constrained by the Operating System's addressable memory space. For 32 bit operating systems the limit is 4 GB. For windows 32 bit, the limitation is 2GB (due to OS limitations) Xms 2048m: Set the initial heap size to be 2G. Sun recommends setting these values of maximum and initial heap size to be equal -Xmn1228m: Set the Eden space to be 60% of the maximum heap size. Eden space is a parameter in JVM garbage collector's generational settings. The garbage collector optimizes collection by classifying objects by how long they live. Most of the ESB engine's objects are short lived, thus they live in the "Eden" space. We recommend Page 5 of 13

6 sizing the "Eden" space to be 60 to 70 percent of the total heap size, as shown in the example. -XX:+AggressiveHeap: Set the aggressive heap. This setting is recommended for multiple CPU non-windows machines. This flag inspects the machine resources (size of memory and number of processors) and attempts to set various parameters to be optimal for long-running, memory allocation-intensive jobs. Note this flag has no effect on Microsoft Windows platform Step 18 Set the application and ESB datasources such that the OCI database driver is utilized instead. eg: <managed-data-source name='my Managed DataSource' jndi-name='jdbc/managedds_1' connection-pool-name='myconnectionpool'/> <connection-pool name='myconnectionpool' min-connections='5' max-connections='80'> <connection-factory factory-class='oracle.jdbc.xa.client.oraclexadatasource' user='scott' password='tiger' oracle.esbtest.com' /> </connection-pool> Step 19 If the following two conditions exist, adding additonal nodes to the OC4J cluster may help to boost the performance: CPU is saturated on ESB machine(s) A backlog appears at the inbound queue (URL: Step 20 In contrast, if the following conditions exist, add nodes to the inbound JMS queue cluster. CPU on ESB machines are under utilized (below 50%) No back log exists at the inbound queue Page 6 of 13

7 (URL: Step 21 Restart the OC4J instance. (URL: Step 22 Test the inbound queue throughput with custom written Java load-test programs and at the same time run another Java program to de-queue messages from the same JMS queue. Page 7 of 13

8 Chapter 2: Advanced Queing Performance Tuning INTRODUCTION Oracle Streams Advanced Queueing (AQ) is the high-performance integrated message queueing feature of the Oracle Database. Negating the need to purchase, test, and integrate, AQ provides the best value and performance to customers for their queueing needs. However, as with all enterprise-level software, some amount of tuning is required to get the very best performance out of AQ. This document assists our customers in maximizing the AQ performance and efficiency in their hardware and software environments. Storage Information storage is by far the most important aspect of running your operations. How well the storage performs is directly related to how well the Oracle Database and Advanced Queueing will perform. The following tips on storage tuning will take you a long way toward yielding the best performance from your hardware. Tablespace allocation Advanced Queueing is more than just queue tables under the covers. There are many different data structures that facilitate the speedy operation of the queue tables. A responsible DBA should thus make sure there is enough tablespace allocated for all of the underlying structures. Retention Retention is an important feature of AQ, but not required by every type of application. When it is not necessary, make sure not to use it. That way, disk space can be conserved, and operation will be slightly faster. Disk Striping Advanced Queueing is a very IO intensive process. Almost anything that increases IO throughput will increase Advanced Queueing performance. One of the most important IO bottlenecks revolves around the disk access. Thus, a great way to improve AQ performance is to stripe the data across drives. The appropriate level of data striping will help AQ performance considerably. init.ora Parameters The initrans parameter should be set to the average number of concurrent transactions pctfree should be sufficient to allow the ITL table to expand in times of heavy load. ENQUEUE/DEQUEUE Tuning Startup the queue monitor to ensure block cleanout of the dequeue IOT Page 8 of 13

9 Queue Monitor processes Queue Monitor (QMN) processes are responsible for a variety of queue maintenance tasks, especially those tasks related to timing and deletion. It is thus important to make sure at least one QMN process is always running. Preferably, run a few QMN processes to manage multiple queues simultaneously, since each QMN can only access a single queue at a time. Using REMOVE_NODATA When a subscriber is interested in purging messages from a queue while in the BROWSE dequeue mode, using the REMOVE_NODATA instead of REMOVE will save a substantial amount of time, especially for messages with large payloads. REMOVE allows the application to read a message and delete it. However, by using the BROWSE dequeue mode, the application already has knowledge of the message and thus has no need to retrieve the message again during the REMOVE operation. The REMOVE_NODATA is thus the preferred dequeue mode during BROWSE. NEXT Navigation Mode When dequeueing messages, it is best to use the default navigation mode, known as NEXT_MESSAGE. This queue navigation method allows the consumer process to iterate through the list of messages in a queue, amortizing the cost of accessing the queue over many messages by caching the results. However, for priority ordered queues, FIRST_MESSAGE is the recommended navigation mode, since the queue is no longer a FIFO queue, making cached results potentially stale. Correlation Identifiers for Dequeueing Correlation identifiers are application specific identifiers that are uninterpreted by Advanced Queueing. Many applications use these identifiers to categorize messages while enqueueing for later dequeueing in batch. When used as a criterion for dequeueing, correlation identifiers should be indexed for fastest operation. Multiple Subscribers for Multiple Dequeue Processes When an application has multiple processes that access the same queue, it is better that each process be a separate subscriber. There will be less contention among processes for data, since one subscriber will not have a lock over multiple messages. Concurrency can increase without much overhead. PROPAGATION Each propagation schedule is assigned a job queue process. We recommend that you have twice the number of job queue processes as propagation schedules, up to a maximum of 999 job queue processes. job queue processes = min ( 2 * propagation_schedules, 999 ) Page 9 of 13

10 Propagation Frequency Tuning the propagation frequency should only be done using the latency and next_time parameters in the propagation schedule. The duration only controls how long a single propagation operation will last. Disabling Propagation Propagation is useful for distributing messages to different databases, but still consumes some computational resources. Thus, if it is not required, turn it off. Pay special attention to Disk I-O The main bottleneck with OC4J-JMS is usually disk I/O and due to how the old (a new provider is in the works for AS11) OC4J-JMS provider serializes all server operations. How to confirm that disk I/O is indeed the bottleneck? Switch destinations to in-memory and re-test What can you do about it? Switch the JMS provider to the machine with the fastest disk Ensure you are not persisting to a remote disk (ex: via a NFS-mounted directory) - unless of course this is done through specialized SAN. distribute the destinations to various servers (needs different JVM) - this of course introduces an administrative overhead (multiple providers to manage). Segment the business that is performing the messaging by a key value For instance create a different queue for each geographical region or a different queue for each supplier. Adding a new supplier just involves creating a new queue. Scalability is then increased as the messaging can occur in parallel. IF business_key = REW THEN enqueue_stock (payload1); ß a stored procedure that enqueues this payload1 to the STOCK queue ELSIF business_key = MSB THEN enqueue_options (payload1); Use load balancing This involves having a number of enqueuers and dequeuers and spreading the load dynamically between them using a round robin or hashing algorithm. For instance the following code takes a message payload and sends it to a different recipient each time. There would be a dequeuer for each recipient acting in parallel. Loop forever For 1..BALFACTOR LOOP message := get the payload; Page 10 of 13

11 message_properties.recipient_list(1):=sys.aq$_agent ( S BALFACTOR,NULL,NULL); enqueue... END LOOP; Combine the above together and try to spread the messaging load. Don t use the same consumer name for multiple dequeuers. Database tuning Stripe the database (system, rollback, redo, temp) and queue tables across as many physical disk spindles (and controllers!) as possible using RAW I/O and a volume manager. A good stripe width is 16K-32K for smaller sized messages of 1K. A sustained I/O rate of 42Mb per second can yield messages per second (mps). AQ is greatly effected (0-50%) by the performance of the LGWR. Single consumer enqueue redo = sizeof (table_insert_redo) + 2 * sizeof (index_insert_redo) Multiple consumer enqueue redo = sizeof (table_insert_redo)+ (( 2 * N) + 1) sizeof(index_insert_redo) where N = Number of Recipients Tune the LGWR by using special cache devices like EMC or stripe the redo at a small stripe width (32K) across multiple disks using RAW I/O. Try to isolate I/O to these disks for just the redo logs. Parallelize - Single Enqueuer performance can be improved by using multiple enqueue processes (450% improvement from 1-9 enqueuers). Single Dequeuer performance can be improved by using multiple dequeue processes (488% improvement from 1-9 dequeuers) Use a large database block size. 8K seemed to work best for a 1K payload. Use a large DB_BLOCK_BUFFERS. Performance of AQ degrades (35%) whilst it is waiting for FREE BUFFERs in the SGA. Isolate the heavy queues into separate buffer pool KEEP caches. Ensure you have enough physical rollback segments. You will find that CR activity on the segment headers is very high. The Index Only Tables for AQ have a default PCTFREE value of 0 (hence not much space left for dynamic TX header expansion). Therefore set INITRANS correctly and FREELISTS correctly to avoid high buffer busy waits for data and index blocks. (2N*CPUS+1) works well where N is increased as performance increases. Page 11 of 13

12 Turn timed_statistics off. It degrades AQ performance by 12%. Dequeue_options.navigation = NEXT (default) performs 45% better than Dequeue_options.navigation = FIRST Page 12 of 13

13 Chapter 3: Mandatory Additional Readings Oracle Adapter documentation: Oracle Enterprise Messaging section of the Oracle Container for J2EE services documentation: Page 13 of 13